Computer programmers, lend me your ear... (deal related)

I added some crappy .css and this what the output page looks like. I hope to get it hosted soon so people can use it.

 

Holy crap, I got my web app working!

The only issue is with heroku who is hosting the app for free, when the app goes and scrapes over 100 dealers it takes a few minutes and it times out and fails. So I had to parse the dealer list down pretty significantly, just to scrape 5 dealers so that is sort of a bummer.

This is the list of dealers is scrapes:

'355toyota',
'camelbacktoyota',
'darcarstoyota',
'passporttoyota',
'koch33toyota'

https://toyotascrape2.herokuapp.com/

For the programming nerds here was the problem: when you build a web app you have to have a file called requirements.txt that tells the web host what tools to install to make everything work. I realized that one of the tools was missing (html5lib). Once I added that in it actually worked! The part of the code where that was being used was in a Try block, and since it failed in the try block, I didn't get any error messages so it was tricky to track down.

 

Looks like heroku will handle 10 dealers no prob and when I have some time, I'll see if it will do 15.

What are the most common Toyota parts you would want to price check the dealers on? I think I'll add some links that to the short bed mat, long bed mat and the TRD Cat back (I really like that exhaust, lol). That way those links are right at hand to easily be copied and pasted to search. Any other parts that people often check?

https://toyotascrape2.herokuapp.com/

 

Way to go & very impressive !

Looks really good too

 

Seems to be working with 20 dealers:

'cobbcountytoyota',
'erniepalmertoyota',
'germaintoyotaofnaples',
'hamertoyota',
'heartlandtoyota',
'hendricktoyotaapex',
'jimellistoyota',
'josephtoyota',
'lagrangetoyota',
'lewistoyota',
'maderatoyota',
'passporttoyota',
'rogersandrogerstoyota',
'rohrichtoyota',
'salinastoyota',
'sheehytoyotafredericksburg',
'toyotaofdecatur',
'toyotaofdeerfieldbeach',
'toyotaofnorthmiami',
'villagetoyota',


https://toyotascrape2.herokuapp.com/get_url/

 

It might be a network traffic cap set by the host. Might want to look into using XMLHttpRequest and buffer the loads to slow down the overall traffic hit. This would allow you to slowly load as many requests as you want in the background and update the display in real time. The overall effect would be to fill in the items from top down in a "wave" effect, but on a mobile device you probably wouldn't notice it anyway. Not having programmed in python before, I'm unable to give you any info on how to implement this though. But, looking good so far, good job!

 

I think I’ll add a button for the long bed mat and the short bed mat, so if you want prices for either of those two items, you just click the respective button and it gets the prices without having to copy/paste the url.

 

Does anyone have a cool pic of their Tacoma they want to give me for the header photo? Something that looks good wide and short, I’m thinking something with a desert/Utah background.

 

Couldn't be much further from a Utah desert.

 

Thanks! You have won my photo contest, lol

 

I'd like to thank the academy, my parents, the lack of other entries...

 

Tried to use your handy tool to sniff out some pro wheel prices and it returns an error. This is the URL I used: https://parts.toyota.com/p/Toyota__/TRD-17-In-Alloy-Wheel---Matte-Gray/69529065/PTR2035110GR.html

Also tried removing everything between the /p and number beginning with 6.

 

Thanks for using it! Not sure why you got an error, it seems to be working for me using that link you posted. Got these results back, hope this saves you a few bucks!:

dealer price url
cobbcountytoyota 179.11 https://parts.cobbcountytoyota.com/...Wheel---Matte-Gray/69529065/PTR2035110GR.html
toyotaofdeerfieldbeach 179.11 https://parts.toyotaofdeerfieldbeac...Wheel---Matte-Gray/69529065/PTR2035110GR.html
toyotaofdecatur 179.11 https://parts.toyotaofdecatur.com/p...Wheel---Matte-Gray/69529065/PTR2035110GR.html
sheehytoyotafredericksburg 179.11 https://parts.sheehytoyotafrederick...Wheel---Matte-Gray/69529065/PTR2035110GR.html
salinastoyota 179.11 https://parts.salinastoyota.com/p/T...Wheel---Matte-Gray/69529065/PTR2035110GR.html
rohrichtoyota 179.11 https://parts.rohrichtoyota.com/p/T...Wheel---Matte-Gray/69529065/PTR2035110GR.html
rogersandrogerstoyota 179.11 https://parts.rogersandrogerstoyota...Wheel---Matte-Gray/69529065/PTR2035110GR.html
passporttoyota 179.11 https://parts.passporttoyota.com/p/...Wheel---Matte-Gray/69529065/PTR2035110GR.html
maderatoyota 179.11 https://parts.maderatoyota.com/p/To...Wheel---Matte-Gray/69529065/PTR2035110GR.html
lewistoyota 179.11 https://parts.lewistoyota.com/p/Toy...Wheel---Matte-Gray/69529065/PTR2035110GR.html
lagrangetoyota 179.11 https://parts.lagrangetoyota.com/p/...Wheel---Matte-Gray/69529065/PTR2035110GR.html
josephtoyota 179.11 https://parts.josephtoyota.com/p/To...Wheel---Matte-Gray/69529065/PTR2035110GR.html
jimellistoyota 179.11 https://parts.jimellistoyota.com/p/...Wheel---Matte-Gray/69529065/PTR2035110GR.html
germaintoyotaofnaples 179.11 https://parts.germaintoyotaofnaples...Wheel---Matte-Gray/69529065/PTR2035110GR.html
erniepalmertoyota 179.11 https://parts.erniepalmertoyota.com...Wheel---Matte-Gray/69529065/PTR2035110GR.html
toyotaofnorthmiami 179.11 https://parts.toyotaofnorthmiami.co...Wheel---Matte-Gray/69529065/PTR2035110GR.html
villagetoyota 179.11 https://parts.villagetoyota.com/p/T...Wheel---Matte-Gray/69529065/PTR2035110GR.html
hendricktoyotaapex 180.67 https://parts.hendricktoyotaapex.co...Wheel---Matte-Gray/69529065/PTR2035110GR.html
hamertoyota 180.67 https://parts.hamertoyota.com/p/Toy...Wheel---Matte-Gray/69529065/PTR2035110GR.html
heartlandtoyota 182.23 https://parts.heartlandtoyota.com/p...Wheel---Matte-Gray/69529065/PTR2035110GR.html

ETA- don't remove any part of the URL, the only thing the app does is insert different dealer names into a base URL. If you remove parts of the base URL, it won't work when the dealer name is inserted into it.

 

Weird. I tried again, this time disabling my network filter and it worked. Then I turned it back on and it failed. Then I tried again and it worked (filter still on). It's a DNS filter so if it's able to resolve a query (filter off) it should still work once the filter is back on since it'll be in the cache. That said, I didn't see any queries that were blocked so I don't know what's going on in the Matrix.

 

Thanks for using it and thanks for the feedback!!!

 

Thanks for taking the time to make it!

 

This tool is money, literally. Used it again on the front header rail for the bed and 1L of transfer case fluid. Found a dealer in both lists and saved like 40 bucks. Even my local dealer couldn't price match. The FREESHIP code is still active so saved an additional 26 on shipping.

I wonder if it's possible to integrate this into TW? It'd absolutely be useful and if hosted on whatever servers TW uses, it'd be able to support more dealers, right?

 

@tcBob - Thoughts?

 

This is a pretty damn good tool and maybe, with some work from the community here at TW, it could actually be expanded to cover many many more dealerships, go by state only, metro area only (or something).
I've only ever written a few things in Python, mostly used PHP, javascript, Perl, SQL, etc...
But if you want some help. let me know.
I also might have a better place for you to host it... maybe.

Might want to create a github for your code too!

a little background:
Probably one of the hardest things I have created programming-wise, so far, is: https://mapping.kg6wxc.net/meshmap
(that page is a mapping tool we use for our Amateur Radio data networks we create, there is also code on the "backend" that goes out and polls each and every one of those "nodes" on that network.)

I am also part of the team that codes the software we use for all the routers on this network (they all run a very customized version of OpenWRT)
(https://www.arednmesh.org)

 

I did basically the same thing with a Google Sheet, which might be better because it allows to scrape a crapload of sites using Google's unlimited capacity. Only bummer is it uses an add-on function called "IMPORTFROMWEB" which I think costs money but I am using it under a free trial. It works fine for me, someone please take it for a spin and see if it works for you as well.

To use it just enter the part URL in the orange cell.

https://docs.google.com/spreadsheets/d/1nB4qU33u-9Q8qXMDfEHxSg4qM89YPuVomthtAHeKrqo/edit?usp=sharing