Computer programmers, lend me your ear... (deal related)

EDIT: this idea has become a web app:
https://toyotascrape2.herokuapp.com/get_url/


I have noticed when shopping for parts online at Toyota dealers, the prices of the parts can swing wildly from dealer to dealer and there's no easy way to comparison shop without visiting each dealer's website which simply isn't practical.

I have also noticed that each dealer's website for any part has basically the same URL, the only difference being the dealer's name in the URL. I made a very simple python script to scrape the sites of a few local dealers and return the prices of the TRD cat-back exhaust by looping through a list of dealer names and replacing the name in a base URL as necessary.

I am not a computer programmer but I have been learning python during quarantine so I have some basic knowledge. I am sure an actual programmer could make an awesome application to check prices of all dealers in the nation if you simply input a part number.

Below is a screen shot of my basic script. As you can see the price of the cat-back can swing from $495 to $393 depending on the dealer. Basically I just hope to inspire someone who knows what they're doing to help TW members find the best dealer pricing for parts.



ETA: this excel version is pretty awesome and easy to use. Credit to @Fargo Taco for building it!

 

I don't know how to script, but I'm very interested! I spent about an hour looking for a TRD PRO IFS and tho whole time I was wishing there was a better way

 

Bro you wrote a python script in Jupyter Notebook or Lab to scrape a website for prices. Yes, your are a programmer. Keep up the learning it could pay off way more than saving some bucks on an exhaust.

Your idea reminds me of something an old college friend tried to get me to help him with 20 years ago. Except it was for used textbooks. Then a few years ago a colleague wrote something similar except it would game those old penny bid sites.

I like your idea. Can you post the script on Github in a repo or as a gist?

 

It wouldn't take long to turn that into a command line, or even a GUI with PyQT or Tkinter. Keep hacking at it!

 

I scrapped a list of 94 dealers and am expanding my pool of dealers to check. I’m also using the long bed mat as the price to check since more people will be interested in buying those

 

Are shipping prices available too? Sometimes those can be dealbreakers.

 

This has some serious potential. There used to be some price aggregators for PC components, ammo, etc back in the day. I could see turning this into a webapp and hosting that with some adds to cover your expenses.

I'm retyping code now to play around with it.

 

I've been using FastAPI the last couple weeks at work. It would be perfect for turning this into a web app.
Or Django, if you want something a little fancier..

 

Something's messed up with my venv. I had to use "html.parser" instead of "html5lib". So if anyone runs into problems, give that change a shot.

 

I haven't heard of FastAPI. I'm not normally a web developer, I do the data science and ML side of things. I've been messing around with Dash which is made by the Plotly folks. Getting ahead of myself, but you could even track the trend of prices.

 

I'm not much of a web programmer either, more just of a generalist.

There's so many ways to go with something like this if someone was sufficiently motivated. Django has a real nice built in database system though if long term tracking was the goal.

Or, you could pipe it into influxdb and then just graph it with grafana....

Options-a-plenty with python.

 

Attached is a list of 90 dealer names in excel. THe price is the price of the long bed tacoma mat for a second gen. The dealer names will help others trying to build a script.

 

I have no help to offer you, I just wanted to say what a great idea this is. I’ve gone through the list of many dealers looking at prices of scooped hoods and the short bed mat.

 

I've never made it very far with Grafana. Not it's fault, just not within my attention span. If it gets into a DB that supports SQL, then I can turn Pandas and Scikit-Learn loose on it and profit.

 

You're my new hero.

 

Also it helps to put the scrape in a try/except block because times the dealer doesn't have the part, etc.

 

Imma hop on board. My degree is in CS but unfortunately that's not my job and my experience lies primarily with Java and Excel/VBA. Keep up the good work!

 

I'm ignorant to everything that has been said in this thread so far. I'll still throw this out here. I've seen truck guys walk into a retail establishment ask for a certain product for his truck and not question the price quoted by the guy behind the counter.

 

@TheDevilYouLove do you have an account on Github?

 

I do not, I'm more than happy to post my jypiter notebook to it though if I can figure out how to do it