Computer programmers, lend me your ear... (deal related)

You could easily pick up Python. I'm also a CS guy and Java was my first professional language. I didn't pick up Python until about 2-3 years ago. In some respects it's easier. It's just not statically typed and it's interpreted, not compiled. So sometimes bugs can easily slip through the cracks.

 

dealers=['355toyota',
'adamstoyota',
'andrewtoyota',
'aubreyalexandertoyota',
'baierltoyota',
'brewerairporttoyota',
'brownstoyota',
'buttetoyota',
'cobbcountytoyota',
'conicellitoyotaofspringfield',
'crowntoyota',
'dandeerytoyota',
'darcarstoyota',
'darcarstoyotaofbaltimore',
'darcarstoyotaoffrederick',
'deserttoyota',
'eastmadisontoyota',
'elmoretoyota',
'evanstoyota',
'foothillstoyota',
'frankforttoyota',
'fredandersontoyota',
'ganleytoyota',
'greenstoyotaoflexington',
'hamertoyota',
'hatchtoyota',
'hendricktoyotaapex',
'hendricktoyotaconcord',
'hickorytoyota',
'invergrovetoyota',
'jimcolemantoyota',
'jimellistoyota',
'josephtoyota',
'kerrytoyota',
'kingstoyota',
'koch33toyota',
'lewistoyota',
'lhmtoyotalemongrove',
'lithiatoyotabillings',
'loyaltytoyotarichmond',
'maderatoyota',
'maitatoyota',
'mariettatoyota',
'mcgeetoyotaofdudley',
'mcgeorgetoyota',
'metrotoyota',
'miltonrubentoyota',
'myalexandertoyota',
'newboldtoyota',
'oxmoortoyota',
'passporttoyota',
'prestigetoyota',
'prestigetoyota-ny',
'pretoy',
'puentehillstoyota',
'reinhardtmotors',
'rogersandrogerstoyota',
'rohrichtoyota',
'rosevilletoyota',
'salinastoyota',
'savannahtoyota',
'schaumburgtoyota',
'sftoyota',
'sheehytoyotafredericksburg',
'sparkstoyota',
'springhilltoyota',
'stcharlestoyota',
'stcloudtoyota',
'stevenscreektoyota',
'stevinsontoyotaeast',
'stevinsontoyotawest',
'sullivanbrotherstoyota',
'sunrisetoyota',
'teamtoyotaglenmills',
'tonydivinotoyota',
'toyotaofdecatur',
'toyotaofelcajon',
'toyotaofgladstone',
'toyotaofgreenville',
'toyotaoflakecity',
'toyotaoflouisville',
'toyotaofredlands',
'toyotaofscranton',
'toyotaofwilmington',
'toyotapaloalto',
'toyotawc',
'tustintoyota',
'tvtoyota',
'venturatoyota',
'waldorftoyota',
'walkertoyota',
'westcoasttoyotalb',
]

 

PM inbound.

 

Maybe try renaming the extension to .txt and attaching it the same way you did the Excel files.

 

If you get to the point where you want to take a stab at making a website, a $5/mo VPS instance is a good place to start. Or, for a little less money, I've been pretty happy with nearlyfreespeech.net

 

Yeah that stuff is so cheap these days and pretty easy to manage. And honestly, if you just scraped the websites as a batch process once or twice a day and not per request, then the processing/bandwidth needs should be minimal.

 

Agreed. Or you could scrape them on request, and then store that value for a period of time. Then you're only scraping the things people are looking for, and not the whole catalog.

 

Yeah that's a good point, treat it like a cache.

 

I love Pandas! I’ve been playing around with stock market data in Pandas and it’s an incredible tool.

 

Oh yeah Pandas is great. I was doing some of that last week. This graph is from Plotly in Jupyer, data loaded and processed using Pandas. Although Plotly in Jupyter is a bit of a pain to set up. It is dynamic so you can scroll, zoom, select data, etc.

 

I'll have to play with plotly... plotting has been a thorn in my side for way too long.




Also, for more inspiration on this project, check out camelcamelcamel.com

 

I know juuuuust enought PowerBI and Excel Power Query to be dangerous. I have managed to be able to separate and recompile the URLs with the dealership list then pull the price for the item, but only from one dealership so far. I'm having issues automating pulling from multiple dealerships.

 

Plotting using something like Matplotlib is fine as long as you don't want to customize the graphs too much and static graphs are OK. There are well known bugs that just make Matplotlib painful. I ran into a major regression last year that cost me a week and using Matplotlib was a requirement.

Plotly is pretty good, just integrating with Jupyter requires some extra steps. You need to install ipywidget, npm, and do some additional steps. When looking at their website, I found it not entirely obvious this was required. If you're doing Pandas, then it is also worth checking out Seaborn. I think it's even made by Pandas maintainers. I haven't used Bokeh very much so I can't really speak to it.

 

If we architected this app right, it could operate not only against Toyota dealers, but also against other sellers. For example, those that sell Prinsu racks. It would just require different parsing for each site. The Toyota site makes it easy.

 

Yeah shipping would be a problem. My thinking was that this would really be helpful when they have the sales like they had for Memorial Day where there was free shipping if you spent >$50. If shipping is free, that makes it easy. Barring that you might have to call the dealer and see if they'd price match a specific other dealer.

 

Yeah the thing I need to do in the very near future is create a time series graph thats dynamic, so I'm looking at Rickshaw first since its based on D3 and I have a coworker that is using d3. But I'm also looking at canvas.js and plotly.

 

I still use http://gunbot.net/ and it finds ammo prices around the net, idk if this is any help

Prices vary from dealer to dealer. For instance a 2020 TRD PRO IFS is $400 at one dealer and at another dealer it is $310 (theoretically)

 

I think imma play around with storing the prices in a database so price changes over time can be analyzed, sort of like what camelcamelcamel does for Amazon pricing. Also need to build out a more comprehensive list of dealers. 100 or so isn’t that many.

 

I can send you want I did yesterday as a staring point. I won’t be able to do any more work on it this week.

It has a DB and pulls the dealers from that. It also has a basic web service to kick things off. Python, Docker, and MySQL.

 

Awesome!