Parsing Toyota Parts Website Help

I'm sure I'm not the only that has gone through the pain of comparing prices on the Toyota Parts Website. The darn website makes you go to different websites in order to see the prices (this is pretty annoying).

(Q) My question to to the community is has anyone tried to automate the process for comparing prices?
I think it would be really beneficial to the community to have a tool that goes through the different dealer websites, grabs the price and writes it to text file.

Desired Output:
DealerName, PartNumber, Price\n


I think it's possible to automate the process since the URL is composed of the following variables: Dealer Name, Part Name, Part ID, and Part Number. Here's an example:
https://parts.elmoretoyota.com/p/toyota__tacoma/TRD-Front-Skid-Plate/75489794/PTR6035190.html.

Here's a screenshot of the HTML code. Notice the URL pattern.

I tried writing something in python using urllib3 and bs4, but ran into some trouble returning the html. When I try to connect to the URL I get the following warning "InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)." After this warning the HTML returned is a RunTime Error page (I'm sure we've all seen this run time error page before).

Here's a snippet of the code:

from bs4 import BeautifulSoup
import urllib3
import sys
#####################################
def getUrl(DealerName,PartName,PartID,PartNumber):
return "https://parts."+DealerName+".com/p/toyota__tacoma/"+PartName + "/" + PartID + "/"+PartNumber+".html"
#####################################
DealerName = "elmoretoyota"
PartName = "TRD-Front-Skid-Plate"
PartID = "75489794"
PartNumber = "PTR6035190"

url_to_srch = getUrl(DealerName,PartName,PartID,PartNumber)
Http = urllib3.PoolManager()
Response = Http.request('GET',url_to_srch,verify=True)
Html = Response.data.decode('utf-8')
Soup = BeautifulSoup(Html, "html.parser")
Data = Soup.find_all("a")

print(url_to_srch)
print(Data)
#####################################

After some time on google, I found that it's related to security certificate.
(Q) Does anyone on here know how to get around this issue?

 

Looks like I didn't search that well. I had some luck following this: https://docs.python-guide.org/scenarios/scrape/.

Can't wait to start working on this on Friday! Stay tuned folks!

 

So I found out that Google pretty much has the functionality covered (faster, cleaner, and better). In any case this was a fun exercise to keep me occupied, as well as familiarize myself with Python a bit. I work with Matlab for a living and am by no means a SW guy.


So here's the official unoffical how-to run the code.
Python version: 3

Packages used: sys, html, requests

How to run :

1. Setup the environment. Copy the python file (ToyotaSearch.py) and setup text file (setupToySearch.txt) to some folder on your PC. ToyotaSearch.py is the actual python script. You don't need to worry about it unless you want to look under the hood and modify it. setupToySearch.txt is a text file that will be populated in step 2. This is temporary and I think it's more user friendly than asking the user to write extra lines into the command line.
2. Navigate to the top level parts website that you are interested in. Copy the URL at the top of the page to setupToySearch.txt.
   SetupToySearch.txt should look something like this:
3. Open open window. Hit windows key and type “cmd” and navigate to the folder where the script is located.
4. To run the python script simply use the following command:python ToyotaSearch.py The program defaults to zip code 90650. This can be configured for any zip code by running the following python ToyotaSearch.py 90210
5. The program writes and creates a .csv file named results_toyota_shop.csv. You can open this with excel and sort as necessary

But hey!! Palo Alto has the cheapest skid plate!!! WUT WUT!!


You can do the sorting in excel or modify the price list in the code by using the sort function. I'll leave that as an exercise for the use

 

Here's the Source Code folks!

import sys
from lxml import html
import requests


def runSetup(File):
setup_components = []

with open(File) as f:
StringArray = f.read().split("/")

ptName = StringArray[-3]
uID = StringArray[-2]
ptNum = StringArray[-1].replace('.html','')
return ptNum,ptName,uID

# overhead stuff that grabs the inputs from the command line
if len( sys.argv ) == 2:
zip_code = sys.argv[1]
else:
zip_code = "90650"

print("*********************************")
SETUPFILE = "./setupToySearch.txt"
print("Returning part number, part name, and unique identifier key from", SETUPFILE)

ptNum,ptName,uID = runSetup(SETUPFILE)
print("Part Name: " ,ptName)
print("Part Number: ",ptNum)
print("Unique ID: " ,uID)
print("*********************************")

top_dealers = "https://parts.toyota.com/findDealer...*modelYear=2018*modelName=tacoma*stockNumber="+ptNum+"*ukey_product="+uID +"&ukey_product="+uID +"&zipCode="+str(zip_code)

print(top_dealers)
page = requests.get(top_dealers)
tree = html.fromstring(page.content)
url_vec = tree.xpath('//div[@class="col-md-3"]/a/@href')
n_url = len(url_vec)

# This section of the code iterates through each url in url_vec and returns the price for each part.
price_vec = []
print("Total # of dealers searching:", n_url)
print("Returning prices from dealers...")
for url in url_vec:
page = requests.get(url)
tree = html.fromstring(page.content)

price = tree.xpath('//span[@class="productPriceSpan money-3"]/text()')
price_vec.append(price[0])

msg = url
print("Done.", "Writing results to file...")

file = open("./results_toyota_shop.csv","w")

#for url in url_vec:
for index in range(len(price_vec)):
# some tring clean up before saving to text file
p = price_vec[index].replace(' ','');
p = p.replace('\n','')
p = p.replace('\r','')
msg = url_vec[index] + ","+p+"\n"
file.write(msg)
file.close()

 

Thanks for sharing! I'll giver a go tomorrow.

 

Reminds me of when I spent over a month on a Matlab script to parse Yahoo Finance to tie into the neural net toolbox for stock trading optimization lol. Since you took it this far, a nice next step would be to work out a webserver to host your sorts.

 

I’m not familiar with webservers and hosting, but I’ll look into it!

 

Awesome!!! Much better than my attempt!