Back in October I wrote a little process to grab data from our Scout system in an attempt to provide a customized view of current traffic on the route from work to home. The process worked about 2.4% of the time. I think my main problem was the way I was trying to hijack user state to keep the page alive. The first attempt was scraping the data off the site.
Fast forward a few months later, and a new job later, and I wanted a process to grab the speed of the speed sensors on my route from work to home. The new gig has different hours so I am fighting the traffic a little more.
I used Fiddler2 to investigate how the data was being populated on the KC Scout homepage. I have learned a little more about Python since the last attempt. Using Fiddler I quickly discovered the data was being passed using JSON. Perfect, if I could figure out the user state problem then I could easily parse the JSON feed.
Some quick research led me to the requests module for Python. BOOM game changer. Again using Fiddler I was able to come up with a HTTP header to request the JSON feed and I was off to the races.
Here is my new Python script:
I am using the requests module to handle the HTTP header. Then I grab the JSON feed and parse it out. I still need to keep an eye on the JSON data being returned because I am parsing the date out to do the conversion to SQL DATETIME.
First few days of testing have been a success. A lot more successful than the previous scrape system. I am loading the data up into a SQL Server database every 5 minutes, about 500 data points per 5 minutes. My dataset is growing pretty quick so I made an archive table to move data older than 7 days to an archive table.
I put a quick web front end on the data and hooked up a Blink(1) process to give me a visual indication of traffic before I roll out for the day.
The web front will show which sensors are slower than average and color code them based on how much slower than average they are currently showing. You can also click on a link to see a map. The map is based on the lattitude and longitude of the sensor you choose. Google Maps will provide current traffic metrics for the area you selected.
The Blink notification system will go red if traffic is below a certain threshold on a couple of the speed sensors in my route home. I write out the speed to a txt file and read it using the Blink control application.
Fast forward a few months later, and a new job later, and I wanted a process to grab the speed of the speed sensors on my route from work to home. The new gig has different hours so I am fighting the traffic a little more.
I used Fiddler2 to investigate how the data was being populated on the KC Scout homepage. I have learned a little more about Python since the last attempt. Using Fiddler I quickly discovered the data was being passed using JSON. Perfect, if I could figure out the user state problem then I could easily parse the JSON feed.
Some quick research led me to the requests module for Python. BOOM game changer. Again using Fiddler I was able to come up with a HTTP header to request the JSON feed and I was off to the races.
Here is my new Python script:
I am using the requests module to handle the HTTP header. Then I grab the JSON feed and parse it out. I still need to keep an eye on the JSON data being returned because I am parsing the date out to do the conversion to SQL DATETIME.
First few days of testing have been a success. A lot more successful than the previous scrape system. I am loading the data up into a SQL Server database every 5 minutes, about 500 data points per 5 minutes. My dataset is growing pretty quick so I made an archive table to move data older than 7 days to an archive table.
I put a quick web front end on the data and hooked up a Blink(1) process to give me a visual indication of traffic before I roll out for the day.
The web front will show which sensors are slower than average and color code them based on how much slower than average they are currently showing. You can also click on a link to see a map. The map is based on the lattitude and longitude of the sensor you choose. Google Maps will provide current traffic metrics for the area you selected.
The Blink notification system will go red if traffic is below a certain threshold on a couple of the speed sensors in my route home. I write out the speed to a txt file and read it using the Blink control application.
Blink(1) Notification
