Friday, February 27, 2015

LCD Finally

thejoestory 2/27/2015 02:01:00 PM No comments

After days of troubleshooting, resoldering, re-coding and such I was finally able to get my LCD to work with the Arduino. I had a few choice words for the LCD right before I took this picture.

I guess the important thing is that I got it to work finally. I found a data sheet on my off brand cheap LCD. Turns out I had the pins reversed so all the hookups were backwards. Once I fixed that boom it worked first time. Amazing. I had spent time resoldering my header pins to the LCD because this was my second solder job. Oh well. It finally works.

If anyone else bought the QC2004 LCD off Amazon see here for the data sheet:

http://forum.classic-computing.de/index.php?page=Attachment&attachmentID=6039

UPDATE:
Added a temp sensor and sent the output to LCD with a few more lines of code.

Thursday, February 19, 2015

Missouri School Enrollment Numbers

thejoestory 2/19/2015 10:20:00 AM No comments

I somehow stumbled on this page in a search for education data: http://mcds.dese.mo.gov/quickfacts/Pages/District-and-School-Information.aspx. One click led to another and now I have Missouri Enrollment education data from 1991 - 2015 in a database for consumption. This is more of a CanvasJS chart practice exercise than an actual deep dive into MO Education enrollment trends. However quickly you can see a few interesting things from a surface level view of the data.

Kansas City 33

The Kansas City School district lost accreditation September 20, 2011 effective January 2012. However from the chart below you can see a definite downward trend in enrollment prior to 2011. In November 2007 KC transferred seven schools to the Independence school district. You can see a sizable dip between the 2008 and 2009 school year.

Here are the numbers from Kansas City for 2007 - 2010 showing the large drop in enrollment pretty much across the board.

2008 Recession Theory

Several district show a dip in Kindergarten enrollment between the 2014 and 2015 school year. My first theory is the 2008-2009 recession. People slowed down on family expansion during that year which would start to show up in Kindergarten enrollment numbers. Here is an example from the Liberty School District. It shows a dip starting in 2011 and bottoming out in 2015. The next few years trend will be interesting to see in Liberty as the economy tries to rebound, more houses are being built, and people continue expanding their family.

Largest Jumps

Using some cool SQL Server Windowing Functions I was able to quickly sort out the districts that had the largest enrollment drops and gains in a single year. I filtered out only those schools with at least 100 students in the grade the previous year. Several rural schools have dramatic year to year increases and decreases. The 100 student filter eliminated some of the noise.

Top 20 K Decreases

district_name	PrevYear	CurrentYear	year	DiffStudents	PctChange
ETHEL HEDGEMAN LYLE ACADEMY	104	0	2013	-104	-100.00%
MEADOW HEIGHTS R-II	108	36	1994	-72	-66.67%
AFFTON 101	369	173	1995	-196	-53.12%
CHILLICOTHE R-II	294	148	1992	-146	-49.66%
HICKMAN MILLS C-1	1112	583	1993	-529	-47.57%
BRENTWOOD	107	57	1994	-50	-46.73%
RITENOUR	1027	565	1992	-462	-44.99%
CENTER 58	310	175	1997	-135	-43.55%
ROCKWOOD R-VI	2179	1251	1993	-928	-42.59%
STRAFFORD R-VI	113	65	2006	-48	-42.48%
FORSYTH R-III	100	59	1992	-41	-41.00%
NORMANDY	776	458	1993	-318	-40.98%
SENECA R-VII	122	74	2015	-48	-39.34%
ETHEL HEDGEMAN LYLE ACADEMY	112	68	2009	-44	-39.29%
ARCADIA VALLEY R-II	113	69	2000	-44	-38.94%
SPECL. SCH. DST. ST. LOUIS CO.	114	72	2009	-42	-36.84%
ST. CHARLES CO. R-V	131	83	1994	-48	-36.64%
PATTONVILLE R-III	740	470	1993	-270	-36.49%
WEBSTER GROVES	546	348	1993	-198	-36.26%
LICKING R-VIII	100	64	2014	-36	-36.00%

Top 20 K Increases

district_name	PrevYear	CurrentYear	year	DiffStudents	PctChange
PATTONVILLE R-III	246	740	1992	494	200.81%
HICKMAN MILLS C-1	563	1112	1992	549	97.51%
CENTER 58	162	310	1996	148	91.36%
KIRKWOOD R-VII	289	550	1994	261	90.31%
ROCKWOOD R-VI	1152	2179	1992	1027	89.15%
WARRENSBURG R-VI	115	211	1992	96	83.48%
WINDSOR C-1	120	204	1992	84	70.00%
ST LOUIS LANG IMMERSION SCHOOL	108	183	2011	75	69.44%
UNIVERSITY CITY	211	341	1992	130	61.61%
MAPLEWOOD-RICHMOND HEIGHTS	105	165	1992	60	57.14%
AFFTON 101	246	369	1994	123	50.00%
WEBSTER GROVES	373	546	1992	173	46.38%
MONETT R-I	136	196	2003	60	44.12%
JENNINGS	194	279	1996	85	43.81%
FREDERICKTOWN R-I	106	149	1995	43	40.57%
MEXICO 59	148	208	2007	60	40.54%
ST. JAMES R-I	101	138	2000	37	36.63%
CARUTHERSVILLE 18	102	139	2006	37	36.27%
AURORA R-VIII	127	173	2010	46	36.22%
KENNETT 39	136	185	1994	49	36.03%

K Findings

Looks like the ETHEL HEDGEMAN Acamedy has shutdown.
I was hoping to see more 2015/2014 year decreases to back up my 2008 recession theory.
I sent Pattonville R-III a tweet to verify the jump in Enrollment in 1992 is legit. I also asked them what happened to cause a 200% enrollment increase followed by a 36% decrease the following year.

Top 20 1st Decreases

district_name	PrevYear	CurrentYear	year	DiffStudents	PctChange
ETHEL HEDGEMAN LYLE ACADEMY	115	56	2009	-59	-51.30%
ST. CHARLES CO. R-V	142	80	1994	-62	-43.66%
BROOKFIELD R-III	101	57	2015	-44	-43.56%
MALDEN R-I	108	71	2001	-37	-34.26%
MAPLEWOOD-RICHMOND HEIGHTS	124	82	1999	-42	-33.87%
SPECL. SCH. DST. ST. LOUIS CO.	107	71	2015	-36	-33.64%
WARSAW R-IX	129	86	2004	-43	-33.33%
ARCADIA VALLEY R-II	107	72	2001	-35	-32.71%
MOUNTAIN GROVE R-III	163	110	2000	-53	-32.52%
CARUTHERSVILLE 18	161	110	2006	-51	-31.68%
MALDEN R-I	127	87	1994	-40	-31.50%
BAYLESS	118	81	1997	-37	-31.36%
DESOTO 73	245	170	2001	-75	-30.61%
WINFIELD R-IV	138	97	1994	-41	-29.71%
STRAFFORD R-VI	101	71	2007	-30	-29.70%
CAMERON R-I	135	95	1996	-40	-29.63%
CHARLESTON R-I	115	81	2003	-34	-29.57%
SPECL. SCH. DST. ST. LOUIS CO.	127	90	2009	-37	-29.13%
SULLIVAN C-2	179	127	2000	-52	-29.05%
ARCADIA VALLEY R-II	101	72	1995	-29	-28.71%

Top 20 1st Grade Increases

district_name	PrevYear	CurrentYear	year	DiffStudents	PctChange
ST LOUIS LANG IMMERSION SCHOOL	113	174	2012	61	53.98%
LOGAN-ROGERSVILLE R-VIII	116	174	2009	58	50.00%
INDEPENDENCE 30	823	1143	2009	320	38.88%
ODESSA R-VII	134	184	2004	50	37.31%
JENNINGS	201	270	2015	69	34.33%
MEXICO 59	150	201	2008	51	34.00%
DUNKLIN R-V	112	149	1992	37	33.04%
RICHMOND R-XVI	112	149	2004	37	33.04%
ST. CLAIR R-XIII	132	175	2005	43	32.58%
MOUNTAIN GROVE R-III	123	163	1999	40	32.52%
ST. CLAIR R-XIII	130	172	2014	42	32.31%
WILLARD R-II	230	304	2004	74	32.17%
WEBB CITY R-VII	269	352	2005	83	30.86%
SMITHVILLE R-II	133	174	2005	41	30.83%
FREDERICKTOWN R-I	124	162	2014	38	30.65%
DESOTO 73	170	222	2002	52	30.59%
HANCOCK PLACE	103	134	2015	31	30.10%
SMITHVILLE R-II	123	160	2003	37	30.08%
KNOB NOSTER R-VIII	126	163	2009	37	29.37%
PERRY CO. 32	122	157	1996	35	28.69%

First Grade Findings

DESOTO 73 shows up in both largest increase and largest decrease lists. 2001 saw a 30.61% decrease in 2001 followed by a 30.59% increase in 2002. Desoto is a kinda small (population 6400 during 2010 census). I searched for a few minutes about what happened in Desoto MO in 2001 with 0 results.
ST. LOUIS LANGUAGE IMMERSION school had a big boost. This is a new charter school in the St. Louis area. Given the state of large metro schools in KC and St. Louis, no surprises here people wanting to try out alternatives.
INDEPENDENCE 30 shows up on the biggest increase list in 2009. This was after the previously mentioned building acquisition from the KC school district.

So I only showed the full results for K and 1st Grade cause I started getting bored with that. I did however want to show the SQL Server Windowing functions I used to get the data. Those are pretty cool.

One last cool thing

While looking at the 5th grade trends (as mentioned I started getting bored with the year to year) I did notice something in the Clinton district. In 1994 Clinton had a 32% decrease in 5th grade enrollment. I wanted to see if you could see that trend before and after the drop in 5th grade. Using the graph I compared all the grades to 5th grade for Clinton. You can see the previous years drop in enrollment leads to the 5th grade year drop as well as the subsequent years drop in enrollment. It was like a ripple effect, cool to see visually.

K vs 5th

1st vs 5th

2nd vs 5th

3rd vs 5th

4th vs 5th

5th vs 6th

5th vs 7th

5th vs 8th

5th vs 9th

5th vs 10th

5th vs 11th

5th vs 12th

Anyway good times. This was just one of the plethora of data sets on the MO DOE web site. Too much data, too little time.

Friday, February 13, 2015

Google Spreadsheet Python

thejoestory 2/13/2015 07:53:00 AM No comments

Just found an awesome python module this morning called gspread. The module allows you to work with Google Spreadsheets. You can read data, write data, work with worksheets, and all kinds of other functions. You can find out more about the module on GitHub

I went from install, to sample code, to hacking spreadsheets in about 35 seconds. It was so easy.

Easy_Install the Module

Open up a console and browse to my Python\scripts folder
Run easy_install (setup instructions for easy_Install) like so: easy_install gspread
Easy Install does all the heavy lifting.

Create a Spreadsheet

Create a spreadsheet in Google Drive and call it whatever you want. Remember the name because you will need it for your sample script.

Sample Script

You will see some sample scripts on the GitHub project page. Here is my script.

It really is that simple. You just setup your username and password. The module will then will convert that to an Oauth session for you to authenticate to Google securely. We open the spreadsheet with the gc.open("Pythontest").sheet1 command.

You can see the update command is commented out: wks.update_acell('B2',"some data here"). You can fetch a range of cells and print out the values. You can retrieve whole columns or rows.

This module is awesome and so easy to use.

Monday, February 9, 2015

Starburst Superfruit Flavor Review

thejoestory 2/09/2015 12:20:00 PM No comments

Co-worker just brought in some Starburst Superfruit chews to sample. Here is how they stack up.

Strawberry Starfruit: Fortunately Wrigley did not jack with the Strawberry flavor we all know and love too much. This pretty much tastes like a strawberry Starburst with a hint of something weird. I'd assume the weird part is the star fruit part. Anyway, they didn't hose it up too much and this is by far the best flavor in the pack. 8/10
Blueberry Acai: I don't even know what an Acai is. At first bite this flavor was third in the standings. However it finished strong at the end and powered ahead of the others. Pretty tasty representation of blueberry, and I'd have to let some hipster weigh in on the Acai part cause I have no clue. 5/10
Raspberry Pomegranate: This flavor tasted like watermelon bubble gum at first until the pomegranate taste overpowered the flavor near the middle of the tasting. This was a very close race between the Blueberry flavor but again the Blueberry flavor finished strong. 4.5/10
Passion fruit punch: I knew I would hate this flavor before trying it. Any candy with the word fruit punch in the title is terrible. Hawaiian C in the liquid form is terrible enough. How can you make it worse? Put it in a chewable format, of course. This one is terrible and should be thrown out like all Orange Flavored regular Starbursts. 1/10

Here is a picture with the flavors ranked in order from best (top) to throw it out (bottom).

Forecast Tracker

thejoestory 2/09/2015 11:30:00 AM No comments

Complex Forecast. New data is in. Not really sure what is going to happen. We'll have to wait and see what the weekend holds. Could be anything from a dusting to 6" of snow.

These are all quotes I have heard while watching our local Kansas City news. Let's face it. Weather teams have the daunting task of predicting the future and are crucified when they get it wrong. Don't feel too bad for them though, they are typically paid pretty well for being wrong a lot. It is a very tough job and they do get it right more often than not. Weather teams in Kansas City seem to have an even tougher job given the setup of Kansas City weather.

The Plan

Measure the accuracy of the local weather stations forecasts to see who is the most accurate. I wanted the process to be hands off, completely automated. I didn't want to enter the data manually etc. I was hoping the local weather stations had some kind of parsable forecast feed on their websites I could use for the data source.

I then decided to compare the forecasts with the observed temperature at the Kansas City International airport. For some reason I decided the time to compare the temps would be 12:00PM.

The Method

I started reviewing the Hourly Forecast for each local weather website. Three of the four weather stations in town had the data in a text format I could scrape from their site. The fourth weather website had the hourly forecast in an image format. I am not skilled enough to figure out how to work with an image so I just through Fox4 out of the running.

I used python to do the heavy lifting. Here are some sample scripts for each site.

KMBC9

KCTV5

KSHB41

I then used Weather Underground to get the temp at 12:00PM at KCI. The Weather Underground temperature is collected at 11:53AM. For my purposes here that is close enough. I could not find on the various weather team websites what location their hourly forecast is predicting. Again this is not an exact calculation of accuracy here...just a general ballpark. If your 12:00PM hourly forecast is off by 8 degrees at KCI, odds are the forecast was incorrect at other spots in the metro.

Wunderground

You can tell by the scripts above that this system is pretty fragile. Any small change in website design, CSS changes, object name changes, etc. will throw this off. I am hoping the thing at least works for a few weeks so I can collect some data.

The python script executes at 11:58AM in order to ensure that the 11:53 Wunderground temperature collection is complete, and before the weather websites remove the 12:00PM hourly forecast from their web sites.

Then at 12:15PM I have a job that executes to measure the accuracy between the forecasted temperature and observed temperature. The results are stored in a table.

Results

I then front-ended the data with a little website used to display the results. Keep in mind that I only have 3 days of data so far, so this is a very limited sample size. Again, I hope the system can work for a few weeks before it completely blows up.

Three Day Desk Temp Study

thejoestory 2/09/2015 09:07:00 AM No comments

The office at my current gig runs a little warm. This is not me complaining. I like wearing short sleeves in the dead of winter and it can always be worse. I would much rather the office run hot than cold. Cold is terrible and miserable. I brought my trusty Acurite Digital Thermometer to work a while back during my Fridge Cooling Zones project, so I thought why not track the temperature at my desk for a few days.

Temperature Collection Technique

A while back I had created a PowerShell Script Reminder System. You set the number of minutes you want to delay the timer and run the script. After the amount of time you entered elapsed, a pop up box would display in the center of your screen. Low tech I know but it seemed to work well.

Here is an example of a 15 minute timer.

param([string]$time = "00:15:00")
$timeReformat = $time -replace ",","."
$seconds = ([TimeSpan]::Parse($timeReformat)).TotalSeconds
$seconds

Start-Sleep -s $seconds
Add-Type -AssemblyName System.Windows.Forms

$Form = New-Object system.Windows.Forms.Form
$Form.Text = "FINAL COUNTDOWN"
$Font = New-Object System.Drawing.Font("Calibri",24)
$Form.Font = $Font
$Label = New-Object System.Windows.Forms.Label
$Label.Text = "REMINDER REMINDER REMINDER"
$Label.AutoSize = $True
$Form.Width = 400
$Form.Height = 200
$Form.MinimizeBox = $False
$Form.AutoSize = $True
$Form.MaximizeBox = $False
$Form.StartPosition = "CenterScreen"
$Form.Controls.Add($Label)
$Form.BackColor = "White"
$Form.ForeColor = "Blue"
$Form.TopMost = "True"
$Form.ShowDialog()

Here is the pop up message.

As I mentioned pretty low tech. A few problems with this.

I have to manually run the timer over and over again
I hardcoded the time instead of passing that into the script.
Sometimes I would forget to restart the timer after 15 minutes

I may tweak that script later.

So anyway, I would run the powershell script and then start working on something else. After 15 minutes I would get that fancy pop up box. I would then hop into a Notepad++ document where I would record the date/time (using a shortcut mapped to Crtl+F5), the temperature, and the humidity.

I would then fire up the PowerShell Script for another 15 minute interval. Again several problems with this low-tech solution. The main problem is that I would forget to restart the PowerShell Script after recording the time.

Day 1

Date: 2/3/2015
KC High Temp: 46
Recorded Office High: 79
Humidity: 16%

Raw data

Date	Temp
2/3/2015 7:34	75
2/3/2015 7:53	75
2/3/2015 8:11	77
2/3/2015 8:26	77
2/3/2015 8:42	77
2/3/2015 8:57	77
2/3/2015 9:13	77
2/3/2015 9:28	79
2/3/2015 9:44	77
2/3/2015 10:07	79
2/3/2015 10:22	79
2/3/2015 10:38	79
2/3/2015 10:53	79
2/3/2015 11:08	79
2/3/2015 11:48	77
2/3/2015 12:21	75
2/3/2015 13:01	75
2/3/2015 13:17	75
2/3/2015 13:32	77
2/3/2015 13:52	77
2/3/2015 14:07	77
2/3/2015 14:22	77
2/3/2015 14:38	77
2/3/2015 14:54	77
2/3/2015 15:09	77
2/3/2015 15:25	77
2/3/2015 15:40	77

Chart:

There was a huge dive near lunch. You could actually feel a cool breeze running through the office area. It was as if someone enabled some kind of AC cooling unit to cool the place down. Humidity remained constant at 16% for all recorded observations. Thus every time I go to grab my coat I get shocked.

Day 2

Date: 2/5/2015
KC High Temp: 28
Recorded Office High: 79
Humidity: 16%

Raw Data

Date	Temp
2/5/2015 7:33	73
2/5/2015 7:50	73
2/5/2015 8:06	73
2/5/2015 8:21	75
2/5/2015 8:36	75
2/5/2015 8:52	75
2/5/2015 9:08	75
2/5/2015 9:23	77
2/5/2015 9:39	77
2/5/2015 9:52	77
2/5/2015 10:07	77
2/5/2015 10:24	77
2/5/2015 10:44	77
2/5/2015 10:59	77
2/5/2015 11:24	77
2/5/2015 11:40	77
2/5/2015 12:12	77
2/5/2015 12:27	77
2/5/2015 12:44	77
2/5/2015 13:07	77
2/5/2015 13:21	77
2/5/2015 13:23	77
2/5/2015 13:38	77
2/5/2015 14:01	77
2/5/2015 14:19	77
2/5/2015 14:34	77
2/5/2015 14:53	79
2/5/2015 15:08	79
2/5/2015 15:24	79
2/5/2015 15:34	79
2/5/2015 15:56	79

Chart

I was a little more with it on this day in the temp collection. I collected more samples than day 1. Day 2 was a more typical trend in temperatures in the office. A gradual incline is the norm and is shown here in day 2.

Day 3

Date: 2/6/2015
KC High Temp: 54
Recorded Office High: 79
Humidity: 16%

Raw

Date	Temp
2/6/2015 6:52	73
2/6/2015 7:05	75
2/6/2015 7:21	75
2/6/2015 7:36	75
2/6/2015 7:59	77
2/6/2015 8:17	77
2/6/2015 8:33	77
2/6/2015 8:47	77
2/6/2015 9:14	77
2/6/2015 9:37	77
2/6/2015 9:57	79
2/6/2015 10:01	79
2/6/2015 10:17	79
2/6/2015 12:01	77
2/6/2015 12:16	77
2/6/2015 12:31	77
2/6/2015 12:47	77
2/6/2015 13:03	77
2/6/2015 13:28	77
2/6/2015 13:54	77
2/6/2015 14:11	77
2/6/2015 14:37	77
2/6/2015 14:53	77
2/6/2015 15:11	77
2/6/2015 15:26	77
2/6/2015 15:38	77
2/6/2015 15:53	79