Monday, December 22, 2014

Yahoo Fantasy Football API Using Python

UPDATE: the YQL module can no longer be installed directly using PIP. You have to install it from pip using the GitHub repository. Check out this Stack Overflow post on how to get er done: http://stackoverflow.com/questions/15268953/how-to-install-python-package-from-github


Game Changer...after about 8 hours of trying to figure out how to use Yahoo's Fantasy Football API I finally made some huge progress. I was able to use Yahoo's 3-legged OAuth method to authenticate to my Yahoo Fantasy Football league. From there it was just a matter of figuring out how to traverse the various JSON responses in order to consume the data.

This post assumes the following:

  • You have Python 2.7 installed
  • You know how to add modules to Python using easy_install or pip
  • You have some general knowledge of moving around in Python
Holler if you have any problems or if you have a specific question.


Authentication and Sample Scripts


OAuth Steps

I had to setup a few things before getting started. I am hitting the yahoo API using YQL which is Yahoo's Query Language. You can try out the YQL Console here. I used the console often to test various queries. There are several samples of queries hitting various APIs.

YQL Module Oauth Update 12/1/2015

The YQL Python module I was using is no longer maintained. The dev does have it out on GitHub for the time being. I had some SSL certificate trouble trying to install the module using Python 2.7.10. Therefore I just downloaded the ZIP from GitHub and then used pip to install the module from the local repository using this command:

(Make sure you are in the root of the repo folder)
pip install -e .

Make sure you include that period on the end of the command

From there I was able to continue with the Yahoo OAuth Process.


You will need to create a project to get a API Consumer key and Consume Secret. Once you login to Yahoo Developer network you can go to the projects page: https://developer.apps.yahoo.com/projects. Here you can create a new app.

Sample Python code for Authentication


You setup a simple cache directory where your authentication token will be stored. If your token is expired the script will request another one. On the initial execution of this script you will need to allow your new application to access your yahoo account data.

When you run the script in Python you will be prompted to visit a URL. This URL will then spit out a code. You take the code and paste it back in your Python Console. This will setup a trust with your new application. You only have to do this to establish the token. Subsequent calls should work fine.

From there you start writing your YQL queries.



Now its just a matter of looping through the JSON response and grabbing the data you need. Here is a full Python Script example.


Data

It took a while for me to figure out how to traverse the various JSON responses from the API. Once you get in there and mess around for a bit it will get a little easier. I targeted the following data sets of interest to me. There are other data sets available. You can use the YQL console to browse around and grab what you want.

Teams

Using the teams API I was able to capture all the teams in the league. The teams are given a unique team_key that will be used later when we go to find rosters and such. 
  • Team_key: unique ID for each team 
  • TeamName: name of the team given by the manager of the team
  • Division: Division ID the team belongs to.
  • Number_of_Moves: Number of player adds/drops
  • Number_of_Trades: Number of player trades
  • Manager_Nick: Nick Name of the manager in their yahoo account
  • Manager_Email: Email address of the manager
Stat Settings

The Stat Settings table is used to store information about the various scoring categories. Each stat has a unique stat_id. I used the LeagueSettings API to grab this data.

  • Stat_ID: Unique ID identifing each stat
  • Enabled: bit value indicating if the stat is enabled
  • stat_name: name of the stat (Example: Passing yards)
  • Stat_modifier: How much the stat is worth. (Example: In our league a passing yard is worth 0.05 points)
I need to figure out how to get the bonus settings. I may just hard code that since I was not having much luck grabbing that from the API.

Matchups

The matchups text file shows the result of each week's head to head matchup. I stored the Team ID to enable me to join that to the team table detailed above.


  • Week Number: Week of the matchup
  • Matchup ID: I number each matchup for SELECT purposes. There are 2 teams for every matchup ID.
  • Team ID: unique team identifier
  • Team Points: Number of points the team scored
  • Team Projected Points: Number of points the team was projected to score.
  • Team Key: Another Unique team identifier.
Roster Stats

For each team's roster stats I execute a terribly inefficent loop where I grab a team id, then hit the API using a WEEK IN (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17). I grab the stats for every time for each week and dump in into the text file. There are two outputs from the rosterStats.py script.

  • Roster
    • Team ID: Unique Team Identifier
    • Fname: Player First Name
    • Lname: Player Last Name
    • Team: Player's Team name (Ex: Kansas City Chiefs)
    • Player ID: Unqiue player identifier
    • Player Key: Unqiue player Identifier
    • Week Number: Season Week Number
    • Total Points: Points scored by player based on our league settings
    • Position: Roster position
  • Roster stats
    • Player ID: Unique player identifier
    • Week Num: Season Week Number
    • Player Key: Unique Player identifier
    • Stat ID: Stat identifier (Ex: Passing Yards)
    • Stat Value: Value of the stat (Ex. 314)

Now I just have to wait until tomorrow when the season is complete. I will do one last data grab and then the real fun can start. I plan to start analyzing the data. I am especially interested in draft analysis to determine the impact of various drafted players on the season, rate how well each manager drafted, etc. I'd also like to play with various scoring scenarios to see how that effects league scoring.

Our league is unique in the fact that we have a 2QB league and the QBs are rated higher due to our 6 point per passing TD setting. In my opinion that represents the NFL better than a standard scoring league since in the NFL good QBs are vital to most successful teams.


20 comments:

  1. Joe,

    Can you briefly explain which ydl library you are using? I don't find a standard 'ydl' with any of the methods you use. Thanks

    ReplyDelete
    Replies
    1. Not sure I completely understand your question. I assume when you say YDL you are referring to the Yahoo Dev Library? If not let me know. Anyway, to start I went here: https://developer.yahoo.com/yql/console/ to the normal Yahoo Query Language Console. I expanded Fantasy sports on the left side of the screen in the console and looked at the JSON responses from the various requests. I started plugging in the keys from my yahoo league.

      Then using the YQL module in python I started hitting the API to grab the data. Not sure this helps you or not. Let me know if this doesn't help.

      Delete
    2. Sorry, that was a typo as I meant *yql

      I can query fine using the YQL console. In python however, I cannot find a yql module. When I 'pip search yql' the only options are
      -yql finance
      -myql
      -baas
      -myql-cli
      -pyql-weather
      YQLStock

      I tried 'myql' , which is described as the 'Python Wrapper for the Yahoo Query Language, but none of the methods you used exist such as 'ThreeLegged'. Does this make sense?

      Delete
    3. Looks like the module I was using is no longer available. I did find the github page for the module but not sure that will help you much: https://github.com/project-fondue/python-yql

      I noticed the myql module you referenced and found an Oauth example on their site here: http://myql.readthedocs.org/en/latest/oauth/

      May be worth a shot. I will give it a go with myql and update the article if I get it working. Sorry for the hassle.

      Delete
    4. Maybe you can use pip to install yql from github. Example here: http://stackoverflow.com/questions/15268953/how-to-install-python-package-from-github

      Delete
    5. Hello @JoeStory,
      With myql you can check outhttps://github.com/josuebrunel/myql/blob/master/tests/tests.py#L175-188.
      Hope it will help.

      Delete
  2. Thanks Joe! Got everything working in yql using the install through GitHub!

    ReplyDelete
    Replies
    1. Awesome news Jeff. Curious as to what you are cooking up with the data. I was using our data to rate our draft, biggest draft busts, biggest draft steal, etc. Was wanting to build some kind of comprehensive draft tool for our league. I already started grabbing all the various draft rankings from the sites and combining them into one spot. Trying to streamline it all, lots of data to process and such.

      Delete
    2. Hey Joe. Tried to respond awhile ago but I guess it didn't post correctly. Just set up basic charts on career statistics and records etc for my leagues. Set up a few more that analyze points scored and place finished based on draft slot. Also set up an automated text messaging program to text everybody in the league on Sundays to notify them of their matchup, their career history with their opponent, and the historical odds of winning (in my leagues) according to Yahoo's Projections. I am doing a few other things, and doing front end work too so I've been working slowly. My next ideas are to get into more of the data it seems you have previously scraped. Just from my initial look it seems like the way yahoo returns the player data, analysis of rosters and statistics of players linked to their manager could be quite cumbersome. Definietly let me know if you have any tips or want to share some code to help each other out.

      Delete
    3. Roger that Jeff. Sounds pretty cool. I like the automated message to setup your lineups. The odds stuff you are messing with sounds awesome. My main target was draft stats. So like the best drafted player, based upon total points, for each team, by position, etc.

      Here is an example screen shot at the team level: http://i.imgur.com/D7GQbhY.png

      Here is a screenshot of the best/worst players drafted by position: http://i.imgur.com/rsMPiSd.png

      I had some other plans but I get bored easy. I also am working on something I call draftView. I get tired of bouncing from website to website to get the projections when I am in the middle of a draft or mock draft. So I am using python to go out and scrape all the rankings into a database and then I wrote a front end on it to allow me to search. So I can filter by position, player, draft pick number, etc. Puts all the information I need at my fingertips.

      Here is a current screenshot of draftview: http://i.imgur.com/dKLEBKH.png

      I am still working on data sources to parse. As you know ESPN changed their site design up recently and their Analyst FFootball rankings sites are all different now so I have to update my scrape stuff. Not that their rankings are that good anyway, just nice to have side by side.

      Delete
  3. This is a really nice sample on how to get the roster of yahoo fantasy football team.
    However, I am confused by he way you retrieve teamkey. In your sample, you grab it from your sql server, but how did you get the key from the first place? Is there any python api available to get teamkey and leaguekey?

    ReplyDelete
    Replies
    1. See my reply below to Craig. If you have any problems let me know.

      Delete
  4. Is there a way to programatically get a list of the leagues I am in? It seems like I need to know the league ID ahead of time.

    ReplyDelete
    Replies
    1. Did you ever solve this? It appears their api doesn't provide this. Didn't know if there was a workaround.

      Delete
  5. Craig Hop on over to the YQL console and run something like this:

    select * from fantasysports.games where use_login=1 and game_key in ('nfl')

    In the response you should see your game key. From there you can query your league by doing something like this:

    select * from fantasysports.leagues where league_key='238.l.627060'

    Replace the 238 with your game key you found in the first query. That last number is your league id which you can find if you go to your league main home page. The league id will be in the URL.

    Let me know if that doesn't work or if you have trouble.

    ReplyDelete
  6. This is awesome, it helped a lot! If I ever get to writing down what I ended up making, I'll send you a link!

    ReplyDelete
  7. NICE!!

    I USE IN http://www.cricbell.com/football/

    ReplyDelete
  8. After a long time I managed to install the correct yql library through git after installing Python2.7 since yql is not valid with Python3.4.

    However when I run you Code I get this Error:

    httplib2.CertificateHostnameMismatch: Server presented certificate that does not match host api.login.yahoo.com: {'notAfter': 'Sep 2 23:59:59 2017 GMT', 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'California'),), (('localityName', u'Sunnyvale'),), (('organizationName', u'Yahoo Inc.'),), (('organizationalUnitName', u'Information Technology'),), (('commonName', u'login.yahoo.com'),))}

    Even though I have created my App and entered my Consumer Key and Consumer Secret.
    Any idea what my problem could be?

    Could my issue be that I put www.google.ca for my callback domain? What did you pick for your application type? Web Application or Installed Application?
    Or something like I can't access the API from Canada?

    ReplyDelete
  9. Hi All. Just wanted to report that pip install via github does not appear to work any longer.

    From what I can gather - it's an issue that probably started October 2017 - related to pypi now requiring https. It will not allow http.

    I was able to install the package, by downloading locally and changing the DEFAULT_URL in distribute_setup.py from http: to https:

    after that running the install locally with pip install -e. worked perfectly.

    ReplyDelete