Scraping Fanduel Sportsbook For Odds With Beautiful Soup
So i'm very new to web scraping with python and I have a question regarding a project i'm working on. I'm attempting to implement a betting strategy based off of something called t
Solution 1:
So you can get the XHR rquest url by looking at the Dev Tools -> Network tab -> XHR.
It'll return a json structure (which is nested) so you'll have to parse it some way (which I did in the function). It just grabs everything, so if you want specific columns, you can just filter out what you need from the dataframes:
Just remember me when you become a millionaire ;-)...
import requests
import pandas as pd
from pandas.io.json import json_normalize
from functools import reduce
def parse_data(jsonData):
results_df = pd.DataFrame()
for alpha in jsonData['events']:
print ('Gathering %s data: %s @ %s' %(alpha['sportname'],alpha['participantname_away'],alpha['participantname_home']))
alpha_df = json_normalize(alpha).drop('markets',axis=1)
for beta in alpha['markets']:
beta_df = json_normalize(beta).drop('selections',axis=1)
beta_df.columns = [str(col) + '.markets' for col in beta_df.columns]
for theta in beta['selections']:
theta_df = json_normalize(theta)
theta_df.columns = [str(col) + '.selections' for col in theta_df.columns]
temp_df = reduce(lambda left,right: pd.merge(left,right, left_index=True, right_index=True), [alpha_df, beta_df, theta_df])
results_df = results_df.append(temp_df, sort=True).reset_index(drop=True)
return results_df
jsonData_fanduel_nfl = requests.get('https://sportsbook.fanduel.com/cache/psmg/UK/57494.3.json').json()
jsonData_fanduel_nba = requests.get('https://sportsbook.fanduel.com/cache/psmg/UK/55978.3.json').json()
jsonData_fanduel_cbb = requests.get('https://sportsbook.fanduel.com/cache/psmg/UK/53474.3.json').json()
nfl = parse_data(jsonData_fanduel_nfl)
nba = parse_data(jsonData_fanduel_nba)
cbb = parse_data(jsonData_fanduel_cbb)
Output:
print (nba.head(10).to_string())
BB.markets awaypitchername awayscoreline big3markettype.markets competitorcode.selections competitornumber.selections csvavailablebettypes.markets csvpricetypes.markets ctag.markets currentewplaceterms.markets currentewreduction.markets currenthandicap.selections currentmatchhandicap.markets currentmatchhandicap.selections currentpricedown.selections currentpriceup.selections currentscore.selections derivativetypes.markets estimatepricedown.selections estimatepriceup.selections eventlistcode eventname eventname.markets ewfactor.selections externaldescription fosportinternalorder.markets gameperiod.markets gameperiodname.markets hadvalue.selections homepitchername homescoreline idfobolifestate.selections idfoevent idfoevent.markets idfohadtype.markets idfomarket.markets idfomarket.selections idfomarketlinetype.markets idfomarketsubjecttype.markets idfomarkettype.markets idfomarkettypeclass.markets idfomeeting idfoparticipant.selections idfoparticipant_away idfoparticipant_home idfoselection.selections idfoselectionsuspensiontype.selections idfosport idfosporttype.markets idfosporttype.selections idfotemplateselection.selections internalorder internalorder.markets internalorder.selections iscurrentlistoffer.selections islive ismainoutright.markets ismatch ismatchhandicaptype.markets isoutright.markets israce istradable istradable.markets istrapbettingoptionon.markets isunderover.markets jockeyname.selections jockeysilks.selections listcode.selections listsubcode.selections lowerband.selections marketinternaldescription.markets mtag.markets name.markets name.selections nontradableon.selections noofmarkets participantname_away participantname_home pitcherhacode.markets price.selections resultpositionfinal.selections selectionhashcode.selections selexternaldescription.selections shortname.selections sportname sppricedown.selections sppriceup.selections teletextname.selections tsstart tsstart.markets ttagleft.markets ttagright.markets upperband.selections
0 False None None TP None None A,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8... CP,SGM None 1.0 1.0 215.5 215.5 215.5 23.0 20.0 None HP,O None None 1771|1772 Detroit Pistons At Cleveland Cavaliers Detroit Pistons At Cleveland Cavaliers None Detroit Pistons At Cleveland Cavaliers 3 1+ Game Lines U None None N 795248.3 795248.3 OU 30740246.3 30740246.3 TOTAL None 45773.1 4778.1 None None 1912153.3 1912141.3 106258137.3 None BASKETBALL BASKETBALL BASKETBALL None 10 5002 20 None True False True True False False True True False True None None None None None None E795248.3|SCORE Total Points (half-point line) Under None 287 Detroit Pistons Cleveland Cavaliers None 1.869565 None 29505 None Under NBA None None Under 2020-01-07T19:10:00 2020-01-07T19:10:00 None None None
1 False None None TP None None A,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8... CP,SGM None 1.0 1.0 215.5 215.5 215.5 21.0 20.0 None HP,O None None 1771|1772 Detroit Pistons At Cleveland Cavaliers Detroit Pistons At Cleveland Cavaliers None Detroit Pistons At Cleveland Cavaliers 3 1+ Game Lines O None None N 795248.3 795248.3 OU 30740246.3 30740246.3 TOTAL None 45773.1 4778.1 None None 1912153.3 1912141.3 106258175.3 None BASKETBALL BASKETBALL BASKETBALL None 10 5002 10 None True False True True False False True True False True None None None None None None E795248.3|SCORE Total Points (half-point line) Over None 287 Detroit Pistons Cleveland Cavaliers None 1.952381 None 6650 None Over NBA None None Over 2020-01-07T19:10:00 2020-01-07T19:10:00 None None None
2 False None None PS None None A,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8... CP None 1.0 1.0 3 3 3 11.0 10.0 None H,HP None None 1771|1772 Detroit Pistons At Cleveland Cavaliers Detroit Pistons At Cleveland Cavaliers None Detroit Pistons At Cleveland Cavaliers 3 1+ Game Lines H None None N 795248.3 795248.3 HA 30740333.3 30740333.3 SIDE None 45778.1 4777.1 None 1.92111e+06 1912153.3 1912141.3 106259029.3 None BASKETBALL BASKETBALL BASKETBALL None 10 2 1010 None True False True True False False True True False False None None None None None None E795248.3|SCORE Spread Betting Cleveland Cavaliers None 287 Detroit Pistons Cleveland Cavaliers None 1.909091 None 47283 None Cleveland NBA None None Cleveland Cavaliers 2020-01-07T19:10:00 2020-01-07T19:10:00 None None None
3 False None None PS None None A,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8... CP None 1.0 1.0 3 3 3 11.0 10.0 None H,HP None None 1771|1772 Detroit Pistons At Cleveland Cavaliers Detroit Pistons At Cleveland Cavaliers None Detroit Pistons At Cleveland Cavaliers 3 1+ Game Lines A None None N 795248.3 795248.3 HA 30740333.3 30740333.3 SIDE None 45778.1 4777.1 None 1.92113e+06 1912153.3 1912141.3 106259067.3 None BASKETBALL BASKETBALL BASKETBALL None 10 2 10 None True False True True False False True True False False None None None None None None E795248.3|SCORE Spread Betting Detroit Pistons None 287 Detroit Pistons Cleveland Cavaliers None 1.909091 None 987 None Detroit P NBA None None Detroit Pistons 2020-01-07T19:10:00 2020-01-07T19:10:00 None None None
4 False None None PS None None A,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8... CP,SGM None 1.0 1.0 3.5 3.5 3.5 23.0 20.0 None H,HP None None 1771|1772 Detroit Pistons At Cleveland Cavaliers Detroit Pistons At Cleveland Cavaliers None Detroit Pistons At Cleveland Cavaliers 3 1+ Game Lines H None None N 795248.3 795248.3 HA 30740357.3 30740357.3 SIDE None 45778.1 4777.1 None 1.92111e+06 1912153.3 1912141.3 106259194.3 None BASKETBALL BASKETBALL BASKETBALL None 10 5001 1010 None True False True True False False True True False False None None None None None None E795248.3|SCORE Spread (half-point line) Cleveland Cavaliers None 287 Detroit Pistons Cleveland Cavaliers None 1.869565 None 44903 None Cleveland NBA None None Cleveland Cavaliers 2020-01-07T19:10:00 2020-01-07T19:10:00 None None None
5 False None None PS None None A,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8... CP,SGM None 1.0 1.0 3.5 3.5 3.5 21.0 20.0 None H,HP None None 1771|1772 Detroit Pistons At Cleveland Cavaliers Detroit Pistons At Cleveland Cavaliers None Detroit Pistons At Cleveland Cavaliers 3 1+ Game Lines A None None N 795248.3 795248.3 HA 30740357.3 30740357.3 SIDE None 45778.1 4777.1 None 1.92113e+06 1912153.3 1912141.3 106259237.3 None BASKETBALL BASKETBALL BASKETBALL None 10 5001 10 None True False True True False False True True False False None None None None None None E795248.3|SCORE Spread (half-point line) Detroit Pistons None 287 Detroit Pistons Cleveland Cavaliers None 1.952381 None 42219 None Detroit P NBA None None Detroit Pistons 2020-01-07T19:10:00 2020-01-07T19:10:00 None None None
6 True None None ML None None A,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8... CP,SGM None 1.0 1.0 None None None 25.0 33.0 None None None None 1771|1772 Detroit Pistons At Cleveland Cavaliers Detroit Pistons At Cleveland Cavaliers None Detroit Pistons At Cleveland Cavaliers 3 1+ Game Lines H None None N 795248.3 795248.3 HA 30740385.3 30740385.3 MONEY None 4937.1 3842.1 None 1.92111e+06 1912153.3 1912141.3 106259265.3 None BASKETBALL BASKETBALL BASKETBALL None 10 1 1010 None True False True False False False True True False False None None None None None None E795248.3|SCORE Moneyline Cleveland Cavaliers None 287 Detroit Pistons Cleveland Cavaliers None 2.320000 None 23349 None Cleveland NBA None None Cleveland Cavaliers 2020-01-07T19:10:00 2020-01-07T19:10:00 None None None
7 True None None ML None None A,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8... CP,SGM None 1.0 1.0 None None None 39.0 25.0 None None None None 1771|1772 Detroit Pistons At Cleveland Cavaliers Detroit Pistons At Cleveland Cavaliers None Detroit Pistons At Cleveland Cavaliers 3 1+ Game Lines A None None N 795248.3 795248.3 HA 30740385.3 30740385.3 MONEY None 4937.1 3842.1 None 1.92113e+06 1912153.3 1912141.3 106259271.3 None BASKETBALL BASKETBALL BASKETBALL None 10 1 10 None True False True False False False True True False False None None None None None None E795248.3|SCORE Moneyline Detroit Pistons None 287 Detroit Pistons Cleveland Cavaliers None 1.641026 None 16669 None Detroit P NBA None None Detroit Pistons 2020-01-07T19:10:00 2020-01-07T19:10:00 None None None
8 False None None PS None None A,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8... CP None 1.0 1.0 1.5 1.5 1.5 28.0 25.0 None None None None 1771|1772 Detroit Pistons At Cleveland Cavaliers Detroit Pistons At Cleveland Cavaliers None Detroit Pistons At Cleveland Cavaliers 3 1-2 1st Half H None None N 795248.3 795248.3 HA 30740405.3 30740405.3 SIDE None 47598.1 5303.1 None None 1912153.3 1912141.3 106259531.3 None BASKETBALL BASKETBALL BASKETBALL None 10 202 1010 None True False True True False False True True False False None None None None None None E795248.3|SCORE 1st Half Spread Cleveland Cavaliers None 287 Detroit Pistons Cleveland Cavaliers None 1.892857 None 30949 None Cleveland NBA None None Cleveland Cavaliers 2020-01-07T19:10:00 2020-01-07T19:10:00 None None None
9 False None None PS None None A,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8... CP None 1.0 1.0 1.5 1.5 1.5 27.0 25.0 None None None None 1771|1772 Detroit Pistons At Cleveland Cavaliers Detroit Pistons At Cleveland Cavaliers None Detroit Pistons At Cleveland Cavaliers 3 1-2 1st Half A None None N 795248.3 795248.3 HA 30740405.3 30740405.3 SIDE None 47598.1 5303.1 None None 1912153.3 1912141.3 106259532.3 None BASKETBALL BASKETBALL BASKETBALL None 10 202 10 None True False True True False False True True False False None None None None None None E795248.3|SCORE 1st Half Spread Detroit Pistons None 287 Detroit Pistons Cleveland Cavaliers None 1.925926 None 13664 None Detroit P NBA None None Detroit Pistons 2020-01-07T19:10:00 2020-01-07T19:10:00 None None None
Post a Comment for "Scraping Fanduel Sportsbook For Odds With Beautiful Soup"