vendredi 17 juin 2016

Scraping a table from webpage with Python

from bs4 import BeautifulSoup
from urllib import urlopen
import csv

player_code = open("/Users/bd/Desktop/testfolder/Player_Code_Test.txt").read()
player_code = player_code.split("r")


for player in player_code:

html =urlopen("https://www.capfriendly.com/players/"+player+"")

soup = BeautifulSoup(html, 'html.parser')

table = soup.findAll(class_='table_c')[0]
rows = table.findAll('tr')

first_columns = []
third_columns = []
for row in rows[1:]:
    first_columns.append(row.findAll('td')[0])
    third_columns.append(row.findAll('td')[2])

for first, third in zip(first_columns, third_columns):
    print(first.text, third.text)     

I have updated the code. It seems to work fine for one player at a time, but when I change to my text file of a list of players that where I am having trouble.

Error code: (u'2016-17', u'$2,750,000') (u'2017-18', u'$2,750,000') (u'2018-19', u'$2,750,000') Traceback (most recent call last): File "/Users/bd/Desktop/untitled-14.py", line 15, in table = soup.findAll(class_='table_c')[0] IndexError: list index out of range

-----I continue to struggle getting this code to work. It should be a simple table to scrape, but I am still having trouble. I am new to python and have searched through many len and index articles. If anyone could help me out, it would be very helpful. Thank you.

Here is a link to a sample player page : https://www.capfriendly.com/players/patrik-elias

Here is a sample of player names that I am adding from a text file to the base url.

patrik-elias

jaromir-jagr

kris-letang

patrick-kane

This is ultimately What I am wanting to do for my text file of 1000+ players

This is ultimately what I am wanting to do for my text file of 1000+ players

Aucun commentaire:

Enregistrer un commentaire