mercredi 15 juin 2016

Multi-threaded URLLIB2 Request

I'm struggling to implement threading in my Hiscore Scraper for runescape.

#!/usr/bin/python

import urllib2

def get_total(username):
    try:
        req = urllib2.Request('http://services.runescape.com/m=hiscore/'
                              'index_lite.ws?player=' + username)
        res = urllib2.urlopen(req).read()
        parts = res.split(',')
        return parts[1]
    except urllib2.HTTPError, e:
        if e.code == 404:
            return "0"
    except:
        return "err"

filename = "valid.txt"

accs = []
handler = open(filename)
for entry in handler.read().split('n'):
    if "No Displayname" not in entry:
        accs.append(entry)
handler.close()

for account in accs:
    display_name = account.split(':')[len(account.split(':')) - 1]
    total = get_total(display_name)
    if "err" not in total:
        rStr = account + ' - ' + total
        handler = open('osrs_stats_tried.txt', 'a')
        handler.write(rStr + 'n')
        handler.close()
        if total != "0" and total != "49":
            handler = open('osrs_stats_f2p.txt', 'a')
            handler.write(rStr + 'n')
            handler.close()
        print rStr
    else:
        print "Error searching"
        accs.append(account)

print "Done"

Any help would be much appreciated.. I'm not entirely sure where to implement the threading code, so if it could be completed and allow me to study from that it would be great, however I understand most wouldn't wish to just give me the code but rather tell me how to implement it.

Aucun commentaire:

Enregistrer un commentaire