I have one issue, when trying to test proxies with Scrapy. I want to check proxies with httpbin.org, and make crawler:
class CheckerSpider(scrapy.Spider):
name = "checker"
start_urls = (
'https://www.httpbin.org/ip'
)
connection = get_connection()
def start_requests(self):
with self.connection.cursor() as cursor:
limit = int((datetime.now() - datetime(1970, 1, 1)).total_seconds()) - 3600
q = """ SELECT *
FROM {}
WHERE active = 1 AND last_checked <= {} OR last_checked IS NULL;""".format(DB_TABLE, limit)
cursor.execute(q)
proxy_list = cursor.fetchall()
for proxy in proxy_list[:15]:
word = get_random_word()
req = scrapy.Request(self.start_urls, self.check_proxy, dont_filter=True)
req.meta['proxy'] = 'https://{}:8080'.format(proxy['ip'])
req.meta['item'] = proxy
user_pass = base64.encodestring('{}:{}'.format(PROXY_USER, PROXY_PASSWORD))
req.headers['Proxy-Authorization'] = 'Basic {}'.format(user_pass)
req.headers['User-Agent'] = get_user_agent()
yield req
def check_proxy(self, response):
print response.request.meta['proxy']
print response.meta['item']['ip']
print response.body
But when I'm testing it, I've see that Scrapy connect to url only with 5 proxies and then didn't change it. Example output (just messed IP):
2016-02-23 14:54:36 [scrapy] DEBUG: Crawled (200) <GET https://www.httpbin.org/ip> (referer: None)
https://192.168.100.130:8080
192.168.100.130
{
"origin": "192.168.100.130"
}
2016-02-23 14:54:36 [scrapy] DEBUG: Crawled (200) <GET https://www.httpbin.org/ip> (referer: None)
https://192.168.100.131:8080
192.168.100.131
{
"origin": "192.168.100.131"
}
2016-02-23 14:54:37 [scrapy] DEBUG: Crawled (200) <GET https://www.httpbin.org/ip> (referer: None)
https://192.168.100.132:8080
192.168.100.132
{
"origin": "192.168.100.132"
}
# Here Scrapy used wrong proxy to connect to site.
2016-02-23 14:54:37 [scrapy] DEBUG: Crawled (200) <GET https://www.httpbin.org/ip> (referer: None)
https://192.168.100.134:8080
192.168.100.134
{
"origin": "192.168.100.130"
}
May be I've make something wrong? Any idea? Thank you.
UPD: Actually, now I'm using a middleware, to add proxy to request. I place it in that order in middlewares:
DOWNLOADER_MIDDLEWARES = {
'checker.middlewares.ProxyCheckMiddleware': 100,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}
But I have same result. This is a my custom middleware to add proxy:
class ProxyCheckMiddleware(object):
def process_request(self, request, spider):
if 'proxy' not in request.meta:
request.meta['proxy'] = 'https://{}:8080'.format(request.meta['item']['ip'])
request.meta['handle_httpstatus_list'] = [302, 503]
user_pass = base64.encodestring('{}:{}'.format(PROXY_USER, PROXY_PASSWORD))
request.headers['Proxy-Authorization'] = 'Basic {}'.format(user_pass)
UPD. Seems like a bug in Scrapy so far. Look at the conversation here: https://github.com/scrapy/scrapy/issues/1807
Aucun commentaire:
Enregistrer un commentaire