我有蜘蛛使用meta处理链中的请求,以产生具有来自多个请求的数据的项目.

我用来生成请求的方式是在首次调用parse函数时启动所有请求,但是,如果我有太多链接无法请求,则并不是所有请求都已计划好,最终我也无法获得所需的一切.

为了解决这个问题,我正在尝试使Spider一次请求5个产品,然后在Spider空闲时再次请求(通过在from_crawler中连接信号).

问题是,由于我的代码现在正确,spider_idle没有运行请求函数,并且蜘蛛立即关闭.好像蜘蛛不会闲置.

这是一些代码：

class ProductSpider(scrapy.Spider):

def __init__(self, *args, **kwargs):

super(ProductSpider, self).__init__(*args, **kwargs)

self.parsed_data = []

self.header = {}

f = open('file.csv', 'r')

f_data = [[x.strip()] for x in f]

count=1

first = 'smth'

for product in f_data:

if first != '':

header = product[0].split(';')

for each in range(len(header[1:])):

self.header[header[each+1]] = each+1

first = ''

else:

product = product[0].split(';')

product.append(count)

count+=1

self.parsed_data.append(product)

f.close()

@classmethod

def from_crawler(cls, crawler, *args, **kwargs):

spider = super(ProductSpider, cls).from_crawler(crawler, *args, **kwargs)

crawler.signals.connect(spider.request, signal=signals.spider_idle)

return spider

name = 'products'

allowed_domains = [domains]

handle_httpstatus_list = [400, 404, 403, 503, 504]

start_urls = [start]

def next_link(self,response):

product = response.meta['product']

there_is_next = False

for each in range(response.meta['each']+1, len(product)-1):

if product[each] != '':

there_is_next = True

yield scrapy.Request(product[each], callback=response.meta['func_dict'][each], meta={'func_dict': response.meta['func_dict'],'product':product,'each':each,'price_dict':response.meta['price_dict'], 'item':response.meta['item']}, dont_filter=True)

break

if not there_is_next:

item = response.meta['item']

item['prices'] = response.meta['price_dict']

yield item

#[...] chain parsing functions for each request

def get_products(self):

products = []

data = self.parsed_data

for each in range(5):

if data:

products.append(data.pop())

return products

def request(self):

item = Header()

item['first'] = True

item['sellers'] = self.header

yield item

func_dict = {parsing_functions_for_every_site}

products = self.get_products()

if not products:

return

for product in products:

item = Product()

price_dict = {1:product[1]}

item['name'] = product[0]

item['order'] = product[-1]

for each in range(2, len(product)-1):

if product[each] != '':

#print each, func_dict, product[each]

yield scrapy.Request(product[each], callback=func_dict[each],

meta={'func_dict': func_dict,'product':product,

'each':each,'price_dict':price_dict, 'item':item})

break

raise DontCloseSpider

def parse(self, response=None):

pass