scrapy分布式爬虫案例

关于 Redis
Redis 是目前公认的速度最快的基于内存的键值对数据库
Redis 作为临时数据的缓存区，可以充分利用内存的高速读写能力大大提高爬虫爬取效率。

关于 scrapy-redis
scrapy-redis 是为了更方便地实现 Scrapy 分布式爬取，而提供的一些以 Redis 为基础的组件。
scrapy 使用 python 自带的 collection.deque 来存放待爬取的 request。scrapy-redis 提供了一个解决方案，把 deque 换成 redis 数据库，能让多个 spider 读取同一个 redis 数据库里，解决了分布式的主要问题。

为什么要使用scrapy-redis分布式爬虫
scrapy是python界出名的一个爬虫框架。Scrapy是一个为了爬取网站数据，提取结构性数据而编写的应用框架。可以应用在包括数据挖掘，信息处理或存储历史数据等一系列的程序中。
虽然scrapy能做的事情很多，但是要做到大规模的分布式应用则捉襟见肘。
一个调度器，容器被封，慢。
有能人改变了scrapy的队列调度，将起始的网址从start_urls里分离出来，改为从redis读取，多个客户端可以同时读取同一个redis，从而实现了分布式的爬虫。

一、redis环境部署

首先需要下载redis数据库和Redis数据可的可视化工具并解压安装，将redis数据库设置为远程连接

1、解压Redis-x64-3.2.100.zip包，修改文件配置

2、master主机修改ip为0.0.0.0，运行slaver从机访问

3、关闭保护模式

4、不开启守护进程模式

5、启动redis服务

redis-server.exe redis.windows.conf
关闭redis命令：
redis-cli
shutdown

二、添加爬取的数据url至redis数据库，相当于放入redis队列

由于我们这里爬取的为淘车项目，涉及到全国的城市及城市下的汽车品牌
列：https://beijing.taoche.com/audi/
beijing 为城市缩写，audi 为汽车品牌
我们基于这种url去请求，爬取数据，实现爬取接口，将全国城市和所有汽车品牌爬取下来，分别存储到一个py文件的列表中

city.py

# 城市编码
CITY_CODE = ['shijiazhuang', 'tangshan', 'qinhuangdao', 'handan', 'xingtai', 'baoding', 'zhangjiakou','chengde', 'cangzhou', 'langfang', 'hengshui', 'taiyuan', 'datong', 'yangquan', 'changzhi', 'jincheng','shuozhou', 'jinzhong', 'yuncheng', 'xinzhou', 'linfen', 'lvliang', 'huhehaote', 'baotou', 'wuhai','chifeng', 'tongliao', 'eerduosi', 'hulunbeier', 'bayannaoer', 'wulanchabu', 'xinganmeng','xilinguolemeng', 'alashanmeng', 'changchun', 'jilin', 'hangzhou', 'ningbo', 'wenzhou', 'jiaxing','huzhou', 'shaoxing', 'jinhua', 'quzhou', 'zhoushan', 'tz', 'lishui', 'bozhou', 'chizhou', 'xuancheng','nanchang', 'jingdezhen', 'pingxiang', 'jiujiang', 'xinyu', 'yingtan', 'ganzhou', 'jian', 'yichun', 'jxfz','shangrao', 'xian', 'tongchuan', 'baoji', 'xianyang', 'weinan', 'yanan', 'hanzhong', 'yl', 'ankang','shangluo', 'lanzhou', 'jiayuguan', 'jinchang', 'baiyin', 'tianshui', 'wuwei', 'zhangye', 'pingliang','jiuquan', 'qingyang', 'dingxi', 'longnan', 'linxia', 'gannan', 'xining', 'haidongdiqu', 'haibei','huangnan', 'hainanzangzuzizhizho', 'guoluo', 'yushu', 'haixi', 'yinchuan', 'shizuishan', 'wuzhong','guyuan', 'zhongwei', 'wulumuqi', 'kelamayi', 'shihezi', 'tulufandiqu', 'hamidiqu', 'changji', 'boertala','bazhou', 'akesudiqu', 'xinjiangkezhou', 'kashidiqu', 'hetiandiqu', 'yili', 'tachengdiqu', 'aletaidiqu','xinjiangzhixiaxian', 'changsha', 'zhuzhou', 'xiangtan', 'hengyang', 'shaoyang', 'yueyang', 'changde','zhangjiajie', 'yiyang', 'chenzhou', 'yongzhou', 'huaihua', 'loudi', 'xiangxi', 'guangzhou', 'shaoguan','shenzhen', 'zhuhai', 'shantou', 'foshan', 'jiangmen', 'zhanjiang', 'maoming', 'zhaoqing', 'huizhou','meizhou', 'shanwei', 'heyuan', 'yangjiang', 'qingyuan', 'dongguan', 'zhongshan', 'chaozhou', 'jieyang','yunfu', 'nanning', 'liuzhou', 'guilin', 'wuzhou', 'beihai', 'fangchenggang', 'qinzhou', 'guigang','yulin', 'baise', 'hezhou', 'hechi', 'laibin', 'chongzuo', 'haikou', 'sanya', 'sanshashi', 'qiongbeidiqu','qiongnandiqu', 'hainanzhixiaxian', 'chengdu', 'zigong', 'panzhihua', 'luzhou', 'deyang', 'mianyang','guangyuan', 'suining', 'neijiang', 'leshan', 'nanchong', 'meishan', 'yibin', 'guangan', 'dazhou', 'yaan','bazhong', 'ziyang', 'aba', 'ganzi', 'liangshan', 'guiyang', 'liupanshui', 'zunyi', 'anshun','tongrendiqu', 'qianxinan', 'bijiediqu', 'qiandongnan', 'qiannan', 'kunming', 'qujing', 'yuxi', 'baoshan','zhaotong', 'lijiang', 'puer', 'lincang', 'chuxiong', 'honghe', 'wenshan', 'xishuangbanna', 'dali','dehong', 'nujiang', 'diqing', 'siping', 'liaoyuan', 'tonghua', 'baishan', 'songyuan', 'baicheng','yanbian', 'haerbin', 'qiqihaer', 'jixi', 'hegang', 'shuangyashan', 'daqing', 'yc', 'jiamusi', 'qitaihe','mudanjiang', 'heihe', 'suihua', 'daxinganlingdiqu', 'shanghai', 'tianjin', 'chongqing', 'nanjing', 'wuxi','xuzhou', 'changzhou', 'suzhou', 'nantong', 'lianyungang', 'huaian', 'yancheng', 'yangzhou', 'zhenjiang','taizhou', 'suqian', 'lasa', 'changdudiqu', 'shannan', 'rikazediqu', 'naqudiqu', 'alidiqu', 'linzhidiqu','hefei', 'wuhu', 'bengbu', 'huainan', 'maanshan', 'huaibei', 'tongling', 'anqing', 'huangshan', 'chuzhou','fuyang', 'sz', 'chaohu', 'luan', 'fuzhou', 'xiamen', 'putian', 'sanming', 'quanzhou', 'zhangzhou','nanping', 'longyan', 'ningde', 'jinan', 'qingdao', 'zibo', 'zaozhuang', 'dongying', 'yantai', 'weifang','jining', 'taian', 'weihai', 'rizhao', 'laiwu', 'linyi', 'dezhou', 'liaocheng', 'binzhou', 'heze','zhengzhou', 'kaifeng', 'luoyang', 'pingdingshan', 'jiyuan', 'anyang', 'hebi', 'xinxiang', 'jiaozuo','puyang', 'xuchang', 'luohe', 'sanmenxia', 'nanyang', 'shangqiu', 'xinyang', 'zhoukou', 'zhumadian','henanzhixiaxian', 'wuhan', 'huangshi', 'shiyan', 'yichang', 'xiangfan', 'ezhou', 'jingmen', 'xiaogan','jingzhou', 'huanggang', 'xianning', 'qianjiang', 'suizhou', 'xiantao', 'tianmen', 'enshi','hubeizhixiaxian', 'beijing', 'shenyang', 'dalian', 'anshan', 'fushun', 'benxi', 'dandong', 'jinzhou','yingkou', 'fuxin', 'liaoyang', 'panjin', 'tieling', 'chaoyang', 'huludao', 'anhui', 'fujian', 'gansu','guangdong', 'guangxi', 'guizhou', 'hainan', 'hebei', 'henan', 'heilongjiang', 'hubei', 'hunan', 'jl','jiangsu', 'jiangxi', 'liaoning', 'neimenggu', 'ningxia', 'qinghai', 'shandong', 'shanxi', 'shaanxi','sichuan', 'xizang', 'xinjiang', 'yunnan', 'zhejiang', 'jjj', 'jzh', 'zsj', 'csj', 'ygc']# 品牌类型列表
CAR_CODE_LIST = ['southeastautomobile', 'sma', 'audi', 'hummer', 'tianqimeiya', 'seat', 'lamborghini', 'weltmeister','changanqingxingche-281', 'chevrolet', 'fiat', 'foday', 'eurise', 'dongfengfengdu', 'lotus-146', 'jac','enranger', 'bjqc', 'luxgen', 'jinbei', 'sgautomotive', 'jonwayautomobile', 'beijingjeep', 'linktour','landrover', 'denza', 'jeep', 'rely', 'gacne', 'porsche', 'wey', 'shenbao', 'bisuqiche-263','beiqihuansu', 'sinogold', 'roewe', 'maybach', 'greatwall', 'chenggongqiche', 'zotyeauto', 'kaersen','gonow', 'dodge', 'siwei', 'ora', 'lifanmotors', 'cajc', 'hafeiautomobile', 'sol', 'beiqixinnengyuan','dorcen', 'lexus', 'mercedesbenz', 'ford', 'huataiautomobile', 'jmc', 'peugeot', 'kinglongmotor','oushang', 'dongfengxiaokang-205', 'chautotechnology', 'faw-hongqi', 'mclaren', 'dearcc','fengxingauto', 'singulato', 'nissan', 'saleen', 'ruichixinnengyuan', 'yulu', 'isuzu', 'zhinuo','alpina', 'renult', 'kawei', 'cadillac', 'hanteng', 'defu', 'subaru', 'huasong', 'casyc', 'geely','xpeng', 'jlkc', 'sj', 'nanqixinyatu1', 'horki', 'venucia', 'xinkaiauto', 'traum','shanghaihuizhong-45', 'zhidou', 'ww', 'riich', 'brillianceauto', 'galue', 'bugatti','guagnzhouyunbao', 'borgward', 'qzbd1', 'bj', 'changheauto', 'faw', 'saab', 'fuqiautomobile', 'skoda','citroen', 'mitsubishi', 'opel', 'qorosauto', 'zxauto', 'infiniti', 'mazda', 'arcfox-289','jinchengautomobile', 'kia', 'mini', 'tesla', 'gmc-109', 'chery', 'daoda-282', 'joylongautomobile','hafu-196', 'sgmw', 'wiesmann', 'acura', 'yunqueqiche', 'volvo', 'lynkco', 'karry', 'chtc', 'gq','redstar', 'everus', 'kangdi', 'chrysler', 'cf', 'maxus', 'smart', 'maserati', 'dayu', 'besturn','dadiqiche', 'ym', 'huakai', 'buick', 'faradayfuture', 'leapmotor', 'koenigsegg', 'bentley','rolls-royce', 'iveco', 'dongfeng-27', 'haige1', 'ds', 'landwind', 'volkswagen', 'sitech', 'toyota','polarsunautomobile', 'zhejiangkaersen', 'ladaa', 'lincoln', 'weilaiqiche', 'li', 'ferrari', 'jetour','honda', 'barbus', 'morgancars', 'ol', 'sceo', 'hama', 'dongfengfengguang', 'mg-79', 'ktm','changankuayue-283', 'suzuki', 'yudo', 'yusheng-258', 'fs', 'bydauto', 'jauger', 'foton', 'pagani','shangqisaibao', 'guangqihinomotors', 'polestar', 'fujianxinlongmaqichegufenyouxiangongsi','alfaromeo', 'shanqitongjia1', 'xingchi', 'lotus', 'hyundai', 'kaiyi', 'isuzu-132', 'bmw', 'ssangyong','astonmartin']

新建一个py文件，将其url拼接的所有结果存储到redis数据库中，用于后面爬取队列提取
redis_url.py

from taoche.spiders.city import CITY_CODE,CAR_CODE_LISTfrom redis import Redis
# import sys
# sys.path.append("..")
class Redis_url():def __init__(self):#1.连接客户端，self.re = Redis("localhost", 6379)def add(self,url):#讲url,利用lpush方法，添加到"taoche:start_urls"键中，url为值self.re.lpush("taoche:start_urls",url)rd = Redis_url()# 先将redis中的requests全部清空
# flushdbRes = rd.flushdb()
for city in CITY_CODE:for car_code in CAR_CODE_LIST:rd.add( "https://{}.taoche.com/{}/".format(city, car_code))

脚本运行之后，城市列表和汽车品牌交叉拼接之后的结果就保存到redis数据库中

修改爬虫文件：
由于分布式爬虫需要部署在多台电脑上，所以起始的url就不能写在某一台电脑的start_urls中，需要指定一个redis_key这个字段用来在redis中写入爬虫文件所需要的第一个url
redis_key这个字段命名规则

这个名字就是存入redis数据库的键

redis_key = “任意的字符串:start_urls”

修改setting配置(redis)：

#调度器
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
#去重
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
#redis服务器地址
REDIS_HOST = 'localhost'（master主机写自己的地址，其他从机slaver写master远程主机地址）
#redis端口号
REDIS_PORT = 6379
##开启队列
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.PriorityQueue'

三、MongoDB数据存储设置

1、更改MongoDB数据库配置文件，主要用于其他从机slaver可以存储到远程的master的MongoDB数据库，添加如下配置

logappend=true
journal=true
quiet=true
port=27017

2、在pipelines.py文件中配置数据存储任务，连接远程的数据库

import pymongoclass TaochePipeline(object):def process_item(self, item, spider):return itemclass TaocheMongoPipeline(object):def __init__(self):# 创建客户端（master）self.client = pymongo.MongoClient(host='10.10.21.184',port=27017)# 创建数据库self.db = self.client['taoche']# 创建集合self.collection = self.db['car']def process_item(self, item, spider):print("="*30)self.collection.insert(dict(item))return item

3、settings配置中使用此pipeline配置,解开注释，更新配置

到此，所有准备工作已完成，只要爬虫写好，就可以完整的运行整个项目进行淘车数据爬取

四、编写爬虫程序

1、爬取定义数据的字段数据，分析淘车页面，列表页，详情页结构，进行相应的数据爬取
item.py

import scrapyclass TaocheItem(scrapy.Item):# define the fields for your item here like:# name = scrapy.Field()# 列表页title = scrapy.Field()  # 标题reg_date = scrapy.Field()  # 上牌日期mile = scrapy.Field()  # 公里数price = scrapy.Field()  # 优惠价格all_price = scrapy.Field()  # 全款价格detail_url = scrapy.Field()  # 详情url# 详情页city_name = scrapy.Field()  # 城市名称pic = scrapy.Field()  # 图片displace = scrapy.Field()  # 排量source_id = scrapy.Field()  # 车源号name = scrapy.Field() # 用于标识master数据库中存储的是哪个从机的数据

2、编写爬虫程序
spider:
s_taoche.py

# -*- coding: utf-8 -*-
import scrapy
import re
from taoche.spiders.city import CITY_CODE,CAR_CODE_LIST
from lxml import etree
from taoche.items import TaocheItem
from scrapy_redis.spiders import RedisSpider # 导入RedisSpider
# 城市编码
city_code = CITY_CODE
# 车 品牌列表
car_code_list = CAR_CODE_LIST
# for city in city_code:
#     for car in car_code_list:
#         url = f'https://{city}.taoche.com/{car}/?page='
#         print(url)
# print(len(city_code)*len(car_code_list))# class STaocheSpider(scrapy.Spider):
class STaocheSpider(RedisSpider):name = 's_taoche'redis_key = 'taoche:start_urls'# allowed_domains = ['taoche.com']# start_urls = ['https://beijing.taoche.com/all/?page=1']# https://beijing.taoche.com/all/?page=2# https://beijing.taoche.com/audi/?page=2#pagetag# 暂时没有找到符合条件的二手车# 'class="pages-next"'# 'class="pages-next pages-disabled"'def parse(self, response):# print("----",response.url)# html = response.body.decode('utf-8')page = response.xpath('//div[@class="paging-box the-pages"]/div/a[last()-1]/text()').extract()if len(page) > 0:page = page[0]else:page = 1# tree = etree.HTML(response.body.decode('utf-8'))# page = tree.xpath('//div[@class="paging-box the-pages"]/div/a[last()-1]/text()')# print("========",page)for p in range(1,int(page)+1):url = response.url + '?page=%s'%p# print("+++++++++",url)yield scrapy.Request(url=url,callback=self.parse_list)def parse_list(self, response):# with open('taoche.html','w',encoding='utf-8')as fp:#     fp.write(response.body.decode('utf-8'))# li_list = response.xpath('//div[@id="container_base"]/ul/li')# html = response.body.decode('utf-8')tree = etree.HTML(response.body.decode('utf-8'))# 判断列表页是否有数据data = tree.xpath('//div[@id="container_base"]')if data:# tree = etree.HTML(response.body.decode('utf-8'))# 汽车信息盒子列表li_list = tree.xpath('//div[@id="container_base"]/ul/li')print("-------",len(li_list))for li in li_list:# 实例化itemitem = TaocheItem()# -----列表页# 标题title = li.xpath('./div[@class="gongge_main"]/a/span/text()')[0]print("-------",title)# 上牌日期reg_date = li.xpath('./div[@class="gongge_main"]/p/i[1]/text()')[0]print("-------", reg_date)# 公里数mile = li.xpath('./div[@class="gongge_main"]/p/i[2]/text()')[0]print("-------", mile)# 优惠价格price = li.xpath('.//div[@class="price"]/i[@class="Total brand_col"]//text()')price = ''.join(price)print("-------", price)# 全款价格all_price = li.xpath('.//div[@class="price"]/i[@class="onepaynor"]/text()')[0]all_price = re.findall(r'原价(.*)',all_price)[0]print("-------", all_price)# 详情页urldetail_url = li.xpath('./div[@class="gongge_main"]/a/@href')[0]detail_url = 'https:' + detail_urlprint("-------", detail_url)# item数据item['title'] = titleitem['reg_date'] = reg_dateitem['mile'] = mileitem['price'] = priceitem['all_price'] = all_priceitem['detail_url'] = detail_urlyield scrapy.Request(url=detail_url,callback=self.parse_detail,meta={'data':item},dont_filter=False)else:passdef parse_detail(self, response):# 继承上面的itemitem = response.meta['data']# with open('car_detail.html','w',encoding='utf-8')as fp:#     fp.write(response.body.decode('utf-8'))tree = etree.HTML(response.body.decode('utf-8'))# 城市名称city_name = tree.xpath('//div[@class="summary-attrs"]/dl[last()]/dd/text()')[0]print("-------", city_name)# 图片pic = tree.xpath('//ul[@id="taoche-details-piclist"]/li/img/@data-src')# print("-------", pic)# 排量displace = tree.xpath('//div[@class="summary-attrs"]/dl[3]/dd/text()')[0]displace = re.findall(r'(.*?)/.*?',displace)[0]print("-------", displace)# 车源号source_id = tree.xpath('//span[@class="car-number"]/text()')[0]source_id = re.findall(r'车源号：(\d+)',source_id)[0]print("-------", source_id)item['city_name'] = city_nameitem['pic'] = picitem['displace'] = displaceitem['source_id'] = source_iditem['name'] = 'zz'yield item

3、启动爬虫：（尽量多个从机slaver同时运行爬虫，一般从机使用的程序脚本都是同样的代码，都是有一个人进行操作多台电脑进行的）

scrapy crawl s_taoche

日志消息截取：

2019-09-05 20:22:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.taoche.com/buycar/b-dealerydg229090677t
.html?source=2808>
{'all_price': '6.52万','city_name': '信阳','detail_url': 'https://www.taoche.com/buycar/b-dealerydg229090677t.html?source=2808','displace': '1.8L','mile': '6.27万公里','name': 'zz','pic': ['//img8.taoche.cn/1j/b555595e-10190dr111.jpg','//img8.taoche.cn/1j/66e9da68-10190dr1es.jpg','//img8.taoche.cn/1j/1d5e781f-10190dr12f.jpg','//img8.taoche.cn/1j/d81fb097-10190dr11f.jpg','//img8.taoche.cn/1j/f4cc9363-10190dr1d2.jpg','//img8.taoche.cn/1j/b8f1350a-10190dr133.jpg','//img8.taoche.cn/1j/57c478c4-10190dr1lg.jpg','//img8.taoche.cn/1j/4899f89a-10190dr1ii.jpg','//img8.taoche.cn/1j/e79e46cc-10190dr1j2.jpg','//img8.taoche.cn/1j/00ea278a-10190dr1jt.jpg','//img8.taoche.cn/1j/ab187361-10190dr1jy.jpg','//img8.taoche.cn/1j/4a66e0a9-10190dr0im.jpg','//img8.taoche.cn/1j/031ebe07-10190dr0j4.jpg','//img8.taoche.cn/1j/70c8758f-10190dr0k9.jpg','//img8.taoche.cn/1j/31af4c14-10190dr0ju.jpg','//img8.taoche.cn/1j/6e98df85-10190dr0u7.jpg','//img8.taoche.cn/1j/de482adf-10190dr0un.jpg','//img8.taoche.cn/1j/e2316c5b-10190dr15u.jpg','//img8.taoche.cn/1j/186afbbf-10190dr0hl.jpg','//img8.taoche.cn/1j/6a4e5dcb-10190dr1h7.jpg','//img8.taoche.cn/1j/241d2ead-10190dr12o.jpg','//img8.taoche.cn/1j/4504536a-10190dr1fu.jpg','//img8.taoche.cn/1j/158fd0d7-10190dr1cf.jpg','//img8.taoche.cn/1j/cc3a79c3-10190dr1hw.jpg','//img8.taoche.cn/1j/9e5127ad-10190dr0gj.jpg'],'price': '6.29万','reg_date': '2017年','source_id': '020688064','title': '东风风光风光580 2016款 1.8L 手动 舒适版'}
------- 重庆
------- 1.8L
------- 020683412
==============================
2019-09-05 20:22:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.taoche.com/buycar/b-dealerydg228940722t
.html?source=2808>
{'all_price': '6.42万','city_name': '重庆','detail_url': 'https://www.taoche.com/buycar/b-dealerydg228940722t.html?source=2808','displace': '1.8L','mile': '3.61万公里','name': 'zz','pic': ['//img5.taoche.cn/1j/387d1be4-10190dpssy.jpg','//img5.taoche.cn/1j/ebb12a45-10190dpszz.jpg','//img5.taoche.cn/1j/a7c54403-10190dpst7.jpg','//img5.taoche.cn/1j/190b53c0-10190dpst3.jpg','//img5.taoche.cn/1j/e3cabc3d-10190dpsxt.jpg','//img5.taoche.cn/1j/21859af8-10190dpstb.jpg','//img5.taoche.cn/1j/7fa07cb6-10190dptdl.jpg','//img5.taoche.cn/1j/c59d0791-10190dptc9.jpg','//img5.taoche.cn/1j/7a83f37d-10190dptc3.jpg','//img5.taoche.cn/1j/15d3483a-10190dptcc.jpg','//img5.taoche.cn/1j/f8624dd2-10190dptc5.jpg','//img5.taoche.cn/1j/51109bd7-10190dpta8.jpg','//img5.taoche.cn/1j/fe31cbbb-10190dpsl5.jpg','//img5.taoche.cn/1j/d73c76dd-10190dptaq.jpg','//img5.taoche.cn/1j/4993dcdb-10190dptau.jpg','//img5.taoche.cn/1j/28654a8e-10190dpskz.jpg','//img5.taoche.cn/1j/ca614179-10190dpsl3.jpg','//img5.taoche.cn/1j/0c395dd8-10190dpstw.jpg','//img5.taoche.cn/1j/7f4c0e6b-10190dpt89.jpg','//img5.taoche.cn/1j/77f69a05-10190dpt75.jpg','//img5.taoche.cn/1j/7b5fa6e4-10190dpstj.jpg','//img5.taoche.cn/1j/2814eb54-10190dpszu.jpg','//img5.taoche.cn/1j/2483fcab-10190dpsxk.jpg','//img5.taoche.cn/1j/9036e828-10190dpt8d.jpg','//img5.taoche.cn/1j/18f55630-10190dpsly.jpg'],'price': '6.18万','reg_date': '2017年','source_id': '020683412','title': '东风风光风光580 2016款 1.8L 手动 舒适版'}
2019-09-05 20:22:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.taoche.com/buycar/b-dealerydg229090961t
.html?source=2808> (referer: https://zsj.taoche.com/dongfengfengguang/?page=34)
------- 黔西南
------- 1.8L
------- 020687925
==============================
2019-09-05 20:22:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.taoche.com/buycar/b-dealerydg229090961t
.html?source=2808>
{'all_price': '8.48万','city_name': '黔西南','detail_url': 'https://www.taoche.com/buycar/b-dealerydg229090961t.html?source=2808','displace': '1.8L','mile': '1.29万公里','name': 'zz','pic': ['//img8.taoche.cn/1j/a0f3191c-10190dr6kc.jpg','//img8.taoche.cn/1j/51b59cbd-10190dr6pq.jpg','//img8.taoche.cn/1j/a15aa5f9-10190dr6mp.jpg','//img8.taoche.cn/1j/6270abb1-10190dr6nn.jpg','//img8.taoche.cn/1j/699d154b-10190dr6oe.jpg','//img8.taoche.cn/1j/e21eeaa4-10190dr6m1.jpg','//img8.taoche.cn/1j/ae4c5772-10190dr6mu.jpg','//img8.taoche.cn/1j/1cb63081-10190dr6lb.jpg','//img8.taoche.cn/1j/bb19e020-10190dr6n1.jpg','//img8.taoche.cn/1j/6a0ab008-10190dr6lu.jpg','//img8.taoche.cn/1j/da4637c8-10190dr6ly.jpg','//img8.taoche.cn/1j/dc00aac0-10190dr6nv.jpg','//img8.taoche.cn/1j/9cd79b92-10190dr6mz.jpg','//img8.taoche.cn/1j/71a8ebfe-10190dr6na.jpg','//img8.taoche.cn/1j/cc2d107b-10190dr6ln.jpg','//img8.taoche.cn/1j/90189e12-10190dr6of.jpg','//img8.taoche.cn/1j/fe7b4739-10190dr6o4.jpg','//img8.taoche.cn/1j/65fee368-10190dr6nb.jpg','//img8.taoche.cn/1j/f96b7470-10190dr6nr.jpg','//img8.taoche.cn/1j/5c622e3a-10190dr6mx.jpg','//img8.taoche.cn/1j/9d009de2-10190dr6ml.jpg','//img8.taoche.cn/1j/909c2a5d-10190dr6k5.jpg','//img8.taoche.cn/1j/3bcc7984-10190dr6o8.jpg','//img8.taoche.cn/1j/13a49dc6-10190dr6ma.jpg','//img8.taoche.cn/1j/bc7f3ba0-10190dr6mf.jpg'],'price': '8.17万','reg_date': '2018年','source_id': '020687925','title': '东风风光风光580 2016款 1.8L 手动 舒适版'}
2019-09-05 20:22:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.taoche.com/buycar/b-dealerydg229090967t
.html?source=2808> (referer: https://zsj.taoche.com/dongfengfengguang/?page=34)
------- 西安
------- 1.8L
------- 020687925
==============================
2019-09-05 20:22:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.taoche.com/buycar/b-dealerydg229090967t
.html?source=2808>
{'all_price': '8.50万','city_name': '西安','detail_url': 'https://www.taoche.com/buycar/b-dealerydg229090967t.html?source=2808','displace': '1.8L','mile': '1.29万公里','name': 'zz','pic': ['//img6.taoche.cn/1j/a0f3191c-10190dr6kc.jpg','//img6.taoche.cn/1j/51b59cbd-10190dr6pq.jpg','//img6.taoche.cn/1j/a15aa5f9-10190dr6mp.jpg','//img6.taoche.cn/1j/6270abb1-10190dr6nn.jpg','//img6.taoche.cn/1j/699d154b-10190dr6oe.jpg','//img6.taoche.cn/1j/e21eeaa4-10190dr6m1.jpg','//img6.taoche.cn/1j/ae4c5772-10190dr6mu.jpg','//img6.taoche.cn/1j/1cb63081-10190dr6lb.jpg','//img6.taoche.cn/1j/bb19e020-10190dr6n1.jpg','//img6.taoche.cn/1j/6a0ab008-10190dr6lu.jpg','//img6.taoche.cn/1j/da4637c8-10190dr6ly.jpg','//img6.taoche.cn/1j/dc00aac0-10190dr6nv.jpg','//img6.taoche.cn/1j/9cd79b92-10190dr6mz.jpg','//img6.taoche.cn/1j/71a8ebfe-10190dr6na.jpg','//img6.taoche.cn/1j/cc2d107b-10190dr6ln.jpg','//img6.taoche.cn/1j/90189e12-10190dr6of.jpg','//img6.taoche.cn/1j/fe7b4739-10190dr6o4.jpg','//img6.taoche.cn/1j/65fee368-10190dr6nb.jpg','//img6.taoche.cn/1j/f96b7470-10190dr6nr.jpg','//img6.taoche.cn/1j/5c622e3a-10190dr6mx.jpg','//img6.taoche.cn/1j/9d009de2-10190dr6ml.jpg','//img6.taoche.cn/1j/909c2a5d-10190dr6k5.jpg','//img6.taoche.cn/1j/3bcc7984-10190dr6o8.jpg','//img6.taoche.cn/1j/13a49dc6-10190dr6ma.jpg','//img6.taoche.cn/1j/bc7f3ba0-10190dr6mf.jpg'],'price': '8.19万','reg_date': '2018年','source_id': '020687925','title': '东风风光风光580 2016款 1.8L 手动 舒适版'}
2019-09-05 20:22:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.taoche.com/buycar/b-dealerydg229090974t
.html?source=2808> (referer: https://zsj.taoche.com/dongfengfengguang/?page=34)
------- 平顶山
------- 1.8L
------- 020687925
==============================
2019-09-05 20:22:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.taoche.com/buycar/b-dealerydg229090974t
.html?source=2808>
{'all_price': '8.56万','city_name': '平顶山','detail_url': 'https://www.taoche.com/buycar/b-dealerydg229090974t.html?source=2808','displace': '1.8L','mile': '1.29万公里','name': 'zz','pic': ['//img5.taoche.cn/1j/a0f3191c-10190dr6kc.jpg','//img5.taoche.cn/1j/51b59cbd-10190dr6pq.jpg','//img5.taoche.cn/1j/a15aa5f9-10190dr6mp.jpg','//img5.taoche.cn/1j/6270abb1-10190dr6nn.jpg','//img5.taoche.cn/1j/699d154b-10190dr6oe.jpg','//img5.taoche.cn/1j/e21eeaa4-10190dr6m1.jpg','//img5.taoche.cn/1j/ae4c5772-10190dr6mu.jpg','//img5.taoche.cn/1j/1cb63081-10190dr6lb.jpg','//img5.taoche.cn/1j/bb19e020-10190dr6n1.jpg','//img5.taoche.cn/1j/6a0ab008-10190dr6lu.jpg','//img5.taoche.cn/1j/da4637c8-10190dr6ly.jpg','//img5.taoche.cn/1j/dc00aac0-10190dr6nv.jpg','//img5.taoche.cn/1j/9cd79b92-10190dr6mz.jpg','//img5.taoche.cn/1j/71a8ebfe-10190dr6na.jpg','//img5.taoche.cn/1j/cc2d107b-10190dr6ln.jpg','//img5.taoche.cn/1j/90189e12-10190dr6of.jpg','//img5.taoche.cn/1j/fe7b4739-10190dr6o4.jpg','//img5.taoche.cn/1j/65fee368-10190dr6nb.jpg','//img5.taoche.cn/1j/f96b7470-10190dr6nr.jpg','//img5.taoche.cn/1j/5c622e3a-10190dr6mx.jpg','//img5.taoche.cn/1j/9d009de2-10190dr6ml.jpg','//img5.taoche.cn/1j/909c2a5d-10190dr6k5.jpg','//img5.taoche.cn/1j/3bcc7984-10190dr6o8.jpg','//img5.taoche.cn/1j/13a49dc6-10190dr6ma.jpg','//img5.taoche.cn/1j/bc7f3ba0-10190dr6mf.jpg'],'price': '8.25万','reg_date': '2018年','source_id': '020687925','title': '东风风光风光580 2016款 1.8L 手动 舒适版'}