Scrapy 爬取盗墓笔记小说

应用 Scrapy框架 爬取盗墓笔记小说数据，存入MongoDB 数据库。

# settings 配置mongodb
MONGODB_HOST = '127.0.0.1'
MONGODB_PORT = 27017
MONGODB_DBNAME = 'MySpider'
MONGODB_DOCNAME = 'daomubiji'

# items 配置抓取数据字段
import scrapyclass NovelItem(scrapy.Item):bookName = scrapy.Field()bookTitle = scrapy.Field()chapterNum = scrapy.Field()chapterName = scrapy.Field()chapterUrl = scrapy.Field()

# spider 抓取数据
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from novel.items import NovelItemclass DaomubijiSpider(CrawlSpider):name = 'daomubiji'allowed_domains = ['daomubiji.com']start_urls = ['http://www.daomubiji.com/']def parse_start_url(self, response):passrules = (Rule(LinkExtractor(restrict_xpaths='//article[@class="article-content"]//a'), callback='parse_item', follow=True),)def parse_item(self, response):item = NovelItem()list = response.xpath('//body')for listItem in list:item['bookName'] = listItem.xpath('.//h1[@class="focusbox-title"]/text()').get().split('：')[0]subList = listItem.xpath('.//div[@class="excerpts"]//article')for subListItem in subList:item['bookTitle'] = subListItem.xpath('.//a/text()').get().split(' ')[0]item['chapterNum'] = subListItem.xpath('.//a/text()').get().split(' ')[1]item['chapterName'] = subListItem.xpath('.//a/text()').get().split(' ')[2]item['chapterUrl'] = subListItem.xpath('.//a/@href').get()yield item

# pipeline 处理数据
from scrapy.conf import settings
import pymongoclass NovelPipeline(object):def __init__(self):host = settings['MONGODB_HOST']port = settings['MONGODB_PORT']dbName = settings['MONGODB_DBNAME']client = pymongo.MongoClient(host=host, port=port)db = client[dbName]self.post = db[settings['MONGODB_DOCNAME']]def open_spider(self, spider):print('This spider is starting!')def process_item(self, item, spider):bookInfo = dict(item)self.post.insert(bookInfo)return itemdef close_spider(self, spider):print('This spider is end!')

Scrapy 爬取盗墓笔记小说相关推荐

scrapy爬取起点小说网
闲来无事,在学习过程中练习用scrapy爬取起点小说名工具:python3.6 操作系统:linux 浏览器:谷歌浏览器创建项目在黑屏终端创建一个项目:scrapy startproject Q ...
Scrapy爬取顶点小说网
Scrapy爬取小说爬取目标:顶点小说网 1.Scrapy的安装 pip install scrapy 2.Scrapy的介绍创建项目 scrapy startproject xxx xxx项目名 ...
Scrapy爬取盗墓笔记 0.2版(mongedb redis)
这次的爬虫相对于上一版来说,增加了对小说内容的爬取,并且利用了Redis数据库. 环境: win10 py3.6 pycharm scrapy1.6 main.py from scrapy impor ...
scrapy爬取百万小说
#爬取小说实列第一步:创建一个scrapy工程[命令行中输入 scrapy startproject demo] 第二步:进入这个工程中,创建一个爬虫[scrapy nss zhuangji.org ...
python如何利用scrapy爬取纵横小说三级链接内容并存储到数据库
效果展示 settings.py# Scrapy settings for zongheng project # # For simplicity, this file contains only s ...
Scrapy爬取顶点小说
一.目标地址:www.23us.so/ 二.准备工作 1.开发环境 MacOS + PyCharm + Python3.5.3 + Scrapy + MySQL 2.安装Scrapy和MySQL自行解 ...
简单的scrapy爬取下载小说
一.scrapy简介 scrapy结构 items.py:用来存放爬虫爬取下来数据的模型 middlewares.py:用来存放各种中间件的文件 pipelines.py:用来将items的模型存储到 ...
python爬网页、爬到前几个就不动了_python scrapy 爬取起点小说，爬虫停止在第四页不动了...
如题,我在测试爬虫的时候,终端出现了如下代码: 2019-04-20 15:04:51 [scrapy.core.engine] DEBUG: Crawled (200) (referer: http ...
起点小说免费看 Scrapy爬取起点小说网数据导入MongoDB数据
本文中我们将详细介绍使用Scrapy抓取数据并存入MongoDB数据库,首先给出我们需要抓取得数据: 抓取起点网得全部作品,网址为:https://www.qidian.com/all 关于Scrap ...

Scrapy 爬取盗墓笔记小说

Scrapy 爬取盗墓笔记小说

Scrapy 爬取盗墓笔记小说相关推荐

最新文章

热门文章