原文链接: 爬虫下载文章 BeautifulSoup

上一篇: HDU 1427 dfs 速算24点

下一篇: 简单爬虫 爬知乎日报

# coding=utf-8from bs4 import BeautifulSoup
import urllib
import urllib2
import sys
reload(sys)
sys.setdefaultencoding('utf8')
url = 'http://www.pythontab.com/html/pythonhexinbiancheng/index.html'request = urllib2.urlopen(url)html = request.read()# print html# 解析方式
soup = BeautifulSoup(html, 'html.parser')
'''<ul class="list lh24 f14" id="catlist"><li><h3>Python高级教程</h3><a href="http://www.pythontab.com/html/2017/pythonhexinbiancheng_0821/1166.html" target="_blank" ><h2>一名数据挖掘工程师给新人整理的入门资料</h2>  </a><p>四年前我一次听说数据挖掘这个词，三年前我学习了数据挖掘理论知识，两年前我做了几个与数据挖掘有关的项目，一年前我成为一名数据挖掘工程   <a href="http://www.pythontab.com/html/2017/pythonhexinbiancheng_0821/1166.html" class="content_detail" target="_blank">[详细]</a></p></li>
'''# 先通过id 找 再通过标签找,注意空格必须有# 找到链接和标题
items = soup.select('#catlist > li > a')# 只找到标题
titles = soup.select('#catlist > li > a > h2')# 找超链接和标题,方便后续分割
links = soup.select('#catlist > li > a')
for i in items:print i.get_text()# zip函数,titles和links是列表通过zip将两个列表下标对应的两个作为字典
'''
l1 = [1,2,3]
l2 = {'a','b','c'}
d = zip(l1,l2)[(1, 'a'), (2, 'c'), (3, 'b')]
'''items = []
for title, link in zip(titles, links):data = {'title': title.get_text(),  # 获取标签的文本'link': link.get('href')  # 获取标签的属性}items.append(data)# 输出标题和链接
for i in items:print i['title'], i['link']'''<div class="content">'''
# 获取超链接内容
for i in items:request = urllib2.urlopen(i['link'])html = request.read().encode('utf-8')soup = BeautifulSoup(html, 'html.parser')title = i['title']texts = soup.select('div.content > p')content = []for t in texts:content.append(t.get_text().encode('utf-8'))with open('file/%s.html' % title, 'wb') as f:f.write(i['title']+'\n'+html)# for cont in content:#    f.write(cont+'\n')