新手菜鸟招聘网站数据爬取

这里写自定义目录标题

准备工作
上代码
我踩过的坑
参考文献

python新手，第一次写爬虫，真是一把辛酸泪啊
对前程无忧 51job进行爬取，选择的关键字为大数据开发工程师和本科
一开始是想对这个网址 51job官网进行爬取，发现采用requests.get爬下来的数据总是缺少想要的招聘信息，因此参考了另外一篇博客，才改用 51job（链接名随便取的，可忽略）

准备工作

1.环境配置
pycharm
需要用到的第三方库有requests、csv、random、time、bs4、beautifulsoup
2.反反爬虫
以防万一，把Network中Headers请求标题部分全部复制粘贴下来了

    headers = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9','Accept-Encoding': 'gzip, deflate, br','Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6','Cache-Control': 'max-age=0','Connection': 'keep-alive','Cookie': 'guid=9b0614a19b30a1ad861333c681eb5aad; nsearch=jobarea%3D%26%7C%26ord_field%3D%26%7C%26recentSearch0%3D%26%7C%26recentSearch1%3D%26%7C%26recentSearch2%3D%26%7C%26recentSearch3%3D%26%7C%26recentSearch4%3D%26%7C%26collapse_expansion%3D; search=jobarea%7E%60030200%7C%21ord_field%7E%600%7C%21recentSearch0%7E%60030200%A1%FB%A1%FA000000%A1%FB%A1%FA0000%A1%FB%A1%FA00%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA9%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA0%A1%FB%A1%FA%B4%F3%CA%FD%BE%DD%BF%AA%B7%A2%B9%A4%B3%CC%CA%A6%A1%FB%A1%FA2%A1%FB%A1%FA1%7C%21recentSearch1%7E%60030200%A1%FB%A1%FA000000%A1%FB%A1%FA0000%A1%FB%A1%FA00%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA9%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA0%A1%FB%A1%FA%B4%F3%CA%FD%BE%DD%A1%FB%A1%FA2%A1%FB%A1%FA1%7C%21recentSearch2%7E%60030200%A1%FB%A1%FA000000%A1%FB%A1%FA0000%A1%FB%A1%FA00%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA9%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA0%A1%FB%A1%FA%BC%C6%CB%E3%BB%FA%A1%FB%A1%FA2%A1%FB%A1%FA1%7C%21recentSearch3%7E%60030200%A1%FB%A1%FA000000%A1%FB%A1%FA0000%A1%FB%A1%FA00%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA9%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA0%A1%FB%A1%FAweb%C7%B0%B6%CB%A1%FB%A1%FA2%A1%FB%A1%FA1%7C%21recentSearch4%7E%60030400%2C030200%A1%FB%A1%FA000000%A1%FB%A1%FA0000%A1%FB%A1%FA00%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA9%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA0%A1%FB%A1%FAweb%C7%B0%B6%CB%A1%FB%A1%FA2%A1%FB%A1%FA1%7C%21collapse_expansion%7E%601%7C%21','Host': 'jobs.51job.com','Referer': 'https://jobs.51job.com/ed04/dashujukaifa/','sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Microsoft Edge";v="90"','sec-ch-ua-mobile': '?0','Sec-Fetch-Dest': 'document','Sec-Fetch-Mode': 'navigate','Sec-Fetch-Site': 'same-origin','Sec-Fetch-User': '?1','Upgrade-Insecure-Requests': '1','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 Edg/90.0.818.46'}

3.观察网页源代码
以第一行为例，在大数据开发工程师那里，右键单击检查，查看网页源码（具体方法视不同浏览器而定，我用的是Microftsoft Edge），可以快速定位到该处对应的源码

观察源码可发现，在

div class=“detlist gbox”

下面有一列的

div class=“e”

每一个div class="e"都对应着一则招聘信息
点开div class=“e”，可以看到里面的标签分别对应着招聘信息中的职位、公司、薪酬等等，因此可以有个大概的思路：写一个循环，每次循环对每一个div class="e"中的解析后的标签进行存储

        for div in total_list:title = div.find_all('a')job = title[0].stringcompany = title[1].stringlocal_name = div.select('.info>span')[1].stringsalary = div.select('.info>span')[2].stringdate = div.select('.info>span')[3].stringinformation = div.find('p', class_="text").string

4.观察网址
https://jobs.51job.com/ed04/dashujukaifa/，ed04和/dashujukaifa分别对应着我已经选择好的标签：本科和大数据开发工程师，可以不用动，尝试分页，发现第二页是https://jobs.51job.com/ed04/dashujukaifa/p2，因此可以猜测，第一页也可以是https://jobs.51job.com/ed04/dashujukaifa/p1，到这里，我们对分页操作也有了个大致的思路，那就是再套一个循环，改变p后面的数字，进行分页爬取

    for value in range(1, 147):url = 'https://jobs.51job.com/ed04/dashujukaifa/p%s' % str(value)html = requests.get(url=url, headers=headers)Html = html.text.encode('iso-8859-1').decode('gbk')#解析数据soup = BeautifulSoup(Html, 'html.parser')total_list = soup.find('div',class_="detlist gbox").find_all('div',class_="e")for div in total_list:title = div.find_all('a')job = title[0].stringcompany = title[1].stringlocal_name = div.select('.info>span')[1].stringsalary = div.select('.info>span')[2].stringdate = div.select('.info>span')[3].stringinformation = div.find('p', class_="text").string

最后别忘了将数据写入csv中

上代码

完整代码如下：

import requests
import csv
import time
import random
from bs4 import BeautifulSoup
#爬数据：
if __name__ == '__main__':headers = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9','Accept-Encoding': 'gzip, deflate, br','Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6','Cache-Control': 'max-age=0','Connection': 'keep-alive','Cookie': 'guid=9b0614a19b30a1ad861333c681eb5aad; nsearch=jobarea%3D%26%7C%26ord_field%3D%26%7C%26recentSearch0%3D%26%7C%26recentSearch1%3D%26%7C%26recentSearch2%3D%26%7C%26recentSearch3%3D%26%7C%26recentSearch4%3D%26%7C%26collapse_expansion%3D; search=jobarea%7E%60030200%7C%21ord_field%7E%600%7C%21recentSearch0%7E%60030200%A1%FB%A1%FA000000%A1%FB%A1%FA0000%A1%FB%A1%FA00%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA9%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA0%A1%FB%A1%FA%B4%F3%CA%FD%BE%DD%BF%AA%B7%A2%B9%A4%B3%CC%CA%A6%A1%FB%A1%FA2%A1%FB%A1%FA1%7C%21recentSearch1%7E%60030200%A1%FB%A1%FA000000%A1%FB%A1%FA0000%A1%FB%A1%FA00%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA9%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA0%A1%FB%A1%FA%B4%F3%CA%FD%BE%DD%A1%FB%A1%FA2%A1%FB%A1%FA1%7C%21recentSearch2%7E%60030200%A1%FB%A1%FA000000%A1%FB%A1%FA0000%A1%FB%A1%FA00%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA9%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA0%A1%FB%A1%FA%BC%C6%CB%E3%BB%FA%A1%FB%A1%FA2%A1%FB%A1%FA1%7C%21recentSearch3%7E%60030200%A1%FB%A1%FA000000%A1%FB%A1%FA0000%A1%FB%A1%FA00%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA9%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA0%A1%FB%A1%FAweb%C7%B0%B6%CB%A1%FB%A1%FA2%A1%FB%A1%FA1%7C%21recentSearch4%7E%60030400%2C030200%A1%FB%A1%FA000000%A1%FB%A1%FA0000%A1%FB%A1%FA00%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA99%A1%FB%A1%FA9%A1%FB%A1%FA99%A1%FB%A1%FA%A1%FB%A1%FA0%A1%FB%A1%FAweb%C7%B0%B6%CB%A1%FB%A1%FA2%A1%FB%A1%FA1%7C%21collapse_expansion%7E%601%7C%21','Host': 'jobs.51job.com','Referer': 'https://jobs.51job.com/ed04/dashujukaifa/','sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Microsoft Edge";v="90"','sec-ch-ua-mobile': '?0','Sec-Fetch-Dest': 'document','Sec-Fetch-Mode': 'navigate','Sec-Fetch-Site': 'same-origin','Sec-Fetch-User': '?1','Upgrade-Insecure-Requests': '1','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 Edg/90.0.818.46'}fp = open('招聘数据终极版.csv', 'a', encoding='utf-8')csv_writer = csv.writer(fp)csv_writer.writerow(["职位名称","发布时间","薪资","公司","岗位信息"])for value in range(1, 147):url = 'https://jobs.51job.com/ed04/dashujukaifa/p%s' % str(value)html = requests.get(url=url, headers=headers)Html = html.text.encode('iso-8859-1').decode('gbk')#解析数据soup = BeautifulSoup(Html, 'html.parser')total_list = soup.find('div',class_="detlist gbox").find_all('div',class_="e")for div in total_list:title = div.find_all('a')job = title[0].stringcompany = title[1].stringlocal_name = div.select('.info>span')[1].stringsalary = div.select('.info>span')[2].stringdate = div.select('.info>span')[3].stringinformation = div.find('p', class_="text").stringcsv_writer.writerow([job, company, local_name, salary, date, information])print(value)time.sleep(random.random()*3)fp.close()

这么点代码我写了这么久真的。。。。。。

我踩过的坑

由于对BeautifulSoup各种方法不了解，在查找标签的时候费了很大功夫。比如要找职位和薪酬的数据，你会发现，他们两个的标签类名长得差不多：

我第一次是这么写的：

local_name = div.find(‘span’,class_=“location name”)
salary= div.find(‘span’,class_=“location”)

结果就是salary和local_name爬出来都是地名，我想的是可能findclass_=“location”匹配到“location name”空格前面就相当于符合了？
不清楚原由，最后使用了select，返回的是列表，更简单了，只要传入想要的标签下标就行
还有注意解析后的网页是否为乱码，加上这一句：
Html = html.text.encode(‘iso-8859-1’).decode(‘gbk’)
以及，可以先试试爬取第一页存到一个本地的html文档，在里面ctirl+F找找自己要的标签是否真的在里面，免的搞了半天一直find不到白费力气…

参考文献

https://www.cnblogs.com/lixin2011/p/14081397.html
https://www.bilibili.com/video/BV1ha4y1H7sx