2019-12-6 爬网页6-request模块+get请求+正则筛选

继续研究爬网页，这次的网站是https://www.yuanjisong.com/job/shanghai
这是一个纯静态的网页，请求方式是get，所以直接使用request模块就行了。

每个任务的相关html代码如下

<div class="weui_panel weui_panel_access weui_panel_access_adapt db_adapt margin-top-2 "><a href="https://www.yuanjisong.com/job/104128" target="_blank"><div class="weui_panel_hd weui_panel_hd_adapt media_desc_adapt_url"><div class="topic_title">系统二次开发</div></div></a><div class="job_list_item_div"><div class="weui_panel_bd "><div class="weui_media_box weui_media_text media_box_adapt"><a href="https://www.yuanjisong.com/job/104128" class="media_desc_content_adapt" target="_blank"><p class="media_desc_adapt "><span class="glyphicon glyphicon-th-large" aria-hidden="true"></span><span class="job_list_item_title ">描述：</span>在系统基础上增加新模块。具体需求加附件QQ我发给你。要求1. 3年以上**********MVC 开发经验;； 2. 至少掌握一种SQL关系型数据库(mysql或sqlserver); 3. 熟练掌握EasyUI、HTML、CSS、JavaScript、jQuery、AJAX、JSON等Web前端技术； 4. 使用Redis、MongoDB参与过实际项目的优先考虑<!--  <span class="more_text">详情...</span> --></p></a></div></div><div class="weui_panel_bd" ><a href="https://www.yuanjisong.com/employer/134659" class="weui_media_box weui_media_appmsg" target="_blank"><div class="weui_media_hd weui_media_hd_adapt"><img class="weui_media_appmsg_thumb radius_img_50" src="/Webpage/Job/../../Public/Img/Avatar/a1.jpg" alt="you" /></div><div class="weui_media_bd"><h4 class="weui_media_title font-color-3">you<!-- <span class="time_span">2019-11-09 17:46</span>--></h4></div></a></div><div class="weui_panel_bd"><div class="weui_media_box weui_media_text media_box_adapt"><p class="media_desc_adapt"><span class="glyphicon glyphicon-hourglass" aria-hidden="true"></span><span class="job_list_item_title ">工时：</span><span>4<span class="margin-left-1">天<span></span></p></div></div><div class="weui_panel_bd"><div class="weui_media_box weui_media_text media_box_adapt"><p class="media_desc_adapt"><span class="glyphicon glyphicon-yen" aria-hidden="true"></span><span class="job_list_item_title ">总价：</span><span class="rixin-text-jobs">2000<span class="margin-left-1">元<span></span></p></div></div></div><div class="weui_panel_bd appoint_div"><a  href="javascript:void(0);" data-jobid="104155" id="appoint_job_104155" class="weui_btn weui_btn_plain_primary vertical_a_m margin_left_17 appoint_confirm"   >投递职位</a>                 <span class="zhushi_span vertical_a_m">已有<i id="i_post_num_104155" class="i_post_num">17</i>人投递</span></div></div>

分析一下，可以得到每个任务包含的信息

任务名称
内容
委托人
工时，单位
报酬，单位
状态（可以投递，或者已完成）

要想得到以上信息，可以采用正则

 reg = re.compile(r'<div class="topic_title">(.*?)</div></div></a>'+'.*?<span class="job_list_item_title ">.*?</span>(.*?)<!--'+'.*?<h4 class="weui_media_title font-color-3">(.*?)<!--'+'.*?<span class="job_list_item_title ">.*?<span>(.*?)<span class="margin-left-1">(.*?)<span>'+'.*?<span class="rixin-text-jobs">(.*?)<span class="margin-left-1">(.*?)<span>'+'.*?margin_left_17.*?>(.*?)</a>',re.S)

关于页码的html代码如下

<ul class="pagination pagination_webpage"><li class="disabled"><span >上一页 <span class="sr-only">(current)</span></span></li><li class="disabled"><span <span aria-hidden="true">首页</span></span></li>              <li class="active"><span>1 <span class="sr-only">(current)</span></span></li><li class=""><a href="https://www.yuanjisong.com/job/shanghai/page2">2 <span class="sr-only">(current)</span></a></li>                              <li class=""><a href="https://www.yuanjisong.com/job/shanghai/page3">3 <span class="sr-only">(current)</span></a></li>                             <li class=""><a href="https://www.yuanjisong.com/job/shanghai/page4">4 <span class="sr-only">(current)</span></a></li>                             <li class=""><a href="https://www.yuanjisong.com/job/shanghai/page5">5 <span class="sr-only">(current)</span></a></li>                     <li class=""><a href="https://www.yuanjisong.com/job/shanghai/page2">下一页 <span class="sr-only">(current)</span></a></li>
</ul>

解决翻页的思路
step1.初始页page=1。
step2.开始循环，拼接页码https://www.yuanjisong.com/job/shanghai/page1
step3.获得上面html中所有链接中倒数第二个的文字，应该是5(current)
step4.通过正则获得其中的数字5
step5.和当前page比较，如果相等，退出循环。如果大于，那么page=page+1

找出最大页码对应数字的正则代码如下

get_page = soup.find('ul', attrs={'class': 'pagination pagination_webpage'}).find_all('a')
max_page = re.findall(r'\d+', get_page[-2].getText())  #最大一页是所有链接中倒数第二个。文本为：5 (current)。所以用正则找出数字

保存到excel

保存excel时候如果报错如下

 File "C:\Python27\lib\site-packages\pandas\core\generic.py", line 2127, in to_excelengine=engine)File "C:\Python27\lib\site-packages\pandas\io\formats\excel.py", line 662, in writefreeze_panes=freeze_panes)File "C:\Python27\lib\site-packages\pandas\io\excel.py", line 1605, in write_cellsxcell.value, fmt = self._value_with_fmt(cell.val)File "C:\Python27\lib\site-packages\openpyxl\cell\cell.py", line 272, in valueself._bind_value(value)File "C:\Python27\lib\site-packages\openpyxl\cell\cell.py", line 229, in _bind_valuevalue = self.check_string(value)File "C:\Python27\lib\site-packages\openpyxl\cell\cell.py", line 186, in check_stringraise IllegalCharacterError
openpyxl.utils.exceptions.IllegalCharacterError

可以在保存时候加入engine=‘xlsxwriter’
具体原因参见https://blog.csdn.net/javajiawei/article/details/97147219
其实定义在cell.py文件中有了，也就是我们保存的文件中出现了以下正则中的字符，这是不符合要求的。

ILLEGAL_CHARACTERS_RE = re.compile(r'[\000-\010]|[\013-\014]|[\016-\037]')

解决办法要么手工替换，要是使用xlsxwriter。

源码下载