python3.6.5 +  pycharm

注意：

一、python3里的 urllib2 已经没有了，改为了 urllbi.request，因此，直接导入 import urllib.request 即可。

二、必须对正则表达式里的引用变量进行格式变换  .decode('utf-8')，否则会报错说 不能在一个字节类的对象上使用字符串格式。

如下代码所示。

### 一、 网站地图爬虫，控制用户代理的设定，可捕获异常、重试下载并设置用户代理。wswp: web scraping with python
import urllib.request  ## -- written by LiSongbo
def Rocky_dnload(url,user_agent='wswp',num_retries = 2):print('Downloading:',url)LiSongbo_he={'User-agent':user_agent}request = urllib.request.Request(url, headers=LiSongbo_he)try:  ## -- written by LiSongbo
html = urllib.request.urlopen(request).read()except urllib.request.URLError as e:  ## -- written by LiSongbo
print('Download error:',e.reason)html = Noneif num_retries > 0:  ## -- written by LiSongbo
if hasattr(e,'code') and 500 <= e.code < 600:return Rocky_dnload(url,user_agent,num_retries-1) ## retry 5xx HTTP errors
return htmlimport re  ## -- written by LiSongbo
def Rocky_crawl_sitemap(url):  ## -- written by LiSongbo
sitemap = Rocky_dnload(url)  ## download the sitmap file
    # sitemap = sitemap.decode('utf-8') ## must add this .
links = re.findall('<loc>(.*?)</loc>', sitemap)  ## extract the sitemap links from flag loc
for link in links:  ## download each link
html = Rocky_dnload(link)  ## crape html here
Rocky_crawl_sitemap('http://example.webscraping.com/sitemap.xml')

运行结果报错：

Downloading: http://example.webscraping.com/sitemap.xml
Traceback (most recent call last):
  File "C:/Users/klooa/my_env/book9/test.py", line 25, in <module>
    Rocky_crawl_sitemap('http://example.webscraping.com/sitemap.xml')
  File "C:/Users/klooa/my_env/book9/test.py", line 22, in Rocky_crawl_sitemap
    links = re.findall('<loc>(.*?)</loc>', sitemap)  ## extract the sitemap links from flag loc
  File "C:\Users\klooa\AppData\Local\Programs\Python\Python36\lib\re.py", line 222, in findall
    return _compile(pattern, flags).findall(string)
TypeError: cannot use a string pattern on a bytes-like object

必须在 sitemap的下一行加上

sitemap = sitemap.decode('utf-8')

修改后的运行结果为：

Downloading: http://example.webscraping.com/sitemap.xml
Downloading: http://example.webscraping.com/places/default/view/Afghanistan-1
Downloading: http://example.webscraping.com/places/default/view/Aland-Islands-2
Downloading: http://example.webscraping.com/places/default/view/Albania-3
Downloading: http://example.webscraping.com/places/default/view/Algeria-4
Downloading: http://example.webscraping.com/places/default/view/American-Samoa-5
Downloading: http://example.webscraping.com/places/default/view/Andorra-6
Downloading: http://example.webscraping.com/places/default/view/Angola-7
Downloading: http://example.webscraping.com/places/default/view/Anguilla-8
Downloading: http://example.webscraping.com/places/default/view/Antarctica-9
Downloading: http://example.webscraping.com/places/default/view/Antigua-and-Barbuda-10
Downloading: http://example.webscraping.com/places/default/view/Argentina-11
Downloading: http://example.webscraping.com/places/default/view/Armenia-12
Download error: TOO MANY REQUESTS
Downloading: http://example.webscraping.com/places/default/view/Aruba-13

Download error: TOO MANY REQUESTS

……

## -- written by LiSongbo