Python爬虫理论Pro | (3) Requests库详解

在本篇博客中，我们将介绍requests库的详细用法，相比于之前的urllib库，requests库的接口更加简洁，如ip代理、cookie设置等操作，使用起来也更加方便，实际使用也更加广泛。

1. 什么是requests

2. 实例引入

3. 请求

4. 响应

5. 高级操作

6. 异常处理

1. 什么是requests

Requests 是⽤Python语言编写，基于 urllib，采用Apache2 Licensed 开源协议的 HTTP 库。它⽐ urllib 更加⽅便，可以节约我们⼤量的工作，完全满足HTTP 测试需求。它是Python实现的简单易用的HTTP库。

安装：pip install requests

2. 实例引入

import requestsresponse = requests.get('https://www.baidu.com/') #请求百度 get请求
print(type(response))
print(response.status_code) #200
print(type(response.text)) #str
print(response.text)
print(response.cookies) #获取cookie

各种请求方式

最常用的是get和post

import requests
requests.post('http://httpbin.org/post')
requests.put('http://httpbin.org/put')
requests.delete('http://httpbin.org/delete')
requests.head('http://httpbin.org/get')
requests.options('http://httpbin.org/get')

3. 请求

基本get请求

基本写法：

import requestsresponse = requests.get('http://httpbin.org/get')
print(response.text)

带参数的get请求：

import requests
#将参数放在?后 每个参数用&分隔
response = requests.get("http://httpbin.org/get?name=germey&age=22")
print(response.text)

import requests
#一种更简便的写法
#将参数写成字典形式 传给params参数 不需要自己写了
data = {'name': 'germey','age': 22
}
response = requests.get("http://httpbin.org/get", params=data)
print(response.text)

解析json:

import requests
import jsonresponse = requests.get("http://httpbin.org/get")
print(type(response.text))
print(response.json()) #将返回结果直接转化为json 在ajax请求中常用 和json.loads(response.text)等价
print("-------------------------")
print(json.loads(response.text))
print(type(response.json()))

获取二进制数据：

import requestsresponse = requests.get("https://github.com/favicon.ico") #github图标
print(type(response.text), type(response.content))#str,bytes
print(response.text)
print(response.content)#二进制内容

import requestsresponse = requests.get("https://github.com/favicon.ico")
with open('favicon.ico', 'wb') as f: #保存图片 以二进制形式写f.write(response.content)f.close()

添加headers：

import requests
#知乎会识别User-Agent  不加headers会返回400
response = requests.get("https://www.zhihu.com/explore")
print(response.text)

import requests
#添加headers
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}
response = requests.get("https://www.zhihu.com/explore", headers=headers)
print(response.text)

基本post请求

import requests
#直接将上传的数据写成字典 不用转码  传给data参数
data = {'name': 'germey', 'age': '22'}
response = requests.post("http://httpbin.org/post", data=data)
print(response.text) #以form data形式上传

import requestsdata = {'name': 'germey', 'age': '22'}
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}
response = requests.post("http://httpbin.org/post", data=data, headers=headers)
print(response.json())#将返回结果转化为json

4. 响应

response属性

import requestsresponse = requests.get('http://www.jianshu.com')
print(type(response.status_code), response.status_code) #状态码
print(type(response.headers), response.headers) #响应头
print(type(response.cookies), response.cookies) #cookies
print(type(response.url), response.url)
print(type(response.history), response.history) #访问历史记录

状态码推断

import requestsresponse = requests.get('http://www.jianshu.com/hello.html') #一个不存在的网址
exit() if not response.status_code == requests.codes.not_found else print('404 Not Found')

也可以直接用整型编码值：

import requestsresponse = requests.get('http://www.jianshu.com')
exit() if not response.status_code == 200 else print('Request Successfully')

状态码对照表：

100: ('continue',),
101: ('switching_protocols',),
102: ('processing',),
103: ('checkpoint',),
122: ('uri_too_long', 'request_uri_too_long'),
200: ('ok', 'okay', 'all_ok', 'all_okay', 'all_good', '\\o/', '✓'),
201: ('created',),
202: ('accepted',),
203: ('non_authoritative_info', 'non_authoritative_information'),
204: ('no_content',),
205: ('reset_content', 'reset'),
206: ('partial_content', 'partial'),
207: ('multi_status', 'multiple_status', 'multi_stati', 'multiple_stati'),
208: ('already_reported',),
226: ('im_used',),# Redirection.
300: ('multiple_choices',),
301: ('moved_permanently', 'moved', '\\o-'),
302: ('found',),
303: ('see_other', 'other'),
304: ('not_modified',),
305: ('use_proxy',),
306: ('switch_proxy',),
307: ('temporary_redirect', 'temporary_moved', 'temporary'),
308: ('permanent_redirect','resume_incomplete', 'resume',), # These 2 to be removed in 3.0# Client Error.
400: ('bad_request', 'bad'),
401: ('unauthorized',),
402: ('payment_required', 'payment'),
403: ('forbidden',),
404: ('not_found', '-o-'),
405: ('method_not_allowed', 'not_allowed'),
406: ('not_acceptable',),
407: ('proxy_authentication_required', 'proxy_auth', 'proxy_authentication'),
408: ('request_timeout', 'timeout'),
409: ('conflict',),
410: ('gone',),
411: ('length_required',),
412: ('precondition_failed', 'precondition'),
413: ('request_entity_too_large',),
414: ('request_uri_too_large',),
415: ('unsupported_media_type', 'unsupported_media', 'media_type'),
416: ('requested_range_not_satisfiable', 'requested_range', 'range_not_satisfiable'),
417: ('expectation_failed',),
418: ('im_a_teapot', 'teapot', 'i_am_a_teapot'),
421: ('misdirected_request',),
422: ('unprocessable_entity', 'unprocessable'),
423: ('locked',),
424: ('failed_dependency', 'dependency'),
425: ('unordered_collection', 'unordered'),
426: ('upgrade_required', 'upgrade'),
428: ('precondition_required', 'precondition'),
429: ('too_many_requests', 'too_many'),
431: ('header_fields_too_large', 'fields_too_large'),
444: ('no_response', 'none'),
449: ('retry_with', 'retry'),
450: ('blocked_by_windows_parental_controls', 'parental_controls'),
451: ('unavailable_for_legal_reasons', 'legal_reasons'),
499: ('client_closed_request',),# Server Error.
500: ('internal_server_error', 'server_error', '/o\\', '✗'),
501: ('not_implemented',),
502: ('bad_gateway',),
503: ('service_unavailable', 'unavailable'),
504: ('gateway_timeout',),
505: ('http_version_not_supported', 'http_version'),
506: ('variant_also_negotiates',),
507: ('insufficient_storage',),
509: ('bandwidth_limit_exceeded', 'bandwidth'),
510: ('not_extended',),
511: ('network_authentication_required', 'network_auth', 'network_authentication'),

5. 高级操作

文件上传

import requests
#读取文件 保存为files字典
#传给files参数 使用post请求 上传文件
files = {'file': open('favicon.ico', 'rb')}
response = requests.post("http://httpbin.org/post", files=files)
print(response.text)

获取cookies

import requestsresponse = requests.get("https://www.baidu.com")
print(response.cookies)
for key, value in response.cookies.items():print(key + '=' + value)

会话维持

模拟登录，维持登录状态：

#维持登录状态
import requests
#两个request独立 没有关联
#相当于两次get在不同的两个浏览器中请求
requests.get('http://httpbin.org/cookies/set/number/123456789')#设置cookie
response = requests.get('http://httpbin.org/cookies') #获取cookie
print(response.text)

import requestss = requests.Session() #声明Session()
#相当于两次get在一个浏览器中请求
s.get('http://httpbin.org/cookies/set/number/123456789') #设置cookie
response = s.get('http://httpbin.org/cookies') #获取cookie
print(response.text)

证书验证

import requestsresponse = requests.get('https://www.12306.cn') #https 请求时会首先检查证书是否合法
print(response.status_code)

import requests#将verify设置为false 不进行证书验证
response = requests.get('https://www.12306.cn', verify=False) #此时会有警告信息
print(response.status_code)

from requests.packages import urllib3 #消除警告信息
urllib3.disable_warnings()response = requests.get('https://www.12306.cn', verify=False)
print(response.status_code)

import requests
#可以自行设置CA证书
response = requests.get('https://www.12306.cn', cert=('/path/server.crt', '/path/key'))
print(response.status_code)

代理设置

import requests
#代理保存为字典形式 传给proxies参数
proxies = {"http": "http://127.0.0.1:9743","https": "https://127.0.0.1:9743",
}response = requests.get("https://www.taobao.com", proxies=proxies)
print(response.status_code)

import requests
#如果代理有用户名和密码
proxies = {"http": "http://user:password@127.0.0.1:9743/",
}
response = requests.get("https://www.taobao.com", proxies=proxies)
print(response.status_code)

!pip install 'requests[socks]' #除http https之外的代理 如socks代理

import requestsproxies = {'http': 'socks5://127.0.0.1:9742','https': 'socks5://127.0.0.1:9742'
}
response = requests.get("https://www.taobao.com", proxies=proxies)
print(response.status_code)

超时设置

import requests
from requests.exceptions import ReadTimeout
#通过timeout参数可设置超时时间 超出时间未响应 会抛出异常
try:response = requests.get("http://httpbin.org/get", timeout = 0.2)print(response.status_code)
except ReadTimeout:print('Timeout')

认证设置

import requests
from requests.auth import HTTPBasicAuth
#有些网站 需要登录验证  通过登录后 才能请求网站 通过auth参数设置
r = requests.get('http://120.27.34.24:9001', auth=HTTPBasicAuth('user', '123'))
print(r.status_code)

import requests
#简写
r = requests.get('http://120.27.34.24:9001', auth=('user', '123'))
print(r.status_code)

6. 异常处理

import requests
from requests.exceptions import ReadTimeout, ConnectionError, RequestException
#异常捕获 先写子类 再写父类
try:response = requests.get("http://httpbin.org/get", timeout = 0.1)print(response.status_code)
except ReadTimeout: #访问超时print('Timeout')
except ConnectionError: #访问不通print('Connection error')
except RequestException: #父类异常print('Error')

Python爬虫理论Pro | (3) Requests库详解相关推荐

python基础代码库-python爬虫基础教程：requests库（二）代码实例
get请求简单使用 import requests ''' 想要学习Python?Python学习交流群:973783996满足你的需求,资料都已经上传群文件,可以自行下载! ''' respons ...
Python爬虫初级（十一）—— Selenium 详解
欢迎关注公众号K的笔记阅读博主更多优质学习内容上一篇内容:Python爬虫初级(九)-- ajax 详解 Selenium 库的安装 Selenium 的安装比起其他 python 库的安装稍显复杂 ...
爬虫笔记：Requests库详解
什么是Requests 之前讲解了爬虫笔记:Urllib库详解发现确实有不方便的地方,比如加一个代理,cookie,发送post请求比较繁琐. Request库能用几句话实现这些. Requests ...
Python 爬虫---（5）Requests库的使用
什么是Requests Requests是用python语言基于urllib编写的,采用的是Apache2 Licensed开源协议的HTTP库如果你看过上篇文章关于urllib库的使用,你会发现, ...
python gpu加速库比matlab快吗_Python之Unittest和Requests库详解
1.按类来执行 import unittest class f1(unittest.TestCase): def setUp(self): pass def tearDown(self): pass ...
python requests库详解_python爬虫之路（一）-----requests库详解
requests库 requests库是python实现的最简单易用的http库. requests库的功能详解. 我们可以自然而然地想到这些方法其实就是http协议对资源的操作. 调用request ...
【python】python爬虫requests库详解
1.安装:pip install requests 简介:Requests是一个优雅而简单的Python HTTP库,与之前的urllibPython的标准库相比,Requests的使用方式非常的简单 ...
[python爬虫之路dya3]: requests库的基本使用
前面我们学习了urllib库进行源代码的爬取,今天来介绍更加人性化的requests库的使用. import requests '''response=requests.get("https ...
python 爬虫学习入门6 requests库添加代理proxies等其他信息
Requests库通过Requests 库提供的方法我们可以在请求页面的时候同时添加更多的信息,在urllib库中同时添加代理和User-agent 我在网上找了一些办法但是没有理解就不在这里说了 ...

Python爬虫理论Pro | (3) Requests库详解

1. 什么是requests

2. 实例引入

3. 请求

4. 响应

5. 高级操作

6. 异常处理

Python爬虫理论Pro | (3) Requests库详解相关推荐

最新文章

热门文章