python语言如何使用隧道爬虫ip

做过大数据抓取的技术员应该都知道，正常市面上的爬虫ip只分为两种，一种是API提取式的，还有一种是账密形式隧道模式的。那边在做数据抓取的业务时候用python语言如果使用隧道爬虫ip？需要学习爬虫的小白可以过来看看。

import base64import timeimport requestsfrom requests.adapters import HTTPAdapterauth_key = "B8998D3E"  # 隧道代理的AuthKeypassword = "xxxx"  # 隧道代理的AuthPwdtunnel_server = "http://jshk.com.cn"  # 隧道代理的地址target_url = "https://d.qg.net/ip"  # 要访问的目标地址proxy_headers = {}proxy = {"http": tunnel_server,"https": tunnel_server}def encode_authorization(key, passwd):# python 使用 bytes 类型进行 base64 编码basic_str = bytes("%s:%s" % (key, passwd), "ascii")# 得到的返回值也是 bytes 类型，所以需要再 decode 为字符串return "Basic %s" % base64.b64encode(basic_str).decode("utf-8")def reset_tunnel_proxy_headers():global proxy_headersproxy_headers = {tunnel_server: {"Proxy-Authorization": encode_authorization(auth_key, password)}}def update_tunnel_proxy_headers(key, val):global proxy_headersproxy_headers[tunnel_server][key] = valdef new_session():adapter = TunnelProxyAdapter()se = requests.Session()se.mount('https://', adapter)se.mount('http://', adapter)return seclass TunnelProxyAdapter(requests.adapters.HTTPAdapter):def proxy_headers(self, p):if p in proxy_headers:print("session with headers:", proxy_headers[p])return proxy_headers[p]else:return Nonedef normal_tunnel():"""结果类似:request on normal modesession with headers: {'Proxy-Authorization': 'Basic xxxx'}request id: 1, code: 200, result: 140.250.149.229"""reset_tunnel_proxy_headers()print("request on normal mode")resp = new_session().get(target_url, proxies=proxy)print("request id: 1, code: %s, result: %s" % (resp.status_code, resp.text))def mark_tunnel():"""结果类似：request with marksession with headers: {'Proxy-Authorization': 'Basic xxxx', 'Proxy-TunnelID': 'channel-1', 'Proxy-TTL': 10}request id: 1 , code: 200, result: 183.166.118.48request id: 2 , code: 200, result: 183.166.118.48request id: 3 , code: 200, result: 183.166.118.48request id: 4 , code: 200, result: 183.166.118.48request id: 5 , code: 200, result: 183.166.118.48request id: 6 , code: 200, result: 183.166.118.48request id: 7 , code: 200, result: 183.142.59.203request id: 8 , code: 200, result: 183.142.59.203request id: 9 , code: 200, result: 183.142.59.203request id: 10, code: 200, result: 123.54.235.89"""reset_tunnel_proxy_headers()update_tunnel_proxy_headers("Proxy-TunnelID", "channel-1")update_tunnel_proxy_headers("Proxy-TTL", 10)se = new_session()print("request with mark")for i in range(1, 12):resp = se.get(target_url, proxies=proxy, headers={"Connection": "close"})print("request id: %-2s, code: %s, result: %s" % (i, resp.status_code, resp.text))time.sleep(1)def multi_channel_tunnel():"""结果类似：request on multi channelrequest id: 1 , channel id: channel-1, code: 200, result: 183.155.88.224request id: 2 , channel id: channel-2, code: 200, result: 125.112.38.153request id: 3 , channel id: channel-3, code: 200, result: 183.155.89.125request id: 4 , channel id: channel-4, code: 200, result: 49.71.121.169request id: 5 , channel id: channel-5, code: 200, result: 115.210.67.220request id: 6 , channel id: channel-6, code: 200, result: 36.25.41.178request id: 7 , channel id: channel-7, code: 200, result: 180.125.162.116request id: 8 , channel id: channel-8, code: 200, result: 140.250.150.158request id: 9 , channel id: channel-9, code: 200, result: 121.227.102.227request id: 10, channel id: channel-10, code: 200, result: 49.88.106.198request id: 1 , channel id: channel-1, code: 200, result: 183.155.88.224request id: 2 , channel id: channel-2, code: 200, result: 125.112.38.153request id: 3 , channel id: channel-3, code: 200, result: 183.155.89.125request id: 4 , channel id: channel-4, code: 200, result: 49.71.121.169request id: 5 , channel id: channel-5, code: 200, result: 115.210.67.220request id: 6 , channel id: channel-6, code: 200, result: 36.25.41.178request id: 7 , channel id: channel-7, code: 200, result: 180.125.162.116request id: 8 , channel id: channel-8, code: 200, result: 140.250.150.158request id: 9 , channel id: channel-9, code: 200, result: 121.227.102.227request id: 10, channel id: channel-10, code: 200, result: 49.88.106.198"""print("request on multi channel")reset_tunnel_proxy_headers()for i in range(1, 11):se = new_session()chan_id = "channel-%s" % iupdate_tunnel_proxy_headers("Proxy-TunnelID", chan_id)resp = se.get(target_url, proxies=proxy, headers={"Connection": "close"})print("request id: %-2s, channel id: %s, code: %s, result: %s" % (i, chan_id, resp.status_code, resp.text))time.sleep(10)# 因为固定时长为1分钟，所以在1分钟内继续使用已有通道，仍是之前的IPfor i in range(1, 11):se = new_session()chan_id = "channel-%s" % iupdate_tunnel_proxy_headers("Proxy-TunnelID", chan_id)resp = se.get(target_url, proxies=proxy, headers={"Connection": "close"})print("request id: %-2s, channel id: %s, code: %s, result: %s" % (i, chan_id, resp.status_code, resp.text))if __name__ == "__main__":normal_tunnel()mark_tunnel()multi_channel_tunnel()

python语言如何使用隧道爬虫ip相关推荐

Python：Python语言的简介(语言特点/pyc介绍/Python版本语言兼容问题(python2 VS Python3))、安装、学习路线(数据分析/机器学习/网页爬等编程案例分析)之详细攻略
Python:Python语言的简介(语言特点/pyc介绍/Python版本语言兼容问题(python2 VS Python3)).安装.学习路线(数据分析/机器学习/网页爬等编程案例分析)之详细攻略 ...
Python语言的简介(语言特点/pyc介绍/Python版本语言兼容问题(python2 VS Python3))、安装、学习路线(数据分析/机器学习/网页爬等编程案例分析)之详细攻略
目录 Python语言的简介 1.Python的应用领域 2.Python语言特点.对比其它语言 2.1.Python语言特点 2.2.Python语言对比其它语言 3.Python版本语言兼容问题( ...
Dataset之MNIST：MNIST(手写数字图片识别+ubyte.gz文件)数据集的下载(基于python语言根据爬虫技术自动下载MNIST数据集)
Dataset之MNIST:MNIST(手写数字图片识别+ubyte.gz文件)数据集的下载(基于python语言根据爬虫技术自动下载MNIST数据集) 目录数据集下载的所有代码 1.主文件 mni ...
使用Python语言开发爬虫有什么优势？
网络爬虫分为很多种,Python爬虫也是其中的一种,那么使用Python语言开发爬虫有什么优势呢?来看看下面的详细介绍. Python分享:使用Python语言开发爬虫有什么优势?截止到目前,网络爬虫 ...
爬虫python代码广告_零基础掌握百度地图兴趣点获取POI爬虫（python语言爬取）（代码篇）...
我是怎么想的,在新浪博客里写代码教程. 这篇博客的内容同步到了CSND博客中,那里不限制外链,也可以复制代码. http://blog.csdn.net/sinat_41310868/article/ ...
零基础掌握百度地图兴趣点获取POI爬虫（python语言爬取）（代码篇）
好,现在进入高阶代码篇. 目的: 爬取昆明市中学的兴趣点POI. 关键词:中学已有ak:9s5GSYZsWbMaFU8Ps2V2VWvDlDlqGaaO 昆明市坐标范围: 左下角:24.390894 ...
python语言-实现半自动爬虫爬取小说《遮天》评论
半自动爬虫-新手作业要求:百度贴吧中寻找一个自己喜欢的贴吧,将其中的一片热门帖子的每一层楼的发帖人.发帖内容和发帖时间抓取下来需要准备的是,能够理解正则表达式的运用,基础的python语法就可以很 ...
基于python语言，使用爬虫和深度学习来预测双色球（二、模型训练与测试）
在上一篇博文中(基于python语言,使用爬虫和深度学习来预测双色球(一.数据的准备)),我们完成了数据的准备,通过爬虫的方式将2003年至今的每一期的中奖数据保存到了txt文件中,那么我们现在就开始 ...
c语言实现爬虫功能,用C/C 扩展Python语言_python 调用c语言 python实现简单爬虫功能_python实现简单爬虫...
用C/C 扩展Python语言 Python是一门功能强大的脚本语言,它的强大不仅表现在功能上,还表现在其扩展性上.她提供大量的API以方便程序员利用C/C++对Python进行扩展.因为执行速度慢几 ...

python语言如何使用隧道爬虫ip

python语言如何使用隧道爬虫ip相关推荐

最新文章

热门文章