scrapy 报错twisted.webnewclient.ResponseNeverReceived。。。及附上请求头大全
网上除了一篇17年的论坛帖，其他关于这个问题的博客、帖子，基本都是从这篇博文直接摘抄过去的，所以，若是你看到这篇博文，其他基本可以不用去看了
报错代码：
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion.>]
报错代码解读：
与目标网站的连接丢失，无法从目标网站获取响应(response)
报错原因：
原因1：程序自身问题：爬虫程序被识别出来，识别特征有:没有合适的请求头，被直接ban掉；请求频率过快，被识别;使用selenium等自动化测试工具其中navigator等被识别出来.....其他特征等,这类问题，一般只需要提高自身伪装就行，比如降低访问频率.啥的。
原因2：网站自身原因，网站宽带不够，网站响应慢，也是会触发的，这类问题，爬虫程序唯一方法就是添加异常机制：一直retry,没有其他方法了。
像评论说的，文章基本到这就可以结束了。
添加一个，若是请求头问题，解决方法：
加上请求头即可
方法一:
利用Python现有的库
fake_useragent使用方法:
from fake_useragent import UserAgent
ua = UserAgent(use_cache_server=False,verify_ssl=False)方法二:
下面附上一个User-agent请求头大全：
{"User-Agent":random.choice(['Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6','Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER','Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11','Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)','Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20','Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6','Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)','Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.12) Gecko/20070731 Ubuntu/dapper-security Firefox/1.5.0.12','Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1','Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.3 Mobile/14E277 Safari/603.1.30','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'])其中：：
Opera
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60
Opera/8.0 (Windows NT 5.1; U; en)
Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.50
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 9.50
Firefox
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0
Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10
Safari
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2
chrome
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16
360
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
淘宝浏览器
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11
猎豹浏览器
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)"
QQ浏览器
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)
sogou浏览器
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; SE 2.X MetaSr 1.0)
maxthon浏览器
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.4.3.4000 Chrome/30.0.1599.101 Safari/537.36
UC浏览器
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 UBrowser/4.0.3214.0 Safari/537.36
==================== 移动浏览器大全 ====================
IPhone
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5
IPod
Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5
IPAD
Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5
Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5
Android
Mozilla/5.0 (Linux; U; Android 2.2.1; zh-cn; HTC_Wildfire_A3333 Build/FRG83D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1
Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1
QQ浏览器 Android版本
MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1
Android Opera Mobile
Opera/9.80 (Android 2.3.4; Linux; Opera Mobi/build-1107180945; U; en-GB) Presto/2.8.149 Version/11.10
Android Pad Moto Xoom
Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13
BlackBerry
Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; en) AppleWebKit/534.1+ (KHTML, like Gecko) Version/6.0.0.337 Mobile Safari/534.1+
WebOS HP Touchpad
Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.0; U; en-US) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/233.70 Safari/534.6 TouchPad/1.0
Nokia N97
Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/20.0.019; Profile/MIDP-2.1 Configuration/CLDC-1.1) AppleWebKit/525 (KHTML, like Gecko) BrowserNG/7.1.18124
Windows Phone Mango
Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan)
UC浏览器
UCWEB7.0.2.37/28/999
NOKIA5700/ UCWEB7.0.2.37/28/999
UCOpenwave
Openwave/ UCWEB7.0.2.37/28/999
UC Opera
Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999
scrapy 报错twisted.webnewclient.ResponseNeverReceived。。。及附上请求头大全相关推荐
- 解决：building 'twisted.test.raiser' extension安装scrapy报错
解决:building 'twisted.test.raiser' extension error: Microsoft Visual C++ 14.0 is required. Get it ...
- 安装scrapy报错
安装scrapy报错 win10 pycharm virtualenv python3.5 (env) C:\Users\lg\PycharmProjects\untitled>pip3 ins ...
- 安装scrapy报错failed with error code 1 in C:\Users\ADMINI~\AppData\Local\Temp\pip-install-4447ozxq\Twist
安装scrapy的时候报错安装scrapy报错failed with error code 1 in C:\Users\ADMINI~1\AppData\Local\Temp\pip-install- ...
- Deepin下安装scrapy报错fatal error: Python.h: 没有那个文件或目录
Deepin下安装scrapy报错(亲测有效!) 问题描述 在deepin操作系统上,使用python3.6,通过sudo pip3 install scrapy安装时候,出现以下错误: src/tw ...
- CMD终端关于pip报错,scrapy报错的一种处理方法
CMD终端关于pip报错,scrapy报错的一种处理方法 如果在终端输入pip,或scrapy,报如下错误: Fatal error in launcher: Unable to create pro ...
- Python3导入scrapy报错1 in C:\Users\ADMINI~1\AppData\Local\Temp\pip-install-831gxniz\Twisted\
在用scrapy框架实现爬虫时,scrapy导包报错: Command "D:\Python\Anaconda\python.exe -u -c "import setuptool ...
- pip install scrapy 报错：缺少Twisted 解决办法 ：error: Microsoft Visual C++ 14.0 is required
报错如下: 报错的最后会显示类似缺少 C++ 14.0之类的 error: Microsoft Visual C++ 14.0 is required. Get it with "Micro ...
- 安装scrapy报错failed with error code 1 in C:\Users\ADMINI~1\AppData\Local\Temp\pip-install-dvrc8k5o\Twis
安装scrapy的时候报错failed with error code 1 in C:\Users\ADMINI~1\AppData\Local\Temp\pip-install-dvrc8k5o\T ...
- Linux easy_install scrapy 报错 Not a recognized archive type: /tmp/easy_install-uem5ldyz/Twisted-18.9.
(本人用的服务器是CentOS7.3, python版本是3.7.0,自己的服务器用的是root用户,哈哈!) 按照正常步骤安装scrapy 1.首先升级yum源 sudo yum update 2. ...
最新文章
- Hinton 发表新作探索流体胶囊网络 | AI日报
- 腾讯首个AI开源项目完成3.0版本迭代，从顶级开源基金会毕业
- Error: A JNI error has occurred, please check your installation and try again
- ajax绑值,vue.js使用ajax绑定数据之post方法
- pycharm pro版本激活
- 3207: 花神的嘲讽计划Ⅰ
- 程序员职场-三人行,必有我师
- 英语六级口语 计算机,英语六级口语
- robot framework RIDE实现鼠标拖拽
- 数据库理论——概念模型、逻辑模型、物理模型
- 时间序列预测——ARIMA
- [NOIP1999]Cantor表
- 电脑的字体改变了、变模糊了怎么办
- 基于RK3399的LED驱动开发
- 5.2-国内知识图谱相关企业和资源
- 【TI-mmWave】三、CCS Projects导入报错：This project requires product SYS/BIOS v6.73.1.01, or equivalent......
- 锐捷——RIP基础配置
- 浅谈大数据时代的大数据技术与应用
- 一维向量转换为n维向量_如何在N维上固定万向节锁
- 最小二乘法直线拟合c语言代码,最小二乘法拟合一条直线(C语言代码)