python异步处理请求_Python 异步请求
1. 正常下载漫画,requests 阻塞式。
import requests, bs4, time
import asyncio, aiohttp
def get_image_urls(url):
res = requests.get(url)
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'lxml')
imgs = soup.select('div.list img')
for img in imgs:
yield img.get('data-kksrc')
def download_image(image_url, filename):
res = requests.get(image_url)
res.raise_for_status()
image = res.content
with open(r'e:\path\mh\{}.jpg'.format(filename), 'wb') as fp:
fp.write(image)
if __name__ == '__main__':
site_url = 'https://www.kuaikanmanhua.com/web/comic/105372/'
t_url = 'https://www.kuaikanmanhua.com/web/comic/15559/'
image_urls = [image_url for image_url in get_image_urls(site_url)]
t_list = [image_url for image_url in get_image_urls(t_url)]
image_urls.extend(t_list)
start = time.time()
for filename, url in enumerate(image_urls, start=1):
download_image(url, filename)
print('总计:{:.2f}s'.format(time.time()-start))
2. 异步包 asyncio 基本用法
import asyncio
import time
async def asynct():
print('asyncio...')
await asyncio.sleep(1)
def main():
start = time.time()
loop = asyncio.get_event_loop()
tasks = [asynct() for i in range(10)]
tasks_wait = asyncio.wait(tasks)
loop.run_until_complete(tasks_wait)
loop.close()
print('total: {:.2f}s'.format(time.time()-start))
if __name__ == '__main__':
main()
结果:
asyncio...
asyncio...
asyncio...
asyncio...
asyncio...
asyncio...
asyncio...
asyncio...
asyncio...
asyncio...
total: 1.01s
asyncio.sleep(1) 并不会阻塞运行,而是返回主线程继续运行,所以不会等待 10s 时间。
3. 异步下载漫画
import asyncio, aiohttp
import requests, bs4
import time
import tqdm
import concurrent.futures as futures
async def download_images(url, filename):
async with aiohttp.ClientSession() as session:
async with session.get(url) as res:
image = await res.content.read()
with open(r'e:\path\mh\{}.jpg'.format(filename), 'wb') as fp:
fp.write(image)
def test():
site_url = 'https://www.kuaikanmanhua.com/web/comic/105372/'
t_url = 'https://www.kuaikanmanhua.com/web/comic/15559/'
image_urls = [image_url for image_url in get_image_urls(site_url)]
t_list = [image_url for image_url in get_image_urls(t_url)]
image_urls.extend(t_list)
start = time.time()
loop = asyncio.get_event_loop()
tasks = [download_images(url, filename) for filename, url in enumerate(image_urls, start=1)]
coro = asyncio.wait(tasks)
loop.run_until_complete(coro)
loop.close()
print('总计:{:.2f}s'.format(time.time() - start))
if __name__ == '__main__':
test()
4. 以线程方式异步请求
def download_image(image_url, filename):
res = requests.get(image_url)
res.raise_for_status()
image = res.content
with open(r'e:\path\mh\{}.jpg'.format(filename), 'wb') as fp:
fp.write(image)
def test1():
site_url = 'https://www.kuaikanmanhua.com/web/comic/105372/'
t_url = 'https://www.kuaikanmanhua.com/web/comic/15559/'
image_urls = [image_url for image_url in get_image_urls(site_url)]
t_list = [image_url for image_url in get_image_urls(t_url)]
image_urls.extend(t_list)
start = time.time()
loop = asyncio.get_event_loop()
for filename, url in enumerate(image_urls, start=1):
loop.run_in_executor(None, download_image, url, filename)
print('总计:{:.2f}s'.format(time.time() - start))
if __name__ == '__main__':
test1()
这种方式是最快的,244张漫画图片只需要 0.11s。上面的方式大约都要 10s,奇怪的是 3 异步下载需要 9s 多,并没有比正常下载快多少。
5. 异步下载加入进度条,更好的观察下载进度
async def download_images(url, filename):
async with aiohttp.ClientSession() as session:
async with session.get(url) as res:
image = await res.content.read()
with open(r'e:\path\mh\{}.jpg'.format(filename), 'wb') as fp:
fp.write(image)
async def download_images_tqdm(image_urls):
tasks = [download_images(url, filename) for filename, url in enumerate(image_urls, start=1)]
fs = asyncio.as_completed(tasks)
tqdm_iter = tqdm.tqdm(fs, total=len(image_urls))
for future in tqdm_iter:
await future
def test2():
site_url = 'https://www.kuaikanmanhua.com/web/comic/105372/'
t_url = 'https://www.kuaikanmanhua.com/web/comic/15559/'
image_urls = [image_url for image_url in get_image_urls(site_url)]
t_list = [image_url for image_url in get_image_urls(t_url)]
image_urls.extend(t_list)
start = time.time()
loop = asyncio.get_event_loop()
loop.run_until_complete(download_images_tqdm(image_urls))
loop.close()
print('总计:{:.2f}s'.format(time.time() - start))
if __name__ == '__main__':
test2()
6. 使用 concurrent.futures 线程方式请求
def download_image2(*args):
res = requests.get(args[0][1])
res.raise_for_status()
image = res.content
with open(r'e:\path\mh\{}.jpg'.format(args[0][0]), 'wb') as fp:
fp.write(image)
def test3():
site_url = 'https://www.kuaikanmanhua.com/web/comic/105372/'
t_url = 'https://www.kuaikanmanhua.com/web/comic/15559/'
image_urls = [image_url for image_url in get_image_urls(site_url)]
t_list = [image_url for image_url in get_image_urls(t_url)]
image_urls.extend(t_list)
start = time.time()
with futures.ThreadPoolExecutor(max_workers=16) as executor:
executor.map(download_image2, enumerate(image_urls, start=1))
print('总计:{:.2f}s'.format(time.time() - start))
if __name__ == '__main__':
test3()
这种方式根据 max_workers 最大线程数设置不同值,得到的结果差距在 1s 左右,总共耗时 3.95s,比异步请求要快。
总结:
各运行两次
正常下载: 9.44s, 9.87s
异步下载: 9.54s, 3.58s
线程方式异步下载: 0.11s, 0.23s
线程方式下载: 3.87s, 4.32s
最快且相对稳定的是 线程方式异步下载
python异步处理请求_Python 异步请求相关推荐
- python网络编程实战_Python 异步网络编程实战
近年来 Python 的发展的非常迅速,"简单"."高效"是 Python 吸引人的一大特色.在国内 Python 开发需求越来越大,Python 具有丰富强大 ...
- python异步处理请求_python异步处理HTTP请求
我需要为django查询集的每个条目生成一个PDF报告.会有3万到4万个条目.在 PDF是通过外部API生成的.由于当前是按需生成的,因此这是通过HTTP请求/响应同步处理的. 对于这个任务,这是不同 ...
- python异步和多线程_Python 异步 IO(asyncio)、多进程、多线程性能对比
IO 密集型应用 IO 密集型应用CPU等待IO时间远大于CPU 自身运行时间,太浪费:常见的 IO 密集型业务包括:浏览器交互.磁盘请求.网络爬虫.数据库请求等 image.png Python 世 ...
- python库下载超时_Python的请求库超时,但是得到了响应
我正在尝试为nba数据创建一个web刮刀.当我运行以下代码时:import requests response = requests.get('https://stats.nba.com/stats/ ...
- python延时一秒_python – 每个请求延迟1秒,不足以达到每小时3600
亚马逊API限制显然是每秒1 req或每小时3600.所以我实现了它: while True: #sql stuff time.sleep(1) result = api.item_lookup(ro ...
- python异步和进程_Python异步缓冲区和进程d
您希望能够尽快处理进入Reader的所有数据,但也不能让多个线程/进程尝试并行处理这些数据:这就是以前使用执行器遇到竞争条件的原因.相反,您应该启动一个可以处理所有数据包数据的工作进程,一次一个,使用 ...
- python的异步网络编程_python异步网络编程怎么使socket关闭之后立即执行一段代码?...
import socket from _socket import getdefaulttimeout class MySocket(socket.socket): """ ...
- IOS 同步请求和异步请求 ios开发教程
2019独角兽企业重金招聘Python工程师标准>>> 下课了,坐下来静静的总结一下今天学的内容 首先是 同步请求和异步请求 同步请求, 使用主线程一次性获得所有请求数据,这就导致一 ...
- python 异步加载_Python学习笔记4——爬取异步加载数据
一.什么是异步加载? 在之前的学习笔记中,爬取的网页是需要手动翻页的网址,但是一些网站是通过自动加载翻页的,如knewone网页.浏览knewone的官网就能发现,当下拉到网页最下端时,网站会自动加载 ...
最新文章
- AndroidStudio3.0无法打开Android Device Monitor的解决办法(An error has occurred on Android Device Monitor)...
- python【数据结构与算法】深入浅出Linear(线性表)
- vc++ mfc 非客户区 描绘线_决战客户端技术
- yum 卸载php及依赖包,yum使用指南-软件卸载、安装、更新、获取软件包
- Make Even(800)
- Android基于mAppWidget实现手绘地图(三)--环境搭建
- \r,\n,\r\n
- CTO职场解惑指南系列(一)
- 北京欢迎你 (歌词)
- scala method入门到熟悉
- 上平台! 车联网智能化晋级高段位!
- 高中数学怎么学好学好高中数学的技巧
- 快递单号物流查询自动识别接口【快递鸟API通用教程】
- 认证 (authentication) 和授权 (authorization) 的区别
- 如何恢复Juniper防火墙管理员密码,手把手教你
- Excel xlsx file; not supported
- win平板计算机,2020年10款最佳Windows系统平板电脑推荐
- 霸占热搜!官方下场发放免单攻略,饿了么营销如何抓住“薅羊毛”心理?
- 服务器出现HTTP400、401、402、403错误原因及解决方法
- unity技术补全计划
热门文章
- 易达物流管理软件 v2007 官网
- plc实验报告流程图_(新编)电气控制与PLC实验实验报告(0401)
- xcode 上传app商店流程记录
- 用 Astropy 拟合数据(一)
- 《缠中说禅108课》12:一吻何能消魂?
- AR的未来在哪 天猫AR“拜年”互动指明方向
- CAD二次开发---python中pyautocad怎么打开文件不提示对话框?
- 聊一款简单且精妙的微前端框架 ice stark(上)
- Server returns invalid timezone
- 《风火之旅》“首部资料片”由你做主!