zoukankan      html  css  js  c++  java
  • python 多协程异步IO爬取网页加速3倍。

     1 from urllib import request
     2 import gevent,time
     3 from gevent import monkey#该模块让当前程序所有io操作单独标记,进行异步操作。
     4 
     5 monkey.patch_all()#对当前程序的io操作打上补丁。没有该monkey方法,异步IO无效。
     6 def  f(url):
     7     print('GET:%s'%url)
     8     resp = request.urlopen(url)#获取网页
     9     data = resp.read()#读取网页
    10     print('%d bytes received from  %s'%(len(data),url))#打印长度
    11 url = ['https://www.yahoo.com/','https://www.python.org/',
    12  'https://github.com/']
    13 start = time.time()
    14 for i in url:
    15     f(i)#循环运行列表中的网页
    16 print('串行执行时间:',time.time() - start)#串行执行时间
    17 async_time = time.time()
    18 gevent.joinall([
    19     gevent.spawn(f,'https://www.yahoo.com/')#异步执行启动协程
    20 ,   gevent.spawn(f,'https://www.python.org/'),
    21     gevent.spawn(f,'https://github.com/'),
    22 ])
    23 print('异步执行时间async time:',time.time() - async_time)#多协程异步IO执行时间


    以下为运行结果,明显多协程的牛逼之处。。。。。。。如果不执行monkey方法,则异步IO就会按串行执行。
    C:UsershushuningAnaconda3python.exe C:/Users/hushuning/PycharmProjects/untitled/njx/把当前程序的所有的io操作单独标记,进行异步操作.py
    GET:https://www.yahoo.com/
    510125 bytes received from  https://www.yahoo.com/
    GET:https://www.python.org/
    48857 bytes received from  https://www.python.org/
    GET:https://github.com/
    51373 bytes received from  https://github.com/
    串行执行时间: 4.710935354232788
    GET:https://www.yahoo.com/
    GET:https://www.python.org/
    GET:https://github.com/
    48857 bytes received from  https://www.python.org/
    512422 bytes received from  https://www.yahoo.com/
    51373 bytes received from  https://github.com/
    异步执行时间async time: 1.6521050930023193
    
    Process finished with exit code 0
    
    
    
     
  • 相关阅读:
    C++ 四种cast 用法
    Wannafly挑战赛1 B Xorto
    python里的闭包
    编译器对类的编译顺序
    class和struct
    typedef类型别名
    decltype类型指示符
    左值和右值
    const限定符
    hdu5678 树上第k小
  • 原文地址:https://www.cnblogs.com/hushuning/p/7922220.html
Copyright © 2011-2022 走看看