zoukankan      html  css  js  c++  java
  • 异步 携程 网络小爬虫


    from urllib import request
    import gevent,time
    from gevent import monkey
    monkey.patch_all() #把当前程序的所有的io操作给我单独的做上标记

    def f(url):
    print('GET: %s' % url)
    resp = request.urlopen(url)
    data = resp.read()
    print('%d bytes received from %s.' % (len(data), url))

    urls = ['https://www.python.org/',
    'https://www.yahoo.com/',
    'https://github.com/' ]
    time_start = time.time()
    for url in urls:
    f(url)
    print("同步cost",time.time() - time_start)
    async_time_start = time.time()
    gevent.joinall([
    gevent.spawn(f, 'https://www.python.org/'),
    gevent.spawn(f, 'https://www.yahoo.com/'),
    gevent.spawn(f, 'https://github.com/'),
    ])
    print("异步cost",time.time() - async_time_start)


    GET: https://www.python.org/
    50114 bytes received from https://www.python.org/.
    GET: https://www.yahoo.com/
    505000 bytes received from https://www.yahoo.com/.
    GET: https://github.com/
    65396 bytes received from https://github.com/.
    同步cost 3.5022003650665283
    GET: https://www.python.org/
    GET: https://www.yahoo.com/
    GET: https://github.com/
    65396 bytes received from https://github.com/.
    50114 bytes received from https://www.python.org/.
    504996 bytes received from https://www.yahoo.com/.
    异步cost 1.332076072692871

  • 相关阅读:
    阿里云nginx创建多站点
    linux 卸载php mysql apache
    centos php环境搭建
    jquery simple modal
    nodejs 安装express
    nodejs fs.open
    nodejs supervisor
    nodejs 运行
    nodejs shell
    PHP array_pad()
  • 原文地址:https://www.cnblogs.com/rongye/p/9983491.html
Copyright © 2011-2022 走看看