zoukankan      html  css  js  c++  java
  • python asyncio 使用ThreadPoolExecutor和asyncio完成阻塞IO请求

    #使用多线程:在协程中集成阻塞io
    import asyncio
    from concurrent.futures import ThreadPoolExecutor
    import socket
    from urllib.parse import urlparse
    
    
    def get_url(url):
        #通过socket请求html
        url = urlparse(url)
        host = url.netloc
        path = url.path
        if path == "":
            path = "/"
    
        #建立socket连接
        client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        # client.setblocking(False)
        client.connect((host, 80)) #阻塞不会消耗cpu
    
        #不停的询问连接是否建立好, 需要while循环不停的去检查状态
        #做计算任务或者再次发起其他的连接请求
    
        client.send("GET {} HTTP/1.1
    Host:{}
    Connection:close
    
    ".format(path, host).encode("utf8"))
    
        data = b""
        while True:
            d = client.recv(1024)
            if d:
                data += d
            else:
                break
    
        data = data.decode("utf8")
        html_data = data.split("
    
    ")[1]
        print(html_data)
        client.close()
    
    
    # 如果在协程中要用到阻塞IO,就把他放到线程池里面去运行,在运行的时候,本质还是线程池,同步的
    if __name__ == "__main__":
        import time
        start_time = time.time()
        loop = asyncio.get_event_loop()
        executor = ThreadPoolExecutor(3)
        tasks = []
        for url in range(20):
            url = "http://shop.projectsedu.com/goods/{}/".format(url)
            task = loop.run_in_executor(executor, get_url, url)
            tasks.append(task)
        loop.run_until_complete(asyncio.wait(tasks))
        print("last time:{}".format(time.time()-start_time))
  • 相关阅读:
    Ubuntu12下未知驱动器处理
    Octopress博客设置
    Windows下搭建Octopress博客
    在Asp.Net中使用JQueryEasyUI
    SQL查询将列转换成字符串(for xml path)
    IIS7上传出现乱码问题解决
    SqlServer开发利器—SQL Prompt5
    读《打造FaceBook》
    在VS2010中使用Git【图文】
    怎样提高开发效率
  • 原文地址:https://www.cnblogs.com/callyblog/p/11220572.html
Copyright © 2011-2022 走看看