zoukankan      html  css  js  c++  java
  • python asyncio 使用ThreadPoolExecutor和asyncio完成阻塞IO请求

    #使用多线程:在协程中集成阻塞io
    import asyncio
    from concurrent.futures import ThreadPoolExecutor
    import socket
    from urllib.parse import urlparse
    
    
    def get_url(url):
        #通过socket请求html
        url = urlparse(url)
        host = url.netloc
        path = url.path
        if path == "":
            path = "/"
    
        #建立socket连接
        client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        # client.setblocking(False)
        client.connect((host, 80)) #阻塞不会消耗cpu
    
        #不停的询问连接是否建立好, 需要while循环不停的去检查状态
        #做计算任务或者再次发起其他的连接请求
    
        client.send("GET {} HTTP/1.1
    Host:{}
    Connection:close
    
    ".format(path, host).encode("utf8"))
    
        data = b""
        while True:
            d = client.recv(1024)
            if d:
                data += d
            else:
                break
    
        data = data.decode("utf8")
        html_data = data.split("
    
    ")[1]
        print(html_data)
        client.close()
    
    
    # 如果在协程中要用到阻塞IO,就把他放到线程池里面去运行,在运行的时候,本质还是线程池,同步的
    if __name__ == "__main__":
        import time
        start_time = time.time()
        loop = asyncio.get_event_loop()
        executor = ThreadPoolExecutor(3)
        tasks = []
        for url in range(20):
            url = "http://shop.projectsedu.com/goods/{}/".format(url)
            task = loop.run_in_executor(executor, get_url, url)
            tasks.append(task)
        loop.run_until_complete(asyncio.wait(tasks))
        print("last time:{}".format(time.time()-start_time))
  • 相关阅读:
    重置csr
    修改node节点名称
    k8s报错解决
    k8s测试容器之间是否互通
    MyEclipse------executeBatch()使用方法
    MyEclipse------execute()使用方法
    MyEclipse------如何查询MySQL数据库里面表的信息
    MyEclipse------如何连接MySQL
    MyEclipse------从服务器下载文件
    MyEclipse------各种问题解决方法
  • 原文地址:https://www.cnblogs.com/callyblog/p/11220572.html
Copyright © 2011-2022 走看看