今日内容
1.线程池和进程池
2.利用线程池实现套接字并发通信
3.协程(利用模块gevent模块,实现单线程下套接字并发通信)
1.线程池与进程池
要用线程池与进程池,首先要导入concurrent.futures这个模块,线程池和进程池规定了产生的线程或者进程的个数,保证计算机不会因为产生过多的进程和线程而导致崩溃,因为如果一台计算机他只能开启500个进程,而现在有1000个任务要运行,若我们让原本只能开启500个进程的计算机现在为了提高效率而开启了1000个进程同时执行这1000个任务,那么计算机会因为开启进程太多而崩溃,所以我们需要通过进程池或者线程池来控制产生的进程或者线程,否则一旦计算机崩溃了,又怎么谈得上效率问题,所以其实站在这个角度,实际上线程池和进程池也是一个提高效率的操作
from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor import requests,time def task(url): res = requests.get(url) if res.status_code == 200: return res.text else: return '下载失败' def parse(future): print('%s处理数据后:%s'%(current_thread().name,len(future.result()))) if __name__ == '__main__': urls = [ 'https://www.baidu.com/', 'https://www.sina.com.cn/', 'https://www.24luxiang.com/', 'https://nba.hupu.com/', 'https://www.baidu.com/', 'https://www.sina.com.cn/', 'https://www.24luxiang.com/', 'https://nba.hupu.com/', ] start = time.time() pool = ThreadPoolExecutor(3) for url in urls: pool.submit(task,url).add_done_callback(parse) pool.shutdown(wait = True) print(time.time()-start)
2.利用线程池和进程池实现套接字并发通信
客户端:
from socket import * from concurrent.futures import ThreadPoolExecutor def Sever(IP,PORT,backlog=5): sever = socket(AF_INET,SOCK_STREAM) sever.bind((IP,PORT)) sever.listen(backlog) return sever def communication(future): conn = future[0] client_address = future[1] while True: try: data = conn.recv(1024) if len(data) == 0: break print(data.decode('utf-8')) conn.send(data.upper()) except ConnectionResetError: print('%s断开连接'%client_address[1]) conn.close() break except OSError: print('%s断开连接' % client_address[1]) conn.close() break if __name__ == '__main__': sever = Sever('192.168.13.131',8080) pool = ThreadPoolExecutor(3) while True: conn, client_address = sever.accept() res = (conn, client_address) print('%s客户端已经连接服务器' % client_address[1]) pool.submit(communication,res)
客户端:
from socket import * client = socket(AF_INET,SOCK_STREAM) client.connect(('192.168.13.131',8080)) while True: msg = input('输入>>:').strip() if len(msg) == 0:continue client.send(msg.encode('utf-8')) data = client.recv(1024) print(data.decode('utf-8'))
3.协程(利用模块gevent模块,实现单线程下套接字并发通信)
首先应该来重申一下并发的含义(看起来多个任务在同时执行),其实并发的原理是线程获取cpu执行权限执行python代码,当遇到io操作或者长时间占用cpu执行权限的情况下会被操作系统强行剥夺走给别的线程使用,所以我们要实现单线程下套接字并发通信,我们就是要模拟操作系统的这个操作,而现在我们有一个geven模块帮我们封装好了这个功能,利用协程尽量降低io操作这样可以使效率有大幅的提升,并且其实单一一个线程就可以实现高并发,大大提升了计算机的并发能力。
客户端:
from gevent import monkey;monkey.patch_all() from gevent import spawn from socket import * def communion(conn): while True: try: data = conn.recv(1024) print(data.decode('utf-8')) conn.send(data.upper()) except ConnectionResetError: break conn.close() def task(): sever = socket(AF_INET, SOCK_STREAM) sever.bind(('127.0.0.1', 8080)) sever.listen(5) while True: conn,address = sever.accept() spawn(communion,conn) if __name__ == '__main__': g1 = spawn(task) g1.join()
客户端:
from socket import * from threading import Thread,current_thread def task(): client = socket(AF_INET,SOCK_STREAM) client.connect(('127.0.0.1', 8080)) n=0 while True: msg = '%shello%s'%(current_thread().name,n) n+=1 client.send(msg.encode('utf-8')) data = client.recv(1024) print(data.decode('utf-8')) if __name__ == '__main__': for i in range(500): p = Thread(target=task) p.start()