一、进程池与线程池
在刚开始学多进程或多线程时，我们迫不及待地基于多进程或多线程实现并发的套接字通信。
然而这种实现方式的致命缺陷是：服务的开启的进程数或线程数都会随着并发的客户端数目地增多而增多，这会对服务端主机带来巨大的压力，甚至于不堪重负而瘫痪。
于是我们必须对服务端开启的进程数或线程数加以控制，让机器在一个自己可以承受的范围内运行，这就是进程池或线程池的用途。
例如进程池，就是用来存放进程的池子，本质还是基于多进程，只不过是对开启进程的数目加上了限制。
1，介绍：
# 官网：https://docs.python.org/dev/library/concurrent.futures.html # concurrent.futures：模块提供了高度封装的异步调用接口 # ThreadPoolExecutor：线程池，提供异步调用 # ProcessPoolExecutor：进程池，提供异步调用 # 这两个接口的用法是一样的，唯一要搞明白的就是什么时候用线程池，什么时候用进程池， # 也就是什么时候用进程(计算密集型，享用多核优势)，什么时候用线程(I/O密集型)。
2，基本方法：
submit(fn,*args,**kwargs) # 异步提交任务 map(func, *iterables, timeout=None, chunksize=1) # 取代 for循环 submit的操作 shutdown(wait=True) """ 相当于进程池的 pool.close() + pool.join() 操作 wait=True 等待池内所有任务执行完毕回收完资源后才继续 wait=False 立即返回，并不会等待池内的任务执行完毕 但不管 wait参数为何值，整个程序都会等到所有任务执行完毕 submit和 wait必须在 shutdown之前 """ result(timeout=None) # 取得结果 add_done_callback(fn) # 回调函数
二、进程池
1，介绍：
# The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned. # class concurrent.futures.ProcessPoolExecutor(max_workers=None, mp_context=None) # An Executor subclass that executes calls asynchronously using a pool of at most max_workers processes. If max_workers is None or not given, it will default to the number of processors on the machine. If max_workers is lower or equal to 0, then a ValueError will be raised.
2，用法：
from concurrent.futures import ProcessPoolExecutor,ThreadPoolExecutor import os,time,random def task(name): print("name:%s pid:%s run." % (name,os.getpid())) time.sleep(random.randint(1,3)) # 模拟每个任务不同的运行时间 if __name__ == '__main__': # 有进程池的情况下，造进程，先把池子造出来，参数指定池子大小，最多装多少个进程，不指定，默认就是CPU核数 pool = ProcessPoolExecutor(4) # 线程池就是把ProcessPoolExecutor换成ThreadPoolExecutor,用法一样 # 启线程的话，大家的pid都是一样的，因为都在一个进程中 for i in range(1,11): pool.submit(task,"托儿所%s" % i) # 提交任务的方式是异步调用，指的是提交完任务以后，不用管这个任务是否起来，只负责提交 # 进程池相当于给你开了一个入口，你可以往里面丢任务，现在是要等着进程池结束，shutdown代表把往里面提交任务的入口给关闭掉了 pool.shutdown() # 默认参数 wait=True，这段代码就相当于，先把入口关掉，然后在原地 join住，等到进程池结束。 # 上面的shutdown执行完，才能执行打印主 print("主") """ name:托儿所1 pid:18016 run. name:托儿所2 pid:17800 run. name:托儿所3 pid:17512 run. name:托儿所4 pid:8796 run. name:托儿所5 pid:8796 run. name:托儿所6 pid:17800 run. name:托儿所7 pid:17512 run. name:托儿所8 pid:18016 run. name:托儿所9 pid:8796 run. name:托儿所10 pid:17512 run. 主 """
三、线程池
1，介绍：
# ThreadPoolExecutor is an Executor subclass that uses a pool of threads to execute calls asynchronously. # class concurrent.futures.ThreadPoolExecutor(max_workers=None, thread_name_prefix='') # An Executor subclass that uses a pool of at most max_workers threads to execute calls asynchronously. # Changed in version 3.5: If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor. # New in version 3.6: The thread_name_prefix argument was added to allow users to control the threading.Thread names for worker threads created by the pool for easier debugging.
2，用法：
# 线程池 from concurrent.futures import ProcessPoolExecutor,ThreadPoolExecutor from threading import currentThread import os,time,random def task(): print("name:%s pid:%s run." % (currentThread().getName(),os.getpid())) time.sleep(random.randint(1,3)) if __name__ == '__main__': pool = ThreadPoolExecutor(5) for i in range(1,11): pool.submit(task,) pool.shutdown(wait=True) print("主") """ 只有5个线程，0,1,2,3,4,5 在一个进程中，pid都一样 name:ThreadPoolExecutor-0_0 pid:12884 run. name:ThreadPoolExecutor-0_1 pid:12884 run. name:ThreadPoolExecutor-0_2 pid:12884 run. name:ThreadPoolExecutor-0_3 pid:12884 run. name:ThreadPoolExecutor-0_4 pid:12884 run. name:ThreadPoolExecutor-0_2 pid:12884 run. name:ThreadPoolExecutor-0_3 pid:12884 run. name:ThreadPoolExecutor-0_0 pid:12884 run. name:ThreadPoolExecutor-0_4 pid:12884 run. name:ThreadPoolExecutor-0_1 pid:12884 run. 主 """
四、map方法
map取代了 for + submit
# map方法 from concurrent.futures import ThreadPoolExecutor import os,time,random def task(n): print('%s is runing' % os.getpid()) time.sleep(random.randint(1,3)) return n**2 if __name__ == '__main__': t = ThreadPoolExecutor(max_workers=3) # for i in range(11): # future=executor.submit(task,i) t.map(task,range(1,12)) # map取代了for+submit