http://www.cnblogs.com/alex3714/articles/5230609.html
http://www.cnblogs.com/wupeiqi/articles/5040827.html
内容
- 进程、与线程区别
- cpu运行原理
- python GIL全局解释器锁
- 线程
- 语法
- join
- 线程锁之LockRlock信号量
- 将线程变为守护进程
- Event事件
- queue队列
- 生产者消费者模型
- Queue队列
- 开发一个线程池
- 进程
- 语法
- 进程间通讯
- 进程池
进程与线程
什么是线程(thread)?
线程是操作系统能够进行运算调度的最小单位。它被包含在进程之中,是进程中的实际运作单位。一条线程指的是进程中一个单一顺序的控制流,一个进程中可以并发多个线程,每条线程并行执行不同的任务
A thread is an execution context, which is all the information a CPU needs to execute a stream of instructions.
Suppose you're reading a book, and you want to take a break right now, but you want to be able to come back and resume reading from the exact point where you stopped. One way to achieve that is by jotting down the page number, line number, and word number. So your execution context for reading a book is these 3 numbers.
If you have a roommate, and she's using the same technique, she can take the book while you're not using it, and resume reading from where she stopped. Then you can take it back, and resume it from where you were.
Threads work in the same way. A CPU is giving you the illusion that it's doing multiple computations at the same time. It does that by spending a bit of time on each computation. It can do that because it has an execution context for each computation. Just like you can share a book with your friend, many tasks can share a CPU.
On a more technical level, an execution context (therefore a thread) consists of the values of the CPU's registers.
Last: threads are different from processes. A thread is a context of execution, while a process is a bunch of resources associated with a computation. A process can have one or many threads.
Clarification: the resources associated with a process include memory pages (all the threads in a process have the same view of the memory), file descriptors (e.g., open sockets), and security credentials (e.g., the ID of the user who started the process).
什么是进程(process)?
An executing instance of a program is called a process.
Each process provides the resources needed to execute a program. A process has a virtual address space, executable code, open handles to system objects, a security context, a unique process identifier, environment variables, a priority class, minimum and maximum working set sizes, and at least one thread of execution. Each process is started with a single thread, often called the primary thread, but can create additional threads from any of its threads.
进程与线程的区别?
- Threads share the address space of the process that created it; processes have their own address space.(线程共享创建它的进程内存地址空间,进程之间不共享内存地址空间)
- Threads have direct access to the data segment of its process; processes have their own copy of the data segment of the parent process.(线程可以直接直接访问创建它的进程数据段,进程则会拷贝其父进程的数据段,不能直接访问其他进程的数据)
- Threads can directly communicate with other threads of its process; processes must use interprocess communication to communicate with sibling processes.(同一进程间的不同线程之间可直接交互,兄弟进程间则需要借且第三方间接通信如队列,PIPE、manager等 )
- New threads are easily created; new processes require duplication of the parent process.(创建线程的开销是很小的,而创建进程开销比较大,需要复制父进程的数据)
- Threads can exercise considerable control over threads of the same process; processes can only exercise control over child processes.(线程可以在同一进程的线程中行使相当大的控制权,进程只能控制子进程)
- Changes to the main thread (cancellation, priority change, etc.) may affect the behavior of the other threads of the process; changes to the parent process does not affect child processes.(主线程的更改可能影响到同一进程中的其他线程;父进程的更改则不会影响到子进程)
Python GIL(Global Interpreter Lock)全局解释锁
In CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from executing Python bytecodes at once. This lock is necessary mainly because CPython’s memory management is not thread-safe. (However, since the GIL exists, other features have grown to depend on the guarantees that it enforces.)
上面的核心意思就是,无论你启多少个线程,你有多少个cpu, Python在执行的时候会淡定的在同一时刻只允许一个线程运行,擦。。。,那这还叫什么多线程呀?莫如此早的下结结论,听我现场讲。(GIL目的,确保同一份数据在同一时刻只有被 一个线程访问修改,python原生线程调用的是c,而在c中python解释器是无法控制,所以加GIL就是为了数据的安全,即线程安全)
首先需要明确的一点是GIL
并不是Python的特性,它是在实现Python解析器(CPython)时所引入的一个概念。就好比C++是一套语言(语法)标准,但是可以用不同的编译器来编译成可执行代码。有名的编译器例如GCC,INTEL C++,Visual C++等。Python也一样,同样一段代码可以通过CPython,PyPy,Psyco等不同的Python执行环境来执行。像其中的JPython就没有GIL。然而因为CPython是大部分环境下默认的Python执行环境。所以在很多人的概念里CPython就是Python,也就想当然的把GIL
归结为Python语言的缺陷。所以这里要先明确一点:GIL并不是Python的特性,Python完全可以不依赖于GIL
这篇文章透彻的剖析了GIL对python多线程的影响,强烈推荐看一下:http://www.dabeaz.com/python/UnderstandingGIL.pdf
Python threading模块
Threading用于提供线程相关的操作,线程是应用程序中工作的最小单元。
线程有2种调用方式,如下:
直接调用
#!/use/local/env python # -*- coding:utf-8 -*- import threading import time def sayhi(num): print('runing on number:%s' %(num)) for i in range(3): time.sleep(1) print('sleep: %s', str(i)) def saybye(zimu): print('say bye:{}'.format(zimu)) time.sleep(3) if __name__ == '__main__': # ''' t1 = threading.Thread(target=sayhi, args=[1,]) t2 = threading.Thread(target=saybye, args=['a',]) t1.start() t2.start() t1.join() t2.join() print(t1.getName()) print(t2.getName()) print('-----main-------') # ''' ''' t_list = [] for i in range(10): t = threading.Thread(target=sayhi, args=[i,]) t.start() t_list.append(t) for i in t_list: i.join() print('-----main-------') '''
继承式调用
#!/use/local/env python # -*- coding:utf-8 -*- import threading import time class MyThread(threading.Thread): def __init__(self, num): threading.Thread.__init__(self) self.num = num def run(self): print('running on number:%s' %(self.num)) time.sleep(3) if __name__ == '__main__': t1 = MyThread(1) t2 = MyThread(2) t1.start() t2.start() print(t1.getName()) print(t2.getName()) t1.join() 让主线程等待t1线程执行完,然后才往下执行 print('-----main-----')
#!/usr/bin/env python # -*- coding:utf-8 -*- import threading import time def show(arg): time.sleep(1) print 'thread'+str(arg) for i in range(10): t = threading.Thread(target=show, args=(i,)) t.start() print 'main thread stop'
上述代码创建了10个“前台”线程,然后控制器就交给了CPU,CPU根据指定算法进行调度,分片执行指令。
更多方法:
- start 线程准备就绪,等待CPU调度
- setName 为线程设置名称
- getName 获取线程名称
- setDaemon 设置为后台线程或前台线程(默认)
如果是后台线程,主线程执行过程中,后台线程也在进行,主线程执行完毕后,后台线程不论成功与否,均停止
如果是前台线程,主线程执行过程中,前台线程也在进行,主线程执行完毕后,等待前台线程也执行完成后,程序停止 - join 等待这个线程执行到结束,
在A线程中调用B线程的join()方法,则当前线程到此线程方法时,A线程暂停执行,
直到B线程执行完毕,A线程再接着B.join()之后的代码 - run 线程被cpu调度后自动执行线程对象的run方法
Join & Daemon
Some threads do background tasks, like sending keepalive packets, or performing periodic garbage collection, or whatever. These are only useful when the main program is running, and it's okay to kill them off once the other, non-daemon, threads have exited.
Without daemon threads, you'd have to keep track of them, and tell them to exit, before your program can completely quit. By setting them as daemon threads, you can let them run and forget about them, and when your program quits, any daemon threads are killed automatically.
#!/use/local/env python # -*- coding:utf-8 -*- import threading, time def run(n): print('[%s]----running---- ' %(n)) time.sleep(2) print('[%s] done' %(n)) def main(): for i in range(5): t = threading.Thread(target=run, args=[i,]) t.start() #t.join(t) print('starting thread', t.getName()) m = threading.Thread(target=main, args=[]) m.setDaemon(True) #将主线程设置为守护线程,它退出时,其他子线程会同时退出 m.start() #print(m.is_alive()) m.join(3) #time.sleep(3) print('the main thread done')
Note:Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event
.
线程锁(互斥锁Mutex)
一个进程下可以启动多个线程,多个线程共享父进程的内存空间,也就意味着每个线程可以访问同一份数据,此时,如果2个线程同时要修改同一份数据,会出现什么状况?
import time import threading def addNum(): global num #在每个线程中都获取这个全局变量 print('--get num:',num ) time.sleep(1) num -=1 #对此公共变量进行-1操作 num = 100 #设定一个共享变量 thread_list = [] for i in range(100): t = threading.Thread(target=addNum) t.start() thread_list.append(t) for t in thread_list: #等待所有线程执行完毕 t.join() print('final num:', num )
正常来讲,这个num结果应该是0, 但在python 2.7上多运行几次,会发现,最后打印出来的num结果不总是0,为什么每次运行的结果不一样呢? 哈,很简单,假设你有A,B两个线程,此时都 要对num 进行减1操作, 由于2个线程是并发同时运行的,所以2个线程很有可能同时拿走了num=100这个初始变量交给cpu去运算,当A线程去处完的结果是99,但此时B线程运算完的结果也是99,两个线程同时CPU运算的结果再赋值给num变量后,结果就都是99。那怎么办呢? 很简单,每个线程在要修改公共数据时,为了避免自己在还没改完的时候别人也来修改此数据,可以给这个数据加一把锁, 这样其它线程想修改此数据时就必须等待你修改完毕并把锁释放掉后才能再访问此数据。
*注:不要在3.x上运行,不知为什么,3.x上的结果总是正确的,可能是自动加了锁
加锁版本(Lock普通锁)
import time import threading def addNum(): global num #在每个线程中都获取这个全局变量 print('--get num:',num ) time.sleep(1) lock.acquire() #修改数据前加锁 num -=1 #对此公共变量进行-1操作 lock.release() #修改后释放 num = 100 #设定一个共享变量 thread_list = [] lock = threading.Lock() #生成全局锁 for i in range(100): t = threading.Thread(target=addNum) t.start() thread_list.append(t) for t in thread_list: #等待所有线程执行完毕 t.join() print('final num:', num )
RLock(递归锁)
说白了就是在一个大锁中还要再包含子锁
import threading,time def run1(): print("grab the first part data") lock.acquire() global num num +=1 lock.release() return num def run2(): print("grab the second part data") lock.acquire() global num2 num2+=1 lock.release() return num2 def run3(): lock.acquire() res = run1() print('--------between run1 and run2-----') res2 = run2() lock.release() print(res,res2) if __name__ == '__main__': num,num2 = 0,0 lock = threading.RLock() for i in range(10): t = threading.Thread(target=run3) t.start() while threading.active_count() != 1: print(threading.active_count()) else: print('----all threads done---') print(num,num2)
一般情况下建议使用Rlocak递归锁,Lock多个锁后无法解开外层锁
Semaphore(信号量)
互斥锁 同时只允许一个线程更改数据,而Semaphore是同时允许一定数量的线程更改数据 ,比如厕所有3个坑,那最多只允许3个人上厕所,后面的人只能等里面有人出来了才能再进去。
import threading,time def run(n): semaphore.acquire() time.sleep(1) print("run the thread: %s " %n) semaphore.release() if __name__ == '__main__': num= 0 semaphore = threading.BoundedSemaphore(5) #最多允许5个线程同时运行 for i in range(20): t = threading.Thread(target=run,args=(i,)) t.start() while threading.active_count() != 1: pass #print threading.active_count() else: print('----all threads done---') print(num)
Events
python线程的事件用于主线程控制其他线程的执行,事件主要提供了三个方法 set、wait、clear。
事件处理的机制:全局定义了一个“Flag”,如果“Flag”值为 False,那么当程序执行 event.wait 方法时就会阻塞,如果“Flag”值为True,那么event.wait 方法时便不再阻塞。
- clear:将“Flag”设置为False
- set:将“Flag”设置为True
An event is a simple synchronization object;
the event represents an internal flag, and threads
can wait for the flag to be set, or set or clear the flag themselves.
event = threading.Event()
# a client thread can wait for the flag to be set
event.wait()
# a server thread can set or reset it
event.set()
event.clear()
If the flag is set, the wait method doesn’t do anything.
If the flag is cleared, wait will block until it becomes set again.
Any number of threads may wait for the same event.
通过Event来实现两个或多个线程间的交互,下面是一个红绿灯的例子,即起动一个线程做交通指挥灯,生成几个线程做车辆,车辆行驶按红灯停,绿灯行的规则。
import threading,time import random def light(): if not event.isSet(): event.set() #wait就不阻塞 #绿灯状态 count = 0 while True: if count < 10: print('