zoukankan      html  css  js  c++  java

  • 线程与进程

    Python threading模块

    线程有2种调用方式，如下：

    直接调用

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    import threading
    import time
     
    def sayhi(num): #定义每个线程要运行的函数
     
        print("running on number:%s" %num)
     
        time.sleep(3)
     
    if __name__ == '__main__':
     
        t1 = threading.Thread(target=sayhi,args=(1,)) #生成一个线程实例
        t2 = threading.Thread(target=sayhi,args=(2,)) #生成另一个线程实例
     
        t1.start() #启动线程
        t2.start() #启动另一个线程
     
        print(t1.getName()) #获取线程名
        print(t2.getName())

    继承式调用

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    import threading
    import time
     
     
    class MyThread(threading.Thread):
        def __init__(self,num):
            threading.Thread.__init__(self)
            self.num = num
     
        def run(self):#定义每个线程要运行的函数
     
            print("running on number:%s" %self.num)
     
            time.sleep(3)
     
    if __name__ == '__main__':
     
        t1 = MyThread(1)
        t2 = MyThread(2)
        t1.start()
        t2.start()

    Join & Daemon

    Some threads do background tasks, like sending keepalive packets, or performing periodic garbage collection, or whatever. These are only useful when the main program is running, and it's okay to kill them off once the other, non-daemon, threads have exited.

    Without daemon threads, you'd have to keep track of them, and tell them to exit, before your program can completely quit. By setting them as daemon threads, you can let them run and forget about them, and when your program quits, any daemon threads are killed automatically.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    import time
    import threading
     
    def run(n):
     
        print('[%s]------running---- ' % n)
        time.sleep(2)
        print('--done--')
     
    def main():
        for i in range(5):
            t = threading.Thread(target=run,args=[i,])
            #time.sleep(1)
            t.start()
            t.join(1)
            print('starting thread', t.getName())
     
     
    m = threading.Thread(target=main,args=[])
    m.setDaemon(True) #将主线程设置为Daemon线程,它退出时,其它子线程会同时退出,不管是否执行完任务
    m.start()
    #m.join(timeout=2)
    print("---main thread done----")
    Note：Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.

    　　

     

     

    线程锁(互斥锁Mutex)

    一个进程下可以启动多个线程，多个线程共享父进程的内存空间，也就意味着每个线程可以访问同一份数据，此时，如果2个线程同时要修改同一份数据，会出现什么状况？

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    import time
    import threading
     
    def addNum():
        global num #在每个线程中都获取这个全局变量
        print('--get num:',num )
        time.sleep(1)
        num  -=1 #对此公共变量进行-1操作
     
    num = 100  #设定一个共享变量
    thread_list = []
    for i in range(100):
        t = threading.Thread(target=addNum)
        t.start()
        thread_list.append(t)
     
    for t in thread_list: #等待所有线程执行完毕
        t.join()
     
     
    print('final num:', num )

    正常来讲，这个num结果应该是0， 但在python 2.7上多运行几次，会发现，最后打印出来的num结果不总是0，为什么每次运行的结果不一样呢？ 哈，很简单，假设你有A,B两个线程，此时都 要对num 进行减1操作， 由于2个线程是并发同时运行的，所以2个线程很有可能同时拿走了num=100这个初始变量交给cpu去运算，当A线程去处完的结果是99，但此时B线程运算完的结果也是99，两个线程同时CPU运算的结果再赋值给num变量后，结果就都是99。那怎么办呢？ 很简单，每个线程在要修改公共数据时，为了避免自己在还没改完的时候别人也来修改此数据，可以给这个数据加一把锁， 这样其它线程想修改此数据时就必须等待你修改完毕并把锁释放掉后才能再访问此数据。 

    *注：不要在3.x上运行，不知为什么，3.x上的结果总是正确的，可能是自动加了锁

    加锁版本

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    import time
    import threading
     
    def addNum():
        global num #在每个线程中都获取这个全局变量
        print('--get num:',num )
        time.sleep(1)
        lock.acquire() #修改数据前加锁
        num  -=1 #对此公共变量进行-1操作
        lock.release() #修改后释放
     
    num = 100  #设定一个共享变量
    thread_list = []
    lock = threading.Lock() #生成全局锁
    for i in range(100):
        t = threading.Thread(target=addNum)
        t.start()
        thread_list.append(t)
     
    for t in thread_list: #等待所有线程执行完毕
        t.join()
     
    print('final num:', num )

    　　

    RLock（递归锁）

    说白了就是在一个大锁中还要再包含子锁

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    import threading,time
     
    def run1():
        print("grab the first part data")
        lock.acquire()
        global num
        num +=1
        lock.release()
        return num
    def run2():
        print("grab the second part data")
        lock.acquire()
        global  num2
        num2+=1
        lock.release()
        return num2
    def run3():
        lock.acquire()
        res = run1()
        print('--------between run1 and run2-----')
        res2 = run2()
        lock.release()
        print(res,res2)
     
     
    if __name__ == '__main__':
     
        num,num2 = 0,0
        lock = threading.RLock()
        for i in range(10):
            t = threading.Thread(target=run3)
            t.start()
     
    while threading.active_count() != 1:
        print(threading.active_count())
    else:
        print('----all threads done---')
        print(num,num2)

    　　

    Semaphore(信号量)

    互斥锁 同时只允许一个线程更改数据，而Semaphore是同时允许一定数量的线程更改数据 ，比如厕所有3个坑，那最多只允许3个人上厕所，后面的人只能等里面有人出来了才能再进去。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    import threading,time
     
    def run(n):
        semaphore.acquire()
        time.sleep(1)
        print("run the thread: %s " %n)
        semaphore.release()
     
    if __name__ == '__main__':
     
        num= 0
        semaphore  = threading.BoundedSemaphore(5) #最多允许5个线程同时运行
        for i in range(20):
            t = threading.Thread(target=run,args=(i,))
            t.start()
     
    while threading.active_count() != 1:
        pass #print threading.active_count()
    else:
        print('----all threads done---')
        print(num)

     

    Events

    An event is a simple synchronization object;

    the event represents an internal flag, and threads
    can wait for the flag to be set, or set or clear the flag themselves.

    event = threading.Event()

    # a client thread can wait for the flag to be set
    event.wait()

    # a server thread can set or reset it
    event.set()
    event.clear()
    If the flag is set, the wait method doesn’t do anything.
    If the flag is cleared, wait will block until it becomes set again.
    Any number of threads may wait for the same event.

    通过Event来实现两个或多个线程间的交互，下面是一个红绿灯的例子，即起动一个线程做交通指挥灯，生成几个线程做车辆，车辆行驶按红灯停，绿灯行的规则。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    import threading,time
    import random
    def light():
        if not event.isSet():
            event.set() #wait就不阻塞 #绿灯状态
        count = 0
        while True:
            if count < 10:
                print('33[42;1m--green light on---33[0m')
            elif count <13:
                print('33[43;1m--yellow light on---33[0m')
            elif count <20:
                if event.isSet():
                    event.clear()
                print('33[41;1m--red light on---33[0m')
            else:
                count = 0
                event.set() #打开绿灯
            time.sleep(1)
            count +=1
    def car(n):
        while 1:
            time.sleep(random.randrange(10))
            if  event.isSet(): #绿灯
                print("car [%s] is running.." % n)
            else:
                print("car [%s] is waiting for the red light.." %n)
    if __name__ == '__main__':
        event = threading.Event()
        Light = threading.Thread(target=light)
        Light.start()
        for i in range(3):
            t = threading.Thread(target=car,args=(i,))
            t.start()

    　　

    　　

    queue队列 

    queue is especially useful in threaded programming when information must be exchanged safely between multiple threads.

    class queue.Queue(maxsize=0) #先入先出
    class queue.LifoQueue(maxsize=0) #last in fisrt out 
    class queue.PriorityQueue(maxsize=0) #存储数据时可设置优先级的队列

    Constructor for a priority queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.

    The lowest valued entries are retrieved first (the lowest valued entry is the one returned by sorted(list(entries))[0]). A typical pattern for entries is a tuple in the form: (priority_number, data).

    exception queue.Empty

    Exception raised when non-blocking get() (or get_nowait()) is called on a Queue object which is empty.

    exception queue.Full

    Exception raised when non-blocking put() (or put_nowait()) is called on a Queue object which is full.

    Queue.qsize()
    Queue.empty() #return True if empty  
    Queue.full() # return True if full 
    Queue.put(item, block=True, timeout=None)

    Put item into the queue. If optional args block is true and timeout is None (the default), block if necessary until a free slot is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Full exception if no free slot was available within that time. Otherwise (block is false), put an item on the queue if a free slot is immediately available, else raise the Full exception (timeout is ignored in that case).

    Queue.put_nowait(item)

    Equivalent to put(item, False).

    Queue.get(block=True, timeout=None)

    Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Empty exception if no item was available within that time. Otherwise (block is false), return an item if one is immediately available, else raise the Empty exception (timeout is ignored in that case).

    Queue.get_nowait()

    Equivalent to get(False).

    Two methods are offered to support tracking whether enqueued tasks have been fully processed by daemon consumer threads.

    Queue.task_done()

    Indicate that a formerly enqueued task is complete. Used by queue consumer threads. For each get() used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.

    If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put() into the queue).

    Raises a ValueError if called more times than there were items placed in the queue.

    Queue.join() block直到queue被消费完毕

    生产者消费者模型

     

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    import time,random
    import queue,threading
    q = queue.Queue()
    def Producer(name):
      count = 0
      while count <20:
        time.sleep(random.randrange(3))
        q.put(count)
        print('Producer %s has produced %s baozi..' %(name, count))
        count +=1
    def Consumer(name):
      count = 0
      while count <20:
        time.sleep(random.randrange(4))
        if not q.empty():
            data = q.get()
            print(data)
            print('33[32;1mConsumer %s has eat %s baozi...33[0m' %(name, data))
        else:
            print("-----no baozi anymore----")
        count +=1
    p1 = threading.Thread(target=Producer, args=('A',))
    c1 = threading.Thread(target=Consumer, args=('B',))
    p1.start()
    c1.start()

    　　

     

    多进程multiprocessing

    multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    from multiprocessing import Process
    import time
    def f(name):
        time.sleep(2)
        print('hello', name)
     
    if __name__ == '__main__':
        p = Process(target=f, args=('bob',))
        p.start()
        p.join()

    To show the individual process IDs involved, here is an expanded example:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    from multiprocessing import Process
    import os
     
    def info(title):
        print(title)
        print('module name:', __name__)
        print('parent process:', os.getppid())
        print('process id:', os.getpid())
        print(" ")
     
    def f(name):
        info('33[31;1mfunction f33[0m')
        print('hello', name)
     
    if __name__ == '__main__':
        info('33[32;1mmain process line33[0m')
        p = Process(target=f, args=('bob',))
        p.start()
        p.join()

     

    进程间通讯　　

    不同进程间内存是不共享的，要想实现两个进程间的数据交换，可以用以下方法：

    Queues

    使用方法跟threading里的queue差不多

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    from multiprocessing import Process, Queue
     
    def f(q):
        q.put([42, None, 'hello'])
     
    if __name__ == '__main__':
        q = Queue()
        p = Process(target=f, args=(q,))
        p.start()
        print(q.get())    # prints "[42, None, 'hello']"
        p.join()

    Pipes

    The Pipe() function returns a pair of connection objects connected by a pipe which by default is duplex (two-way). For example:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    from multiprocessing import Process, Pipe
     
    def f(conn):
        conn.send([42, None, 'hello'])
        conn.close()
     
    if __name__ == '__main__':
        parent_conn, child_conn = Pipe()
        p = Process(target=f, args=(child_conn,))
        p.start()
        print(parent_conn.recv())   # prints "[42, None, 'hello']"
        p.join()

    The two connection objects returned by Pipe() represent the two ends of the pipe. Each connection object has send() and recv() methods (among others). Note that data in a pipe may become corrupted if two processes (or threads) try to read from or write to the same end of the pipe at the same time. Of course there is no risk of corruption from processes using different ends of the pipe at the same time.

     

    Managers

    A manager object returned by Manager() controls a server process which holds Python objects and allows other processes to manipulate them using proxies.

    A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value and Array. For example,

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    from multiprocessing import Process, Manager
     
    def f(d, l):
        d[1] = '1'
        d['2'] = 2
        d[0.25] = None
        l.append(1)
        print(l)
     
    if __name__ == '__main__':
        with Manager() as manager:
            d = manager.dict()
     
            l = manager.list(range(5))
            p_list = []
            for i in range(10):
                p = Process(target=f, args=(d, l))
                p.start()
                p_list.append(p)
            for res in p_list:
                res.join()
     
            print(d)
            print(l)

    　　

     
     

    进程同步

    Without using the lock output from the different processes is liable to get all mixed up.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    from multiprocessing import Process, Lock
     
    def f(l, i):
        l.acquire()
        try:
            print('hello world', i)
        finally:
            l.release()
     
    if __name__ == '__main__':
        lock = Lock()
     
        for num in range(10):
            Process(target=f, args=(lock, num)).start()

    　　

     

    进程池　　

    进程池内部维护一个进程序列，当使用时，则去进程池中获取一个进程，如果进程池序列中没有可供使用的进进程，那么程序就会等待，直到进程池中有可用进程为止。

    进程池中有两个方法：

    • apply
    • apply_async
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    from  multiprocessing import Process,Pool
    import time
     
    def Foo(i):
        time.sleep(2)
        return i+100
     
    def Bar(arg):
        print('-->exec done:',arg)
     
    pool = Pool(5)
     
    for i in range(10):
        pool.apply_async(func=Foo, args=(i,),callback=Bar)
        #pool.apply(func=Foo, args=(i,))
     
    print('end')
    pool.close()
    pool.join()#进程池中进程执行完毕后再关闭，如果注释，那么程序直接关闭。
  • 相关阅读:
    N天学习一个linux命令之lsof
    N天学习一个linux命令之ps
    N天学习一个linux命令之yum
    N天学习一个linux命令之rsync
    N天学习一个linux命令之ss
    N天学习一个linux命令之netstat
    N天学习一个linux命令之vmstat
    N天学习一个linux命令之sort
    N天学习一个linux命令之rpm
    跨域问题
  • 原文地址：https://www.cnblogs.com/bruceg/p/5264419.html
Copyright © 2011-2022 走看看