Are locks unnecessary in multi-threaded Python code because of the GIL?
https://stackoverflow.com/questions/105095/are-locks-unnecessary-in-multi-threaded-python-code-because-of-the-gil
由于GIL的存在, 导致所有 python的多线程 只能是 并发 , 并不是并行。
但是这样的情况下, 共享变量,同一时间只能被一个线程访问,那么为什么 多线程 模块还设计 lock?
If you are relying on an implementation of Python that has a Global Interpreter Lock (i.e. CPython) and writing multithreaded code, do you really need locks at all?
If the GIL doesn't allow multiple instructions to be executed in parallel, wouldn't shared data be unnecessary to protect?
sorry if this is a dumb question, but it is something I have always wondered about Python on multi-processor/core machines.
same thing would apply to any other language implementation that has a GIL.
answer -- data inconsistent
对于 进程间的 共享变量, 多个线程访问, 没有物理上的冲突。
但是对于 业务上的 逻辑, 会导致 业务上的冲突, 读取后, 然后在更新, 两个进程交替执行, 会导致数据不一致性。
You will still need locks if you share state between threads. The GIL only protects the interpreter internally. You can still have inconsistent updates in your own code.
Here, your code can be interrupted between reading the shared state (
balance = shared_balance
) and writing the changed result back (shared_balance = balance
), causing a lost update. The result is a random value for the shared state.To make the updates consistent, run methods would need to lock the shared state around the read-modify-write sections (inside the loops) or have some way to detect when the shared state had changed since it was read.
#!/usr/bin/env python import threading shared_balance = 0 class Deposit(threading.Thread): def run(self): for _ in xrange(1000000): global shared_balance balance = shared_balance balance += 100 shared_balance = balance class Withdraw(threading.Thread): def run(self): for _ in xrange(1000000): global shared_balance balance = shared_balance balance -= 100 shared_balance = balance threads = [Deposit(), Withdraw()] for thread in threads: thread.start() for thread in threads: thread.join() print shared_balance
solution 1 - thread lock
https://docs.python.org/3.6/library/threading.html#lock-objects
用法
https://haicoder.net/python/python-thread-lock.html
import threading num = 0 # 创建互斥锁 lock = threading.Lock() def handler_incry(): global num for i in range(100000): num += 1 print("handler_incry done, num =", num) def handler_decry(): global num for i in range(100000): num -= 1 print("handler_decry done, num =", num) if __name__ == '__main__': print("嗨客网(www.haicoder.net)") # 创建线程 t1 = threading.Thread(target=handler_incry) t2 = threading.Thread(target=handler_decry) # 启动线程 t1.start() t2.start() t1.join() t2.join()
改造
https://github.com/fanqingsong/code_snippet/blob/master/python/multithread/threadlock_plock.py
#!/usr/bin/env python import threading from multiprocessing import Process, Lock shared_balance = 0 lock = Lock() class Deposit(threading.Thread): def run(self): for _ in xrange(1000000): lock.acquire() global shared_balance balance = shared_balance balance += 100 shared_balance = balance lock.release() class Withdraw(threading.Thread): def run(self): for _ in xrange(1000000): lock.acquire() global shared_balance balance = shared_balance balance -= 100 shared_balance = balance lock.release() threads = [Deposit(), Withdraw()] for thread in threads: thread.start() for thread in threads: thread.join() print shared_balance
solution2 -- process lock
https://docs.python.org/3.6/library/multiprocessing.html#multiprocessing.Lock
A non-recursive lock object: a close analog of
threading.Lock
. Once a process or thread has acquired a lock, subsequent attempts to acquire it from any process or thread will block until it is released; any process or thread may release it. The concepts and behaviors ofthreading.Lock
as it applies to threads are replicated here inmultiprocessing.Lock
as it applies to either processes or threads, except as noted.
改造
https://github.com/fanqingsong/code_snippet/blob/master/python/multithread/threadlock_plock.py
使用 进程锁 也可以 实现 关键代码互斥。
进程级别的互斥, 可以兼容 线程级的互斥。
#!/usr/bin/env python import threading from multiprocessing import Process, Lock shared_balance = 0 lock = Lock() class Deposit(threading.Thread): def run(self): for _ in xrange(1000000): lock.acquire() global shared_balance balance = shared_balance balance += 100 shared_balance = balance lock.release() class Withdraw(threading.Thread): def run(self): for _ in xrange(1000000): lock.acquire() global shared_balance balance = shared_balance balance -= 100 shared_balance = balance lock.release() threads = [Deposit(), Withdraw()] for thread in threads: thread.start() for thread in threads: thread.join() print shared_balance
Shared memory + proccess lock == cover thread and process data sharing
Shared memory
https://docs.python.org/3.6/library/multiprocessing.html#sharing-state-between-processes
Data can be stored in a shared memory map using
Value
orArray
. For example, the following code
from multiprocessing import Process, Value, Array def f(n, a): n.value = 3.1415927 for i in range(len(a)): a[i] = -a[i] if __name__ == '__main__': num = Value('d', 0.0) arr = Array('i', range(10)) p = Process(target=f, args=(num, arr)) p.start() p.join() print(num.value) print(arr[:])
Integration of Both
https://docs.python.org/3.6/library/multiprocessing.html#sharing-state-between-processes
#!/usr/bin/env python import threading from multiprocessing import Process, Value, Array from multiprocessing import Process, Lock shared_balance = Value('i', 0) lock = Lock() class Deposit(threading.Thread): def run(self): for _ in xrange(1000000): lock.acquire() global shared_balance balance = shared_balance.value balance += 100 shared_balance.value = balance lock.release() class Withdraw(threading.Thread): def run(self): for _ in xrange(1000000): lock.acquire() global shared_balance balance = shared_balance.value balance -= 100 shared_balance.value = balance lock.release() threads = [Deposit(), Withdraw()] def deposit_func(shared_balance): for _ in xrange(100): lock.acquire() balance = shared_balance.value balance += 100 shared_balance.value = balance lock.release() def withdraw_func(shared_balance): for _ in xrange(100): lock.acquire() balance = shared_balance.value balance -= 100 shared_balance.value = balance lock.release() p_deposit = Process(target=deposit_func, args=(shared_balance,)) p_withdraw = Process(target=withdraw_func, args=(shared_balance,)) for thread in threads: thread.start() for thread in threads: thread.join() p_deposit.start() p_withdraw.start() p_deposit.join() p_withdraw.join() print shared_balance.value