如何解决多处理:池:直接迭代器或使用变量来存储迭代器
是否有区别:
with Pool(...) as pool :
res = pool.imap_unordered(fun,iterator)
for val in res :
........
和..
with Pool(...) as pool :
for val in pool.imap_unordered(fun,iterator) :
........
是第二个变体更快!
解决方法
这里是两种情况的反汇编代码:
from multiprocessing import Pool
def fun(x):
pass
def method1(it):
with Pool() as pool:
res = pool.imap_unordered(fun,it)
for val in res:
pass
def method2(it):
with Pool() as pool:
for val in pool.imap_unordered(fun,it):
pass
import dis
dis.dis(method1)
print()
print('-' * 80)
print()
dis.dis(method2)
打印:
8 0 LOAD_GLOBAL 0 (Pool)
2 CALL_FUNCTION 0
4 SETUP_WITH 28 (to 34)
6 STORE_FAST 1 (pool)
9 8 LOAD_FAST 1 (pool)
10 LOAD_METHOD 1 (imap_unordered)
12 LOAD_GLOBAL 2 (fun)
14 LOAD_FAST 0 (it)
16 CALL_METHOD 2
18 STORE_FAST 2 (res)
10 20 LOAD_FAST 2 (res)
22 GET_ITER
>> 24 FOR_ITER 4 (to 30)
26 STORE_FAST 3 (val)
11 28 JUMP_ABSOLUTE 24
>> 30 POP_BLOCK
32 BEGIN_FINALLY
>> 34 WITH_CLEANUP_START
36 WITH_CLEANUP_FINISH
38 END_FINALLY
40 LOAD_CONST 0 (None)
42 RETURN_VALUE
--------------------------------------------------------------------------------
14 0 LOAD_GLOBAL 0 (Pool)
2 CALL_FUNCTION 0
4 SETUP_WITH 24 (to 30)
6 STORE_FAST 1 (pool)
15 8 LOAD_FAST 1 (pool)
10 LOAD_METHOD 1 (imap_unordered)
12 LOAD_GLOBAL 2 (fun)
14 LOAD_FAST 0 (it)
16 CALL_METHOD 2
18 GET_ITER
>> 20 FOR_ITER 4 (to 26)
22 STORE_FAST 2 (val)
16 24 JUMP_ABSOLUTE 20
>> 26 POP_BLOCK
28 BEGIN_FINALLY
>> 30 WITH_CLEANUP_START
32 WITH_CLEANUP_FINISH
34 END_FINALLY
36 LOAD_CONST 0 (None)
38 RETURN_VALUE
结果
无论迭代器的长度或 method1
完成的处理量如何,或在处理fun
返回的结果。换句话说,如此微不足道的差异,无需担心。
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。