Multiprocessing Pool() 方法对性能没有影响

如何解决Multiprocessing Pool() 方法对性能没有影响

我在多处理器机器上的 Linux/Debian 测试中使用 Python 3.9.2。我想了解多处理的工作原理。

我编写了两个简单的脚本来执行两个指数函数，一个没有多处理，另一个有。

这是没有多处理的：

from timeit import default_timer as timer


def sqr(n):

    a = n ** n

    return a


def sqr_2(m):

    b = m ** m

    return b


def main():

    start = timer()
    
    print(f'sqr = {sqr(100000)}\nsqr_2= {sqr_2(200000)}')
    
    end = timer()


    print(f'time frame in which the operation is resolved: {end - start} seconds')


if __name__ == '__main__':
    main()

这是使用多处理的脚本：

from multiprocessing import Pool,cpu_count
from timeit import default_timer as timer


def sqr_1(n):

    return n ** n


def sqr_2(m):

    return m ** m


def main():

    cpu_cnt = cpu_count()
    pool = Pool(processes = cpu_cnt)     #In this case there are 12 processors

    start = timer()
    
    val_1 = (100000,)
    val_2 = (200000,)
    
    process_1 = pool.map_async(sqr_1,val_1)
    process_2 = pool.map_async(sqr_2,val_2)
    
    print(f'Results: {process_1.get(),process_2.get()}')

    end = timer()

    print(f'time frame in which the operation is resolved: {end - start} seconds')


if __name__ == '__main__':
    main()

问题是第二个脚本的进程没有任何错误地完成，在相同的时间内（大约 14 秒）执行了与第一个脚本相同的任务。因此，第二个脚本中的多处理不起作用。我提前感谢任何想指出这是错误的人！

解决方法

考虑以下脚本。它让您可以在运行时选择调用函数的次数，以及是串行还是并行调用。它也只是计算值；它不会尝试将字符串表示形式写入标准输出（因为对于大 n**n，将 n 的结果转换为字符串比实际计算要耗时得多）。

from multiprocessing import Pool,cpu_count
from timeit import default_timer as timer
import sys


def f(n):
    return n ** n


def main():
    cpu_cnt = cpu_count()
    n = int(sys.argv[2])
    start = timer()
    if sys.argv[1] == "s":
        s = [f(100000) for _ in range(n)]
    else:
        pool = Pool(processes = cpu_cnt)
        s = [pool.map_async(f,(100000,)) for _ in range(n)]
        results = [x.get() for x in s]
    end = timer()
    print(f'time frame in which the operation is resolved: {end - start} seconds')


if __name__ == '__main__':
    main()

以下是我的 4 核机器上 2、6、12、24、48、96 和 192 次函数调用的结果：

% for n in 2 6 12 24 48 96 192; do print $n; for x in s p; do python3 tmp.py $x $n; done; done
2
time frame in which the operation is resolved: 0.146144435 seconds
time frame in which the operation is resolved: 0.178840965 seconds
6
time frame in which the operation is resolved: 0.423103791 seconds
time frame in which the operation is resolved: 0.24940852500000002 seconds
12
time frame in which the operation is resolved: 0.848754817 seconds
time frame in which the operation is resolved: 0.340022419 seconds
24
time frame in which the operation is resolved: 1.691312521 seconds
time frame in which the operation is resolved: 0.571664972 seconds
48
time frame in which the operation is resolved: 3.415401498 seconds
time frame in which the operation is resolved: 1.029526396 seconds
96
time frame in which the operation is resolved: 6.76773454 seconds
time frame in which the operation is resolved: 2.016387216 seconds
192
time frame in which the operation is resolved: 13.529949021999998 seconds
time frame in which the operation is resolved: 3.770171452 seconds

由于并行化本身的开销，只有 2 个并行进程没有加速。 （实际上，存在减速。）一旦您开始运行更多进程，加速就会增加，但对于 n 内核，您永远不会看到 n 的加速。