微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦!

Python多处理:将任务异步分配给管理器

如何解决Python多处理:将任务异步分配给管理器

我需要在不同的进程中保存python对象的实例,调用这些对象的方法并异步收集结果。我的方法是为每个流程和对象创建一个管理器,然后通过管理器调用复制方法

from multiprocessing.managers import SyncManager

class RemoteObject():
    
    def __init__(self):
        
        print('Start expensive init')
        ... # do expensive stuff
        print('End expensive init')

    def expensiveJob(self,*args,**kwargs):
        print('Start expensive job')
        ... # do expensive stuff
        print('End expensive job')

class MyManager(SyncManager): pass

managers = []
instances = []

numInstances = 4

# Create the remote objects
for i in range(numInstances):
    manager = MyManager()
    manager.register('RemoteObject',RemoteObject)
    manager.start()
    managers.append(manager)
    instances.append(manager.RemoteObject())

# do the jobs
myResult = [instance.expensiveJob() for instance in instances]

尽管每个实例现在处于不同的进程中,并且作业在这些不同的进程中执行,但是对象初始化和昂贵的作业都接连发生。也就是说,输出类似于

Start expensive init
End expensive init
Start expensive init
End expensive init
Start expensive init
End expensive init
Start expensive init
End expensive init
Start expensive job
End expensive job
Start expensive job
End expensive job
Start expensive job
End expensive job
Start expensive job
End expensive job

而我希望看到类似的东西

Start expensive init
Start expensive init
Start expensive init
Start expensive init
End expensive init
End expensive init
End expensive init
End expensive init
Start expensive job
Start expensive job
Start expensive job
Start expensive job
End expensive job
End expensive job
End expensive job
End expensive job

我怎么到那里?是否有call_asynchronously方法或我可以使用的类似方法? (我在多处理模块的文档中找不到类似的东西。)

请注意,经典进程池不是此处的选择,因为我需要在多个昂贵作业之间将远程实例保持在内存中。

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。