使用Joblib并行评估Cloudpickle对象

如何解决使用Joblib并行评估Cloudpickle对象

我想使用计算机上的多个内核来评估一个cloudpickle对象（由sympy的lambdify创建）。下面是我尝试做的一个最小示例。

import sympy as sy
import cloudpickle
from joblib import Parallel,delayed

x,y = sy.symbols("x,y")
expr = sy.sin(x) + sy.cos(y)
expr_numpy = sy.lambdify((x,y),expr,modules="numpy")

with open('pickle_test/sample_pickle.pkl','wb') as f:
     f.write(cloudpickle.dumps(expr_numpy))

with open('pickle_test/sample_pickle.pkl','rb') as f:
    sample_pickle = cloudpickle.load(f)

def evaluate_pickle(x,y):
    result = sample_pickle(x,y)
    return x,y,result

trials = 10
value_x = np.random.uniform(0,5,trials)
value_y = np.random.uniform(0,trials)

Parallel(n_jobs=4)(delayed(evaluate_pickle)(value_x,value_y) for i in range(trials))

我收到以下错误：

joblib.externals.loky.process_executor.brokenProcesspool: A task has Failed to un-serialize. Please ensure that the arguments of the function are all picklable.

解决方法

我实际上能够在没有任何错误的情况下运行代码（在将路径从 init 更改为 import numpy as np from typing import Optional class PricingLimits: def __init__(self,min_price: Optional[float] = None,min_percent: Optional[float] = None,max_percent: Optional[float] = None,min_delta: Optional[float] = None): if not min_price or np.isnan(min_price): self.min_price = None else: self.min_price = min_price if not min_percent or np.isnan(min_percent): self.min_percent = None else: self.min_percent = min_percent if not max_percent or np.isnan(max_percent): self.max_percent = None else: self.max_percent = max_percent if not min_delta or np.isnan(min_delta): self.min_delta = None else: self.min_delta = min_delta 之后）。

以下是此代码成功运行的以下软件版本：numpy 1.19.5、sympy 1.7.1、cloudpickle 1.6.0、joblib 1.0.1