如何解决Dill 反序列化 NMF 子类失败

总结

我创建了一个扩展 sklearn's NMF 的子类。如果我之前没有重新定义该类，则对其进行酸洗并使用 dill 取消酸洗实例会失败。

最小（非）工作示例

首先，创建一些自定义的类和实例，并pickle它们：

# imports
import dill
from sklearn.decomposition import NMF

# custom classes
class NMFwrapper(NMF): # custom child class
    def fit(self,X,y):
        print("Slightly modifying NMF")
        self = super().fit(X=X,y=y)
        return self

class CustomClass(): # dummy class
    def onlymethod(self):
        print("I'm the only method")

class CustomChild(CustomClass): # dummy child class
    def second_method(self):
        print("I'm a new child method")

# instances
normal_nmf = NMF()
new_nmf = NMFwrapper()
custom = CustomClass()
child = CustomChild()

# pickle all of them
with open("nmf.pkl","wb") as f:
    dill.dump(normal_nmf,f)
with open("nmf_wrapper.pkl","wb") as f:
    dill.dump(new_nmf,f)
with open("custom.pkl","wb") as f:
    dill.dump(custom,f)
with open("child.pkl","wb") as f:
    dill.dump(child,f)

然后在第二个终端/内核/REPL/...中执行以下操作：

import dill
with open("nmf.pkl","rb") as f:
    normal_nmf = dill.load(f) # works
with open("custom.pkl","rb") as f:
    custom = dill.load(f) # works
with open("child.pkl","rb") as f:
    child = dill.load(f) # works
with open("nmf_wrapper.pkl","rb") as f:
    new_nmf = dill.load(f) # FAILS

问题

最后一次加载失败，堆栈跟踪如下：

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-2327d8f76861> in <module>
      7     child = dill.load(f)
      8 with open("nmf_wrapper.pkl","rb") as f:
----> 9     new_nmf = dill.load(f)

~/miniconda3/envs/test/lib/python3.9/site-packages/dill/_dill.py in load(file,ignore,**kwds)
    276 def load(file,ignore=None,**kwds):
    277     """unpickle an object from a file"""
--> 278     return Unpickler(file,ignore=ignore,**kwds).load()
    279 
    280 def loads(str,**kwds):

~/miniconda3/envs/test/lib/python3.9/site-packages/dill/_dill.py in load(self)
    479 
    480     def load(self): #NOTE: if settings change,need to update attributes
--> 481         obj = StockUnpickler.load(self)
    482         if type(obj).__module__ == getattr(_main_module,'__name__','__main__'):
    483             if not self._ignore:

~/miniconda3/envs/test/lib/python3.9/site-packages/dill/_dill.py in find_class(self,module,name)
    469             return type(None) #XXX: special case: nonetype missing
    470         if module == 'dill.dill': module = 'dill._dill'
--> 471         return StockUnpickler.find_class(self,name)
    472 
    473     def __init__(self,*args,**kwds):

AttributeError: Can't get attribute 'NMFwrapper' on <module '__main__'>

变化

用 dill 替换 pickle 更糟糕：只有第一次加载（正常 NMF）有效，其他三个都失败。

重新定义自定义 Wrapper（用于 dill）和自定义类/子类（用于 pickle）之前 unpickling 工作正常。

问题

为什么 dill/pickle 可以反序列化一个 NMF 实例，但在自定义类/子类上有各自的麻烦？能够反序列化自定义类和自定义类的子类，而不是“普通”类的子类，这很奇怪。