转换tf检查点时，“ BertEmbeddings”对象没有属性“ bias”

如何解决转换tf检查点时，“ BertEmbeddings”对象没有属性“ bias”

当尝试将自我预训练的tensorflow BERT模型的检查点（使用Google的 create-pretraining.py 脚本）转换为使用 {{3} } 。

我总是会遇到以下错误： AttributeError：“ BertEmbeddings”对象没有属性“ bias”

init_vars名称（只是第一个）看起来像这样：

['bert/embeddings/layer_normalization/beta','bert/embeddings/layer_normalization/beta/adam_m','bert/embeddings/layer_normalization/beta/adam_v','bert/embeddings/layer_normalization/gamma','bert/embeddings/layer_normalization/gamma/adam_m','bert/embeddings/layer_normalization/gamma/adam_v']

产生错误的代码如下：

for m_name in name:                                                     
            if re.fullmatch(r"[A-Za-z]+_\d+",m_name):                          
                scope_names = re.split(r"_(\d+)",m_name)                       
            else:                                                               
                scope_names = [m_name]                                          
            if scope_names[0] == "kernel" or scope_names[0] == "gamma":         
                pointer = getattr(pointer,"weight")                            
            elif scope_names[0] == "output_bias" or scope_names[0] == "beta":   
                print(scope_names)                                              
                pointer = getattr(pointer,"bias")                              
            elif scope_names[0] == "output_weights":                            
                pointer = getattr(pointer,"weight")                            
            elif scope_names[0] == "squad":                                     
                pointer = getattr(pointer,"classifier")                        
            else:                                                               
                try:                                                            
                    pointer = getattr(pointer,scope_names[0])                  
                except AttributeError:                                          
                    logger.info("Skipping {}".format("/".join(name)))

遍历所有名称并从模型中获取正确的属性。当涉及到BertEmbeddings中的“层归一化”时，脚本会产生错误。之前有没有其他人来掩饰这个错误？您是如何解决的？

又是整个堆栈跟踪：

Traceback (most recent call last):
  File "convert_bert_original_tf_checkpoint_to_pytorch.py",line 62,in <module>
    convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path,args.bert_config_file,args.pytorch_dump_path)
  File "convert_bert_original_tf_checkpoint_to_pytorch.py",line 37,in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model,config,tf_checkpoint_path)
  File "/modeling_bert.py",line 136,in load_tf_weights_in_bert
    pointer = getattr(pointer,"bias")
  File "module.py",line 594,in __getattr__
    type(self).__name__,name))
AttributeError: 'BertEmbeddings' object has no attribute 'bias'

Bert Config如下：

Building PyTorch model from configuration: BertConfig {
  "attention_probs_dropout_prob": 0.1,"gradient_checkpointing": false,"hidden_act": "gelu","hidden_dropout_prob": 0.1,"hidden_size": 512,"initializer_range": 0.02,"intermediate_size": 2048,"layer_norm_eps": 1e-12,"max_position_embeddings": 512,"model_type": "bert","num_attention_heads": 8,"num_hidden_layers": 8,"pad_token_id": 0,"type_vocab_size": 2,"vocab_size": 30522
}

解决方法

原来，这只是“ layer_normalization”的名称，而不是“ LayerNorm”。我刚刚更改了脚本，现在可以使用了。

,

要添加到@ blueberry-cake666，可以将检查点中的tf变量从“ layer_normalization”重命名为“ LayerNorm”。您可以使用此脚本（https://gist.github.com/fvisin/578089ae098424590d3f25567b6ee255）重命名tf变量