ASR - Speech to Text - 如何通过将标签张量与音频样本数组的长度对齐来预处理音频文件的文字转录标签

如何解决ASR - Speech to Text - 如何通过将标签张量与音频样本数组的长度对齐来预处理音频文件的文字转录标签

我正在为端到端自动语音识别任务实施深度学习模型。我正在使用 LibriSpeech 语料库数据集。我在数据集的预处理中遇到了一个问题。至于音频 (.wav) 数据，我获取采样音频，然后提取 mfcc 特征，这代表我的输入张量。至于成绩单，我通过将每个带有相应索引的字符编码到字母字典中来生成一个数组。我的模型将输出一个长度等于 mfcc 特征张量的时间长度的张量，我想要一个长度相同的标签张量，即通过将字符和/或单词编码对齐到标签张量中音频文件中字符/单词的发声时间。 为了更好地阐明我的意思，我在下面留下了一个 python 脚本，它可以做我想要的，但在音素识别的情况下。我正在尝试修改它以适应字符/单词识别的情况，但我不确定识别音频中话语的正确方法。在下面所示的情况下，音素文件抄本提供了音素出现在音频文件中的开始和结束时间。是否有可能对字符/单词识别也有类似的结果，而无需事先了解每个字符/单词进入原始音频的时间？ 提前感谢您的回答。 干杯

phn_name = str(label_files[i])
    wav_name = str(wav_files[i])

    if (wav_name.startswith("SA")):  # specific for TIMIT: these files contain strong dialects; don't use them
        continue

    # Get MFCC of the WAV
    X_val,total_frames = create_mfcc(wav_name,feat_args,feat_order) # get 3 or 4 levels: 0th,1st,2nd (and 3rd) derivative (=> 3*13 + 1 = 39 * num_levels coefficients)
                                      # max_sample_len)
    total_frames = int(total_frames)

    X.append(X_val)

    # Get phonemes and valid frame numbers out of .phn files
    total_duration = get_total_duration(phn_name)
    fr = open(phn_name)

    # some .PHN files don't start at 0. Set default phoneme to silence (expected at the end of phoneme_set_list)
    y_vals = np.zeros(total_frames) + phoneme_classes[phoneme_set_list[-1]]
    valid_frames_vals = []

    for line in fr:
        [start_time,end_time,phoneme] = line.rstrip('\n').split()
        start_time = int(start_time)
        end_time = int(end_time)
        start_ind = int(np.round(start_time / (total_duration / total_frames)))
        end_ind = int(np.round(end_time / (total_duration / total_frames)))

        valid_ind = int((start_ind + end_ind) / 2)
        valid_frames_vals.append(valid_ind)

        phoneme_num = phoneme_classes[phoneme]
        # check that phoneme is found in dict
        if (phoneme_num == -1):
            logger.error("In file: %s,phoneme not found: %s",phn_name,phoneme)
            pdb.set_trace()
        y_vals[start_ind:end_ind] = phoneme_num

        if verbose and logger:
            logger.debug('%s',(total_frames / float(total_duration)))
            logger.debug('TIME  start: %s end: %s,phoneme: %s,class: %s',start_time,phoneme,phoneme_num)
            logger.debug('FRAME start: %s end: %s,start_ind,end_ind,phoneme_num)
    fr.close()