来自 Android 相机的视频与 libavformat 混合无法在所有播放器中播放且音频未同步

如何解决来自 Android 相机的视频与 libavformat 混合无法在所有播放器中播放且音频未同步

我正在使用 avformat 将从 Android 接收到的编码视频和音频混合到一个 mp4 文件中。生成的文件可通过 ffplay 播放，但有时会输出“No Frame！”在播放过程中。 VLC 可以播放它，但会出现一些小故障，看起来就像一个视频的运动数据与另一个视频的颜色数据相结合时的效果。我手机上的视频播放器根本不播放。

最重要的是，音频没有正确同步，即使 MediaCodec 设法生成了一个正确的文件，仅用下面的代码可用（即 presentationTimeStamp 以微秒为单位。

这是我的代码（为清楚起见省略了错误检查）：

// Initializing muxer
AVStream *videoStream = avformat_new_stream(outputContext,nullptr);
videoStreamIndex = videoStream->index;

videoStream->codecpar->codec_type = AVMEDIA_TYPE_VIDEO;
videoStream->codecpar->codec_id = AV_CODEC_ID_H264;
videoStream->codecpar->bit_rate = bitrate;
videoStream->codecpar->width = width;
videoStream->codecpar->height = height;
videoStream->time_base.num = 1;
videoStream->time_base.den = 90000;

AVStream* audioStream = avformat_new_stream(outputContext,nullptr);
audioStreamIndex = audioStream->index;
audioStream->codecpar->codec_type = AVMEDIA_TYPE_AUdio;
audioStream->codecpar->codec_id = AV_CODEC_ID_MP4ALS;
audioStream->codecpar->bit_rate = audiobitrate;
audioStream->codecpar->sample_rate = audiosampleRate;
audioStream->codecpar->channels = audioChannelCount;
audioStream->time_base.num = 1;
audioStream->time_base.den = 90000;

avformat_write_header(outputContext,&opts);

writtenAudio = writtenVideo = false;


// presentationTimeUs is the absolute timestamp when the encoded frame was received in Android code. 
// This is what is usually fed into MediaCodec
int writeVideoFrame(uint8_t *data,int size,int64_t presentationTimeUs) {
    AVPacket pkt;
    av_init_packet(&pkt);
    pkt.flags |= AV_PKT_FLAG_KEY; // I kNow setting this on every frame is wrong. When do I set it?
    pkt.data = data;
    pkt.size = size;
    pkt.dts = AV_nopTS_VALUE;
    pkt.pts = presentationTimeUs;
    if (writtenVideo) { // since the timestamp is absolute we have to subtract the initial offset
        pkt.pts -= firstVideoPts;
    }
    // rescale from microseconds to the stream timebase
    av_packet_rescale_ts(&pkt,AVRational { 1,1000000 },outputContext->streams[videoStreamIndex]->time_base);
    pkt.dts = AV_nopTS_VALUE;
    pkt.stream_index = videoStreamIndex;
    if (!writtenVideo) {
        AVStream* videoStream = outputContext->streams[videoStreamIndex];
        videoStream->start_time = pkt.pts;
        firstVideoPts = presentationTimeUs;
    }
    if (av_interleaved_write_frame(outputContext,&pkt) < 0) {
        return 1;
    }
    writtenVideo = true;
    return 0;
}

int writeAudioFrame(uint8_t *data,int64_t presentationTimeUs) {
    AVPacket pkt;
    av_init_packet(&pkt);
    pkt.data = data;
    pkt.size = size;
    pkt.stream_index = audioStreamIndex;
    pkt.pts = presentationTimeUs;
    av_packet_rescale_ts(&pkt,1000000},outputContext->streams[audioStreamIndex]->time_base);
    pkt.flags |= AV_PKT_FLAG_KEY;
    pkt.dts = AV_nopTS_VALUE;
    if (!writtenAudio) {
        outputContext->streams[audioStreamIndex]->start_time = pkt.pts;
    }
    if (av_interleaved_write_frame(outputContext,&pkt) < 0) {
        return 1;
    }
    writtenAudio = true;
    return 0;
}

void close() {
    av_write_trailer(outputContext);
    running = false;

    // cleanup AVFormatContexts etc
}

我想我正在做与 avformat 文档和示例中所示相同的操作，并且制作的视频有些可用（使用 ffmpeg 对其重新编码会产生一个工作视频）。但有些地方肯定还是有问题。