音视频学习系列第（六）篇---音视频的分离与合成

音视频系列

什么是音视频的分离和合成

分离就是将视频1的声音和图像分别取出来
合成就是将视频1的图像和非视频1的声音组合成一个新的视频

如何进行音视频的分离和合成

安卓提供了两个API来帮助我们完成这个操作
MediaExtractor用于分离视频
MediaMuxer用于合成视频

下面我就来介绍一下这两个API的使用

MediaExtractor

分离音频
1.设置音频源

 MediaExtractor  audioExtractor = new MediaExtractor();
 audioExtractor.setDataSource(audioPath);

2.获取源文件中轨道的数量，并遍历找到我们需要的音频轨

  for (int i = 0; i < audioExtractor.getTrackCount(); i++) {
      MediaFormat format = audioExtractor.getTrackFormat(i);
      if (format.getString(MediaFormat.KEY_MIME).startsWith("audio/")) {
          srcATrackIndex = i;
          audioTrackIndex = muxer.addTrack(format);
          break;
      }
  }

分离视频
1.设置视频源

MediaExtractor  videoExtractor = new MediaExtractor();
videoExtractor.setDataSource(videoPath);

2.获取视频源文件中的轨道数，并遍历找到我们所需要的视频轨

for (int i = 0; i < videoExtractor.getTrackCount(); i++) {
       MediaFormat format = videoExtractor.getTrackFormat(i);
       if (format.getString(MediaFormat.KEY_MIME).startsWith("video/")) {
            srcVTrackIndex = i;
            videoTrackIndex = muxer.addTrack(format);
            break;
        }
 }

音频和视频额的分离方法一模一样，区别在于MediaFormat类型的不同，MediaFormat封装了媒体的数据格式信息

MediaMuxer

如何合成视频？
1.设置合成后视频的路径和格式

MediaMuxer  muxer = new MediaMuxer(outPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);

2.将MediaExtractor分离出来的音轨和视轨添加到自己的轨道中

audioTrackIndex = muxer.addTrack(format);
videoTrackIndex = muxer.addTrack(format);

3.添加完所有轨道后start

muxer.start();

4.采集音频源的音轨样本

audioExtractor.selectTrack(srcATrackIndex);   //移动到音频轨上
        if (audioTrackIndex != -1) {
            MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
            info.presentationTimeUs = 0;
            ByteBuffer buffer = ByteBuffer.allocate(100 * 1024);
            while (true) {
                int sampleSize = audioExtractor.readSampleData(buffer, 0);
                if (sampleSize < 0) {
                    //没有可获取的样本，退出循环
                    break;
                }

                info.offset = 0;
                info.size = sampleSize;
                info.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME;
                info.presentationTimeUs = audioExtractor.getSampleTime();

                muxer.writeSampleData(audioTrackIndex, buffer, info);//将样本写入新的轨道
                audioExtractor.advance(); //进入下一个样本
            }
 }

5.采集视频源的视频轨样本

videoExtractor.selectTrack(srcVTrackIndex);   //移动到视频轨上
        if (videoTrackIndex != -1) {
            MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
            info.presentationTimeUs = 0;
            ByteBuffer buffer = ByteBuffer.allocate(100 * 1024);
            while (true) {
                int sampleSize = videoExtractor.readSampleData(buffer, 0);
                if (sampleSize < 0) {
                    //没有可获取的样本，退出循环
                    break;
                }

                info.offset = 0;
                info.size = sampleSize;
                info.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME;
                info.presentationTimeUs = videoExtractor.getSampleTime();

                muxer.writeSampleData(videoTrackIndex, buffer, info);//将样本写入新的轨道
                videoExtractor.advance(); //进入下一个样本
            }
 }

6.停止并释放资源

muxer.stop();
muxer.release();

完整代码

public class MediaUtil {

private static final String TAG = "MediaUtil";
/**
 * @param audioPath 音频文件路劲
 * @param videoPath 视频文件路径
 * @param outPath   合成之后的保存路径
 */
public static void combineVideo(String audioPath, String videoPath, String outPath) {
    MediaMuxer muxer = null;
    MediaExtractor audioExtractor = null;
    MediaExtractor videoExtractor = null;
    try {
        muxer = new MediaMuxer(outPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);

        //找到音频文件的音频轨
        audioExtractor = new MediaExtractor();
        audioExtractor.setDataSource(audioPath);
        int srcATrackIndex = -1;    //音频源的音频轨
        int audioTrackIndex = -1; //音频轨添加到muxer后返回的新的轨道
        //在此循环，目的是找到我们需要的音频轨
        for (int i = 0; i < audioExtractor.getTrackCount(); i++) {
            MediaFormat format = audioExtractor.getTrackFormat(i);
            if (format.getString(MediaFormat.KEY_MIME).startsWith("audio/")) {
                srcATrackIndex = i;
                audioTrackIndex = muxer.addTrack(format);
                break;
            }
        }

        Log.d(TAG,"音频轨源索引："+srcATrackIndex+" 音频轨新索引:"+audioTrackIndex);

        //找到视频文件的视频轨
        videoExtractor = new MediaExtractor();
        videoExtractor.setDataSource(videoPath);
        int srcVTrackIndex = -1;  //视频源的视频轨
        int videoTrackIndex = -1;  //视频轨添加到muxer后返回的新的轨道
        for (int i = 0; i < videoExtractor.getTrackCount(); i++) {
            MediaFormat format = videoExtractor.getTrackFormat(i);
            if (format.getString(MediaFormat.KEY_MIME).startsWith("video/")) {
                srcVTrackIndex = i;
                videoTrackIndex = muxer.addTrack(format);
                break;
            }
        }

        Log.d(TAG,"视频轨源索引："+srcVTrackIndex+" 视频频轨新索引:"+videoTrackIndex);


        //添加完所有轨道后start
        muxer.start();
        Log.d(TAG,"开始合成视频...");

        //封装音频track
        audioExtractor.selectTrack(srcATrackIndex);   //移动到音频轨上
        if (audioTrackIndex != -1) {
            MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
            info.presentationTimeUs = 0;
            ByteBuffer buffer = ByteBuffer.allocate(100 * 1024);
            while (true) {
                int sampleSize = audioExtractor.readSampleData(buffer, 0);
                if (sampleSize < 0) {
                    //没有可获取的样本，退出循环
                    break;
                }

                info.offset = 0;
                info.size = sampleSize;
                info.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME;
                info.presentationTimeUs = audioExtractor.getSampleTime();

                muxer.writeSampleData(audioTrackIndex, buffer, info);//将样本写入新的轨道
                audioExtractor.advance(); //进入下一个样本
            }
        }
        Log.d(TAG,"音频轨样本采集完成");

        //封装视频track
        videoExtractor.selectTrack(srcVTrackIndex);   //移动到视频轨上
        if (videoTrackIndex != -1) {
            MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
            info.presentationTimeUs = 0;
            ByteBuffer buffer = ByteBuffer.allocate(100 * 1024);
            while (true) {
                int sampleSize = videoExtractor.readSampleData(buffer, 0);
                if (sampleSize < 0) {
                    //没有可获取的样本，退出循环
                    break;
                }

                info.offset = 0;
                info.size = sampleSize;
                info.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME;
                info.presentationTimeUs = videoExtractor.getSampleTime();

                muxer.writeSampleData(videoTrackIndex, buffer, info);//将样本写入新的轨道
                videoExtractor.advance(); //进入下一个样本
            }
        }
        muxer.stop();
        Log.d(TAG,"视频轨样本采集完成");
        Log.d(TAG,"视频合成完毕："+outPath);
    } catch (IOException e) {
        e.printStackTrace();
        Log.d(TAG,"合成出错:"+e.getMessage());
    } finally {
        //释放资源
        if(audioExtractor!=null){
            audioExtractor.release();
        }
        if(videoExtractor!=null){
            videoExtractor.release();
        }
        if(muxer!=null){
            muxer.release();
        }
    }
}
}

踩坑总结

问题1
MPEG4Writer: Unsupported mime 'audio/mpeg'
当音频文件是mp3时会报这个错误 ,需要的格式是AAC，m4a
stackoverflow
我将两个源文件都替换了mp4格式后不再报错了

问题2
WVMExtractor: Failed to open libwvm.so: dlopen failed: library "libwvm.so" not found
小米5 6.0.1机型上出现的问题，意思是缺少so库

由于我就是这款手机，也没测过其他手机有木有这个so库
后续我会用其他手机在测试这个问题

代码地址

MediaUtil类的封装放置在libplayer下的util包下
方法调用放置在app/demo/media/track下