ffmpeg内存模型及AVPacket和AVFrame API基本使用

ffmpeg内存模型及AVPacket和AVFrame API解释

---

目录

1. ffmpeg内存模型
2. AVPacket常用API
3. AVPacket Demo
4. AVFrame常用API

---

1. ffmpeg内存模型

/**
 * Supply raw packet data as input to a decoder.
 *
 * Internally, this call will copy relevant AVCodecContext fields, which can
 * influence decoding per-packet, and apply them when the packet is actually
 * decoded. (For example AVCodecContext.skip_frame, which might direct the
 * decoder to drop the frame contained by the packet sent with this function.)
 *
 * @warning The input buffer, avpkt->data must be AV_INPUT_BUFFER_PADDING_SIZE
 *          larger than the actual read bytes because some optimized bitstream
 *          readers read 32 or 64 bits at once and could read over the end.
 *
 * @warning Do not mix this API with the legacy API (like avcodec_decode_video2())
 *          on the same AVCodecContext. It will return unexpected results now
 *          or in future libavcodec versions.
 *
 * @note The AVCodecContext MUST have been opened with @ref avcodec_open2()
 *       before packets may be fed to the decoder.
 *
 * @param avctx codec context
 * @param[in] avpkt The input AVPacket. Usually, this will be a single video
 *                  frame, or several complete audio frames.
 *                  Ownership of the packet remains with the caller, and the
 *                  decoder will not write to the packet. The decoder may create
 *                  a reference to the packet data (or copy it if the packet is
 *                  not reference-counted).
 *                  Unlike with older APIs, the packet is always fully consumed,
 *                  and if it contains multiple frames (e.g. some audio codecs),
 *                  will require you to call avcodec_receive_frame() multiple
 *                  times afterwards before you can send a new packet.
 *                  It can be NULL (or an AVPacket with data set to NULL and
 *                  size set to 0); in this case, it is considered a flush
 *                  packet, which signals the end of the stream. Sending the
 *                  first flush packet will return success. Subsequent ones are
 *                  unnecessary and will return AVERROR_EOF. If the decoder
 *                  still has frames buffered, it will return them after sending
 *                  a flush packet.
 *
 * @return 0 on success, otherwise negative error code:
 *      AVERROR(EAGAIN):   input is not accepted in the current state - user
 *                         must read output with avcodec_receive_frame() (once
 *                         all output is read, the packet should be resent, and
 *                         the call will not fail with EAGAIN).
 *      AVERROR_EOF:       the decoder has been flushed, and no new packets can
 *                         be sent to it (also returned if more than 1 flush
 *                         packet is sent)
 *      AVERROR(EINVAL):   codec not opened, it is an encoder, or requires flush
 *      AVERROR(ENOMEM):   failed to add packet to internal queue, or similar
 *      other errors: legitimate decoding errors
 */
int avcodec_send_packet(AVCodecContext *avctx, const AVPacket *avpkt);

/**
 * Return decoded output data from a decoder.
 *
 * @param avctx codec context
 * @param frame This will be set to a reference-counted video or audio
 *              frame (depending on the decoder type) allocated by the
 *              decoder. Note that the function will always call
 *              av_frame_unref(frame) before doing anything else.
 *
 * @return
 *      0:                 success, a frame was returned
 *      AVERROR(EAGAIN):   output is not available in this state - user must try
 *                         to send new input
 *      AVERROR_EOF:       the decoder has been fully flushed, and there will be
 *                         no more output frames
 *      AVERROR(EINVAL):   codec not opened, or it is an encoder
 *      AVERROR_INPUT_CHANGED:   current decoded frame has changed parameters
 *                               with respect to first decoded frame. Applicable
 *                               when flag AV_CODEC_FLAG_DROPCHANGED is set.
 *      other negative values: legitimate decoding errors
 */
int avcodec_receive_frame(AVCodecContext *avctx, AVFrame *frame);

2. 从av_read_frame读取到一个AVPacket后怎么放入队列？从avcodec_recevice_frame读取到一个AVFrame后又怎么放入队列？

1. 从现有的Packet拷贝一个新Packet的时候，有两种情况：

   1. 两个Packet的buf引用的是同一数据缓存空间，这时候要注意数据缓存空间的释放问题；
   2. 两个Packet的buf引用不同的数据缓存空间，每个Packet都有数据缓存空间的copy；
      

2. AVPacket和AVFrame内部都封装了AVBufferRef

3. AVBufferRef真正存储数据的是AVBuffer

4. AVBuffer的data是真正存数据的，refcount是引用计数
   

5. 更精确一点
   
   
   

6. 对于多个AVPacket共享同一个缓存空间， FFmpeg使用的引用计数的机制（reference-count） ：

   1. 初始化引用计数为0，只有真正分配AVBuffer的时候，引用计数初始化为1;
   2. 当有新的Packet引用共享的缓存空间时， 就将引用计数+1；
   3. 当释放了引用共享空间的Packet，就将引用计数-1；引用计数为0时，就释放掉引用的缓存空间AVBuffer。

7. AVFrame也是采用同样的机制。

---

2. AVPacket常用API

函数原型	说明
AVPacket *av_packet_alloc(void);	分配AVPacket，这个时候和buffer没有关系
void av_packet_free(AVPacket **pkt);	释放AVPacket和_alloc对应
void av_init_packet(AVPacket *pkt);	初始化AVPacket，只是单纯初始化pkt字段
int av_new_packet(AVPacket *pkt, int size);	给AVPacket的buf分配内存， 引用计数初始化为1
int av_packet_ref(AVPacket *dst, const AVPacket *src);	增加引用计数
void av_packet_unref(AVPacket *pkt);	减少引用计数
void av_packet_move_ref(AVPacket *dst, AVPacket *src);	转移引用计数
AVPacket *av_packet_clone(const AVPacket *src);	等于av_packet_alloc()+av_packet_ref()

0. AVPacket结构体

/**
 * This structure stores compressed data. It is typically exported by demuxers
 * and then passed as input to decoders, or received as output from encoders and
 * then passed to muxers.
 *
 * For video, it should typically contain one compressed frame. For audio it may
 * contain several compressed frames. Encoders are allowed to output empty
 * packets, with no compressed data, containing only side data
 * (e.g. to update some stream parameters at the end of encoding).
 *
 * AVPacket is one of the few structs in FFmpeg, whose size is a part of public
 * ABI. Thus it may be allocated on stack and no new fields can be added to it
 * without libavcodec and libavformat major bump.
 *
 * The semantics of data ownership depends on the buf field.
 * If it is set, the packet data is dynamically allocated and is
 * valid indefinitely until a call to av_packet_unref() reduces the
 * reference count to 0.
 *
 * If the buf field is not set av_packet_ref() would make a copy instead
 * of increasing the reference count.
 *
 * The side data is always allocated with av_malloc(), copied by
 * av_packet_ref() and freed by av_packet_unref().
 *
 * @see av_packet_ref
 * @see av_packet_unref
 */
typedef struct AVPacket {
    /**
     * A reference to the reference-counted buffer where the packet data is
     * stored.
     * May be NULL, then the packet data is not reference-counted.
     */
    AVBufferRef *buf;
    /**
     * Presentation timestamp in AVStream->time_base units; the time at which
     * the decompressed packet will be presented to the user.
     * Can be AV_NOPTS_VALUE if it is not stored in the file.
     * pts MUST be larger or equal to dts as presentation cannot happen before
     * decompression, unless one wants to view hex dumps. Some formats misuse
     * the terms dts and pts/cts to mean something different. Such timestamps
     * must be converted to true pts/dts before they are stored in AVPacket.
     */
    int64_t pts;
    /**
     * Decompression timestamp in AVStream->time_base units; the time at which
     * the packet is decompressed.
     * Can be AV_NOPTS_VALUE if it is not stored in the file.
     */
    int64_t dts;
    uint8_t *data;
    int   size;
    int   stream_index;
    /**
     * A combination of AV_PKT_FLAG values
     */
    int   flags;
    /**
     * Additional packet data that can be provided by the container.
     * Packet can contain several types of side information.
     */
    AVPacketSideData *side_data;
    int side_data_elems;

    /**
     * Duration of this packet in AVStream->time_base units, 0 if unknown.
     * Equals next_pts - this_pts in presentation order.
     */
    int64_t duration;

    int64_t pos;                            ///< byte position in stream, -1 if unknown

#if FF_API_CONVERGENCE_DURATION
    /**
     * @deprecated Same as the duration field, but as int64_t. This was required
     * for Matroska subtitles, whose duration values could overflow when the
     * duration field was still an int.
     */
    attribute_deprecated
    int64_t convergence_duration;
#endif
} AVPacket;

1. AVPacket *av_packet_alloc(void);

1. 解释

   1. 简单的创建一个AVPacket，将其字段设为默认值（data为空，没有数据缓存空间），data的指针需要另外去赋值。
   2. 分配AVPacket，初始化pkt字段，这个时候和buffer没有关系，内部调用了void av_init_packet(AVPacket *pkt);

2. 源码

AVPacket *av_packet_alloc(void)
{
    AVPacket *pkt = av_mallocz(sizeof(AVPacket));
    if (!pkt)
        return pkt;

    av_init_packet(pkt);

    return pkt;
}

2. void av_packet_free(AVPacket **pkt);

1. 解释

   1. 释放AVPacket和_alloc对应。内部调用了void av_packet_unref(AVPacket *pkt);

2. 源码

void av_packet_free(AVPacket **pkt)
{
    if (!pkt || !*pkt)
        return;

    av_packet_unref(*pkt);
    av_freep(pkt);
}

3. void av_init_packet(AVPacket *pkt);

1. 解释

   1. 初始化AVPacket，只是单纯初始化pkt字段.

2. 源码

void av_init_packet(AVPacket *pkt)
{
    pkt->pts                  = AV_NOPTS_VALUE;
    pkt->dts                  = AV_NOPTS_VALUE;
    pkt->pos                  = -1;
    pkt->duration             = 0;
#if FF_API_CONVERGENCE_DURATION
FF_DISABLE_DEPRECATION_WARNINGS
    pkt->convergence_duration = 0;
FF_ENABLE_DEPRECATION_WARNINGS
#endif
    pkt->flags                = 0;
    pkt->stream_index         = 0;
    pkt->buf                  = NULL;
    pkt->side_data            = NULL;
    pkt->side_data_elems      = 0;
}

4. int av_new_packet(AVPacket *pkt, int size);

1. 解释
   1. 给AVPacket的AVBufferRef分配内存， 引用计数初始化为1。内部调用了void av_init_packet(AVPacket *pkt)
2. 源码

int av_new_packet(AVPacket *pkt, int size)
{
    AVBufferRef *buf = NULL;
    int ret = packet_alloc(&buf, size);
    if (ret < 0)
        return ret;

    av_init_packet(pkt);
    pkt->buf      = buf;
    pkt->data     = buf->data;
    pkt->size     = size;

    return 0;
}

5. int av_packet_ref(AVPacket *dst, const AVPacket *src);

1. 解释
   1. 使用引用计数的浅拷贝
   2. 该函数会先拷贝所有非缓存类数据，然后创建一个src->buf的新的引用计数。如果src已经设置了引用计数发（src->buf不为空），则直接将其引用计数+1；
      如果src没有设置引用计数（src->buf为空），则为dst创建一个新的引用计数buf，并复制src->data到dst->buf->data和dst-data中。
   3. 最后，复制src的其他字段到dst中。所以av_packet_ref()是将2个AVPacket共用一个缓存的。
2. 源码

int av_packet_ref(AVPacket *dst, const AVPacket *src)
{
    int ret;

    dst->buf = NULL;

    ret = av_packet_copy_props(dst, src);
    if (ret < 0)
        goto fail;

    if (!src->buf) {
        ret = packet_alloc(&dst->buf, src->size);
        if (ret < 0)
            goto fail;
        av_assert1(!src->size || src->data);
        if (src->size)
            memcpy(dst->buf->data, src->data, src->size);

        dst->data = dst->buf->data;
    } else {
        dst->buf = av_buffer_ref(src->buf);
        if (!dst->buf) {
            ret = AVERROR(ENOMEM);
            goto fail;
        }
        dst->data = src->data;
    }

    dst->size = src->size;

    return 0;
fail:
    av_packet_unref(dst);
    return ret;
}

6. void av_packet_unref(AVPacket *pkt);

1. 解释
   1. 使用引用计数清理数据
   2. 将缓存空间的引用计数-1，并将Packet中的其他字段设为初始值。如果引用计数为0，自动的释放缓存空间。
2. 源码

void av_packet_unref(AVPacket *pkt)
{
    av_packet_free_side_data(pkt);
    av_buffer_unref(&pkt->buf);
    av_init_packet(pkt);
    pkt->data = NULL;
    pkt->size = 0;
}

7. void av_packet_move_ref(AVPacket *dst, AVPacket *src);

1. 解释：
   1. 转移引用计数
   2. 把src整个结构体直接赋值给dst，所以引用计数没有发生变化，并且src被av_init_packet重置
2. 源码

void av_packet_move_ref(AVPacket *dst, AVPacket *src)
{
    *dst = *src;
    av_init_packet(src);
    src->data = NULL;
    src->size = 0;
}

8. AVPacket *av_packet_clone(const AVPacket *src);

1. 解释
   1. 先创建一个新的AVPacket，然后再进行计数引用+数据拷贝，使得新的AVPacket指向老的AVPacket同一个data。
   2. 等于av_packet_alloc()+av_packet_ref()
2. 源码

AVPacket *av_packet_clone(const AVPacket *src)
{
    AVPacket *ret = av_packet_alloc();

    if (!ret)
        return ret;

    if (av_packet_ref(ret, src))
        av_packet_free(&ret);

    return ret;
}

3. AVPacket Demo

#define MEM_ITEM_SIZE (20*1024*102)
#define AVPACKET_LOOP_COUNT 1000
// 测试 内存泄漏
/**
 * @brief 测试av_packet_alloc和av_packet_free的配对使用
 */
void av_packet_test1() {
    AVPacket *pkt = NULL;
    int ret = 0;

    pkt = av_packet_alloc();
    ret = av_new_packet(pkt, MEM_ITEM_SIZE); // 引用计数初始化为1
    memccpy(pkt->data, (void *) &av_packet_test1, 1, MEM_ITEM_SIZE);
    av_packet_unref(pkt);       // 要不要调用
    av_packet_free(&pkt);       // 如果不free将会发生内存泄漏,内部调用了 av_packet_unref
}

/**
 * @brief 测试误用av_init_packet将会导致内存泄漏
 */
void av_packet_test2() {
    AVPacket *pkt = NULL;
    int ret = 0;

    pkt = av_packet_alloc();
    ret = av_new_packet(pkt, MEM_ITEM_SIZE);
    memccpy(pkt->data, (void *) &av_packet_test1, 1, MEM_ITEM_SIZE);
//    av_init_packet(pkt);        // 这个时候init就会导致内存无法释放
    av_packet_free(&pkt);
}

/**
 * @brief 测试av_packet_move_ref后，可以av_init_packet
 */
void av_packet_test3() {
    AVPacket *pkt = NULL;
    AVPacket *pkt2 = NULL;
    int ret = 0;

    pkt = av_packet_alloc();
    ret = av_new_packet(pkt, MEM_ITEM_SIZE);
    memccpy(pkt->data, (void *) &av_packet_test1, 1, MEM_ITEM_SIZE);
    pkt2 = av_packet_alloc();   // 必须先alloc
    av_packet_move_ref(pkt2, pkt);//内部其实也调用了av_init_packet
    av_init_packet(pkt);
    av_packet_free(&pkt);
    av_packet_free(&pkt2);
}

/**
 * @brief 测试av_packet_clone
 */
void av_packet_test4() {
    AVPacket *pkt = NULL;
    // av_packet_alloc()没有必要，因为av_packet_clone内部有调用 av_packet_alloc
    AVPacket *pkt2 = NULL;
    int ret = 0;

    pkt = av_packet_alloc();
    ret = av_new_packet(pkt, MEM_ITEM_SIZE);
    memccpy(pkt->data, (void *) &av_packet_test1, 1, MEM_ITEM_SIZE);
    pkt2 = av_packet_clone(pkt); // av_packet_alloc()+av_packet_ref()
    av_init_packet(pkt);
    av_packet_free(&pkt);
    av_packet_free(&pkt2);
}

/**
 * @brief 测试av_packet_ref
 */
void av_packet_test5() {
    AVPacket *pkt = NULL;
    AVPacket *pkt2 = NULL;
    int ret = 0;

    pkt = av_packet_alloc(); //
    if (pkt->buf)        // 打印referenc-counted，必须保证传入的是有效指针
    {
        printf("%s(%d) ref_count(pkt) = %d\n", __FUNCTION__, __LINE__,
               av_buffer_get_ref_count(pkt->buf));
    }

    ret = av_new_packet(pkt, MEM_ITEM_SIZE);
    if (pkt->buf)        // 打印referenc-counted，必须保证传入的是有效指针
    {
        printf("%s(%d) ref_count(pkt) = %d\n", __FUNCTION__, __LINE__,
               av_buffer_get_ref_count(pkt->buf));
    }
    memccpy(pkt->data, (void *) &av_packet_test1, 1, MEM_ITEM_SIZE);

    pkt2 = av_packet_alloc();   // 必须先alloc
    av_packet_move_ref(pkt2, pkt); // av_packet_move_ref
//    av_init_packet(pkt);  //av_packet_move_ref

    av_packet_ref(pkt, pkt2);
    av_packet_ref(pkt, pkt2);     // 多次ref如果没有对应多次unref将会内存泄漏
    if (pkt->buf)        // 打印referenc-counted，必须保证传入的是有效指针
    {
        printf("%s(%d) ref_count(pkt) = %d\n", __FUNCTION__, __LINE__,
               av_buffer_get_ref_count(pkt->buf));
    }
    if (pkt2->buf)        // 打印referenc-counted，必须保证传入的是有效指针
    {
        printf("%s(%d) ref_count(pkt) = %d\n", __FUNCTION__, __LINE__,
               av_buffer_get_ref_count(pkt2->buf));
    }
    av_packet_unref(pkt);   // 将为2
    av_packet_unref(pkt);   // 做第二次是没有用的
    if (pkt->buf)
        printf("pkt->buf没有被置NULL\n");
    else
        printf("pkt->buf已经被置NULL\n");
    if (pkt2->buf)        // 打印referenc-counted，必须保证传入的是有效指针
    {
        printf("%s(%d) ref_count(pkt) = %d\n", __FUNCTION__, __LINE__,
               av_buffer_get_ref_count(pkt2->buf));
    }
    av_packet_unref(pkt2);


    av_packet_free(&pkt);
    av_packet_free(&pkt2);
}

/**
 * @brief 测试AVPacket整个结构体赋值, 和av_packet_move_ref类似
 */
void av_packet_test6() {
    AVPacket *pkt = NULL;
    AVPacket *pkt2 = NULL;
    int ret = 0;

    pkt = av_packet_alloc();
    ret = av_new_packet(pkt, MEM_ITEM_SIZE);
    memccpy(pkt->data, (void *) &av_packet_test1, 1, MEM_ITEM_SIZE);

    pkt2 = av_packet_alloc();   // 必须先alloc
    *pkt2 = *pkt;   // 有点类似  pkt可以重新分配内存
    av_init_packet(pkt);

    av_packet_free(&pkt);
    av_packet_free(&pkt2);
}

 

---

3. AVFrame常用API

函数原型	说明
AVFrame *av_frame_alloc(void);	分配AVFrame
void av_frame_free(AVFrame **frame);	释放AVFrame
int av_frame_ref(AVFrame *dst, const AVFrame *src);	增加引用计数
void av_frame_unref(AVFrame *frame);	减少引用计数
void av_frame_move_ref(AVFrame *dst, AVFrame *src);	转移引用计数
int av_frame_get_buffer(AVFrame *frame, int align);	根据AVFrame分配内存
AVFrame *av_frame_clone(const AVFrame *src);	等于av_frame_alloc()+av_frame_ref()

1. 解释同上