[FFmpeg-devel] [PATCH] avcodec/h264, videotoolbox: do not return invalid frames on failure

wm4 nfxjfg at gmail.com
Tue Nov 17 15:19:29 CET 2015

If videotoolbox_common_end_frame failed, then the AVFrame was returned
to the API user with the dummy buffer (in AVFrame.buf[0]) still set, and
the decode call indicating success.

These "half-set" AVFrames with dummy buffer are a videotoolbox specific
hack, because the decoder requires an allocated AVFrame for its internal
logic. Videotoolbox on the other hand allocates its frame itself
internally, and outputs it only on end_frame. At this point, the dummy
buffer is replaced with the real frame (unless decoding fails).
Better solutions welcome. For now, this fixes the API returning garbage
which can crash or confuse API users,
 libavcodec/h264_refs.c    | 3 ++-
 libavcodec/videotoolbox.c | 2 ++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/libavcodec/h264_refs.c b/libavcodec/h264_refs.c
index 619f2ed..9d0641a 100644
--- a/libavcodec/h264_refs.c
+++ b/libavcodec/h264_refs.c
@@ -515,7 +515,8 @@ void ff_h264_remove_all_refs(H264Context *h)
     if (h->short_ref_count && !h->last_pic_for_ec.f->data[0]) {
         ff_h264_unref_picture(h, &h->last_pic_for_ec);
-        ff_h264_ref_picture(h, &h->last_pic_for_ec, h->short_ref[0]);
+        if (h->short_ref[0]->f->buf[0])
+            ff_h264_ref_picture(h, &h->last_pic_for_ec, h->short_ref[0]);
     for (i = 0; i < h->short_ref_count; i++) {
diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c
index cc1e592..2f4d531 100644
--- a/libavcodec/videotoolbox.c
+++ b/libavcodec/videotoolbox.c
@@ -353,6 +353,8 @@ static int videotoolbox_common_end_frame(AVCodecContext *avctx, AVFrame *frame)
     AVVideotoolboxContext *videotoolbox = avctx->hwaccel_context;
     VTContext *vtctx = avctx->internal->hwaccel_priv_data;
+    av_buffer_unref(&frame->buf[0]);
     if (!videotoolbox->session || !vtctx->bitstream)
         return AVERROR_INVALIDDATA;

More information about the ffmpeg-devel mailing list