[Libav-user] Assertion failure with AV_PIX_FMT_VIDEOTOOLBOX

Nikolai Lineckiy nikolai.lineckiy at yandex.ru
Mon Dec 25 10:17:58 EET 2017

Hi. I came across a problem when using h264_videotoolbox encoder with AV_PIX_FMT_VIDEOTOOLBOX. It crashes with assertion failure on avcodec_open2:

Assertion *cv_img failed at src/libavcodec/videotoolboxenc.c:2049

The issue seems to be in following code in create_cv_pixel_buffer:

if (avctx->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
    av_assert0(frame->format == AV_PIX_FMT_VIDEOTOOLBOX);

    *cv_img = (CVPixelBufferRef)frame->data[3];

    return 0;

The calling sequence seems to be vtenc_init -> vtenc_populate_extradata -> vtenc_send_frame -> create_cv_pixel_buffer. It checks frame->data[3] for nullability, and always failes becuse during the construction of internal frame in vtenc_populate_extradata there is no initialization code for data[3].

Or maybe I'm missing something? 

More information about the Libav-user mailing list