[Libav-user] Problem with h264_videotoolbox and AV_PIX_FMT_VIDEOTOOLBOX

[Steve Green]

While trying to debug a problem using YUV420P with h264_videotoolbox, I came across AV_PIX_FMT_VIDEOTOOLBOX in the encoder code.  Depending on what I discover here, I might be back in another thread about a YUV issue.

It appears that one can specify AV_PIX_FMT_VIDEOTOOLBOX and then provide CVPixelBuffers in data[3].  The issue Im having is that avcodec_open2() finds its way to vtenc_send_frame().  I don’t understand what is going on at this level but the internally constructed AVFrame will not have anything in data[3] and the following code in create_cv_pixel_buffer() fails on the second av_assert0().

>     if (avctx->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
>         av_assert0(frame->format == AV_PIX_FMT_VIDEOTOOLBOX);
> 
>         *cv_img = (CVPixelBufferRef)frame->data[3];
>         av_assert0(*cv_img);
> 
>         CFRetain(*cv_img);
>         return 0;
>     }

Is there something that I need to do to make this work?

Thanks..

~S