[Libav-user] Decoding h264 (NV12) stream from NVENC

[Michael IV]

Hi! I am using NVENC encoder to create h264 stream.The YUV format is NV12.
Now, I need also to decode that stream back to YUV.I wrote a module for
that,based on
this example:

https://gist.github.com/roxlu/9329339

But, because my YUV format is not 420P, but NV12 I am trying to force it on
AVCodecContext when initializing it with this line:

        avcodec_register_all();
        mCodec = avcodec_find_decoder(AV_CODEC_ID_H264);
        mCodecContext = avcodec_alloc_context3(mCodec);
        mCodecContext->pix_fmt = AVPixelFormat::AV_PIX_FMT_NV12;


But when I am receiving the decoded AVFrame, I see that the AVFrame->format
property is
set to zero, which is enum for 420P. Also inspecting the planes and other
props of this struct I see that the decoder tried to decode it as 420P. (3
planes , 3 line sizes etc.) .And of course,when trying to play the output
of such a YUV frame , there is just a green screen.
I am trying to understand where is my error.Is there some data that decoder
misses from the h264 stream and it doesn't allow it to guess the pixel
format?Or maybe I need to set pixel format for the codec context in a some
different way?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20170731/51c1f56f/attachment.html>