Incorrect Memory Layout when using avpicture_layout with PIX_FMT_NV12

[jkersch]

when decoding (e.g. a h264 stream) with output format PIX_FMT_NV12, the AVFrame contains 3 data buffers with Luma and Cb and Cr planes.

when using avpicture_layout to copythe AVPicture to a previously allocated memory buffer, the bytes are layouted incorrectly.
NV12 is all Luma bytes first, and after that the Cb and Cr bytes interleaved (see  http://msdn.microsoft.com/en-us/library/windows/desktop/dd206750(v=vs.85).aspx#nv12)

i´ve worked around this manually, but it would be great if this incorrect behavior could be fixed within libavcodec itself.

[cehoyos]

Replying to jkersch:

when decoding (e.g. a h264 stream) with output format PIX_FMT_NV12

I don't think this is possible with libavcodec.

The only decoder that supports NV12 is rawvideo.
A rawvideo with pix_fmt nv12 produced with FFmpeg (that can be decoded correctly with FFmpeg) plays also fine with mplayer -vo vdpau (which supports displaying NV12).

[jkersch]

the Luma plane is correct, however, for Cb and Cr there are two different buffers instead of an interleaved one where the byte are merged in the form (Cb)(Cr)(Cb)(Cr).
linesize however correctly displays only 2 planes.
any chance to get this layouted directly within avpicture_layout?

i want to pack those buffers into a media sample to use avcodec as a Media Foundation Transform decoder. (which works great, except the hack to output NV12)

[cehoyos]

Replying to jkersch:

the Luma plane is correct, however, for Cb and Cr there are two different buffers instead of an interleaved one where the byte are merged in the form (Cb)(Cr)(Cb)(Cr).

That is because libavcodec's H.264 decoder can only output to yuv420p (if the original sample is 8bit yuv420 which I assume). It cannot output to nv12, no matter what the input is.

libswscale can convert yuv420p to nv12.