[FFmpeg-devel] Hardware decoding buffer size

[Steve Lhomme]

Hi,

We have this bug in VLC with an MPEG2 file that decodes with green
bars at the top. After investigation it's because we allocate a
1920x1080 texture when it should actually need 1920x1088.
https://trac.videolan.org/vlc/ticket/17856

In software decoding it works because we adjust the
`ctx->coded_height` with avcodec_align_dimensions2(). But we do not
call it for hardware decoding. And looking at the code it seems it's
not meant to happen anyway (the DXVA pixel formats have no size
constraint).

So my question is: is it the job of libavcodec hardware decoders to
set the proper coded_width/coded_height, in which case there's a bug,
or it should be done outside of libavcodec ? And if so, how ?

Thanks