[FFmpeg-devel] avpicture_layout() writes passed the buffer length for pseudo-paletted formats (bug)

[Matthew Einhorn]

Hi,

The problem is that in the docs avpicture_layout() says it writes a
picture with size avpicture_get_size(). So one would make the buffer
input to avpicture_layout() of that size. However, for pseudo-paletted
formats (as listed in avpicture_get_size()) the amount written to
buffer is avpicture_get_size()+1024 (which is also the
avpicture_fill() size).

If the buffer size is less than avpicture_get_size() (as indicated by
dest_size in avpicture_layout()) an error is returned by
avpicture_layout(). However, for these pseudo-paletted formats no
error is returned if the buffer is less than avpicture_get_size()+1024
and the function simply writes passed the actual buffer length for a
total of avpicture_get_size()+1024.

The reason for this it seems is that avpicture_layout() wasn't
supposed to write the palette for these pseudo-paletted formats, but
the function still writes it resulting in the problem.


The solution is to:
*Either remove the exception for pseudo-paletted formats so that
avpicture_get_size() now completely relies on avpicture_fill() (will
probably break backwards compatibility) and both return the same size.
*Make an exception in avpicture_layout() as well for the
pseudo-paletted formats so that the palette is not written for them.
*If this "is" intended behavior update avpicture_layout() docs to
indicate that for these formats the buffer needs to be larger than
avpicture_get_size() by 1024.
*Or maybe there's some other deeper issue here (other then to prevent
copying of a standard palette) as to why these pseudo-paletted formats
are treated specially?

I'd probably select option 2 otherwise it'll probably break backward
compatibility.

I can write the patch for whatever is selected.

Thanks,
Matt