[FFmpeg-devel] [PATCH] Fix segv when decoding gray8

[Reimar Döffinger]

On Mon, Sep 27, 2010 at 02:56:45PM -0700, Baptiste Coudurier wrote:
> On 06/03/2010 02:56 PM, Baptiste Coudurier wrote:
> >On 06/03/2010 01:48 PM, Michael Niedermayer wrote:
> >>On Thu, Jun 03, 2010 at 12:11:41PM -0700, Baptiste Coudurier wrote:
> >>>On 06/03/2010 03:36 AM, Michael Niedermayer wrote:
> >>>>On Wed, Jun 02, 2010 at 06:39:26PM -0700, Baptiste Coudurier wrote:
> >>>>>Hi guys,
> >>>>>
> >>>>>$subject.
> >>>>>
> >>>>>frame->data[1] is set by avpicture_fill which is called with buf
> >>>>>passed
> >>>>>from the demuxer.
> >>>>>However, for gray8 the data stored is only width*height, so is too
> >>>>>small
> >>>>>for the palette, and will segv, memcpying.
> >>>>>
> >>>>>I'm not sure what is the right fix here, buf is supposed to be const.
> >>>>
> >>>>as this case can only arrise from use of deprecated (and buggy) palette
> >>>>passing API, the correct (long term goal) is to change all codecs so
> >>>>they
> >>>>finally stop using this highly unpredictable race condition ridden api.
> >>>>until then, your patch may be ok
> >>>
> >>>I don't think this is related to the API.
> >>>
> >>>You cannot generate the palette for gray8 (according to pixfmt.h)
> >>>_anyway_
> >>>since the data buffer is not big enough, since it comes from the demuxer
> >>>itself.
> >>>So either, the picture needs to be copied and the palette generated,
> >>>or we
> >>>skip the palette generation.
> >>
> >>maybe iam missing something but why cant data[1] that contains the
> >>palette
> >>point to some pal[256] from the context while data[0] would point to the
> >>buffer that originates from the demuxer?
> >
> >Sorry, I'll put this aside for now. This whole palette thing is too
> >complicated, and I prefer spending this time on something else.
> >
> 
> I believe Reimar kindly fixed this issue when addressing the palette
> problem.

I fear I failed to understand the explanation of the problem
(in particular that it is related to rawdec), otherwise
I might have fixed it earlier.