[FFmpeg-devel] [PATCH] get rid of nonsense palette pointer for RGB8, GRAY8 etc.

[Reimar Döffinger]

On Fri, Mar 20, 2009 at 11:28:14PM +0100, Michael Niedermayer wrote:
> On Fri, Mar 20, 2009 at 09:58:37PM +0100, Reimar D?ffinger wrote:
> > On Fri, Mar 20, 2009 at 09:12:49PM +0100, Michael Niedermayer wrote:
> > > On Fri, Mar 20, 2009 at 08:30:38PM +0100, Reimar D?ffinger wrote:
> > > > About fixing the swscale planarCopy, are you fine with this ugly hack (I
> > > > assume there is no need to "reset" the GRAY8 palette data in dst):
> > > 
> > > no, sws_scale() passes dst through and this would end up changing the
> > > users dst[] i think
> > 
> > Sorry, that was a thinking error. Even more ugly, but this is what I
> > meant:
> > Index: libswscale/swscale.c
> > ===================================================================
> > --- libswscale/swscale.c        (revision 29018)
> > +++ libswscale/swscale.c        (working copy)
> > @@ -2107,10 +2107,15 @@
> >      return srcSliceH;
> >  }
> >  
> > -static int planarCopy(SwsContext *c, uint8_t* src[], int srcStride[], int srcSliceY,
> > -                      int srcSliceH, uint8_t* dst[], int dstStride[])
> > +static int planarCopy(SwsContext *c, uint8_t* srcp[], int srcStride[], int srcSliceY,
> > +                      int srcSliceH, uint8_t* dstp[], int dstStride[])
> >  {
> >      int plane;
> > +    uint8_t *src[4] = {srcp[0], srcp[1], srcp[2], srcp[3]};
> > +    uint8_t *dst[4] = {dstp[0], dstp[1], dstp[2], dstp[3]};
> > +    // ignore palette for GRAY8
> > +    if (!dst[2]) dst[1] = NULL;
> > +    if (!src[2]) src[1] = NULL;
> 
> a
> if(plane==1 && !dst[2])
>     continue;
> seems less hackish

Agreed, but it is only half the solution, so we would get this:
Index: libswscale/swscale.c
===================================================================
--- libswscale/swscale.c        (revision 29018)
+++ libswscale/swscale.c        (working copy)
@@ -2118,7 +2118,9 @@
         int height= (plane==0 || plane==3) ? srcSliceH: -((-srcSliceH)>>c->chrDstVSubSample);
 
         if (!dst[plane]) continue;
-        if (!src[plane])
+        // ignore palette for GRAY8
+        if (plane == 1 && !dst[2]) continue;
+        if (!src[plane] || (plane == 1 && !src[2]))
             fillPlane(dst[plane], dstStride[plane], length, height, y, (plane==3) ? 255 : 128);
         else
         {