[FFmpeg-devel] [PATCH]lavf/webpenc: Do not set alpha flag for yuv420p

[Carl Eugen Hoyos]

On Monday 02 May 2016 08:57:31 pm wm4 wrote:
> On Mon, 2 May 2016 08:36:16 -0400
>
> "Ronald S. Bultje" <rsbultje at gmail.com> wrote:
> > Hi,
> >
> > On Mon, May 2, 2016 at 5:49 AM, Carl Eugen Hoyos <cehoyos at ag.or.at> wrote:
> > > On Wednesday 13 April 2016 10:27:03 am you wrote:
> > > > Maybe it should just check whether the pixel format is alpha, instead
> > > > of whether it's not yuv420p. (Just a suggestion.)
> > >
> > > Isn't that exactly equivalent to what my patch is doing?
> > > Or do I just misunderstand?
> > >
> > > I just thought that "if (pix_fmt != AV_PIX_FMT_YUV420P)" is shorter
> > > than "if (pix_fmt == AV_PIX_FMT_RGB32 || pix_fmt ==
> > > AV_PIX_FMT_YUVA420P)". Do you
> > > disagree?
> >
> > I think he means:
> >
> > AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(pix_fmt);
> > if (desc->flags & AV_PIX_FMT_FLAG_ALPHA) {
> >     flags |= 16;
> > }
>
> Pretty much - if a better way can't be found.

Sorry, I just couldn't understand your original comment.

New patch attached, Carl Eugen
-------------- next part --------------
From b6748ef99db9bd05c249be6a0f16868701adeccb Mon Sep 17 00:00:00 2001
From: Carl Eugen Hoyos <cehoyos at ag.or.at>
Date: Tue, 3 May 2016 10:16:34 +0200
Subject: [PATCH] lavf/webpenc: Do not set alpha flag for yuv420p.

Fixes one usecase of ticket #5431.
---
 libavformat/webpenc.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/libavformat/webpenc.c b/libavformat/webpenc.c
index 2e0147c..3ac833b 100644
--- a/libavformat/webpenc.c
+++ b/libavformat/webpenc.c
@@ -21,6 +21,7 @@
 
 #include "libavutil/intreadwrite.h"
 #include "libavutil/opt.h"
+#include "libavutil/pixdesc.h"
 #include "avformat.h"
 #include "internal.h"
 
@@ -107,7 +108,9 @@ static int flush(AVFormatContext *s, int trailer, int64_t pts)
         if (w->frame_count == 1) {
             if (!trailer) {
                 vp8x = 1;
-                flags |= 2 + 16;
+                flags |= 2;
+                if (av_pix_fmt_desc_get(st->codecpar->format)->flags && AV_PIX_FMT_FLAG_ALPHA)
+                    flags |= 16; // stream may contain transparency information
             }
 
             if (vp8x) {
-- 
1.7.10.4