[FFmpeg-devel] [PATCH 01/12] avutil/frame: add AVFrame.alpha_mode

[Niklas Haas]

On Thu, 20 Feb 2025 10:39:56 +0100 Nicolas George <george at nsup.org> wrote:
> Niklas Haas (HE12025-02-19):
> > FFmpeg currently handles alpha in a quasi-arbitrary way. Some filters/codecs
> > assume alpha is premultiplied, others assume it is independent. If there is
> > to be any hope for order in this chaos, we need to start by defining an enum
> > for the possible range of values.
>
> Please, not like that.
>
> To start with: in libavfilter, if a filter expects full range and
> receives premultiplied, then automatic conversion needs to happen. That
> means integrating the new flag into the negotiation process. And given
> how fragile and uncovered by FATE the negotiation process is, good luck
> with that without breaking anything.¹

This is true; I am thinking about adding negotiation to this in libavfilter
down the line as well, for the same reason. I just want to get the basic
infrastructure in place first.

Most likely, full negotiation will have to wait until after my swscale rewrite,
which will be able to easily handle alpha (un)premultiplication for any format.

> But more importantly: “We have a bug that makes your output subtly wrong
> in some corner cases. To fix that, we introduce this new flag. You have
> to take it into account because otherwise any of your output might be
> wrong.” This is a terrible API change for applications.

I don't see it as being worse than the status quo of silently doing the wrong
thing.

> IIRC, we use full range alpha everywhere except in a few specialized
> filters. I remember thinking adding these specialized filters was a
> terrible idea at the time. It is possible that more such cases have been
> added while I was not paying attention.

I think that the bigger issue is that some formats and sources can only deal
with premultiplied alpha. As a side note, something as simple as scaling an
image should only ever be done on premultiplied alpha - otherwise the
background fill color may leak into interpolated edge pixels.

> So my main suggestion is to keep it that way: decide that FFmpeg uses
> full range alpha, period. Make it clear in the documentation that the
> few filters that use premultiplied are a specialized case for experts
> only, with the responsibility of checking the format and converting
> resting squarely on the shoulders of these expert users.

I think what bothers me about this approach is that it means we will need to
duplicate the options about whether the input is premultiplied or not for
every filter that can handle premul alpha. For example, vf_overlay, vf_scale,
vf_libplacebo. In the case of the latter, it's not even possible to handle
cleanly because the filter may have multiple inputs, some of which are premul
and others which are not.

IMO cleaner to have only a single frame flag and defer this to vf_setparams.
If you are concerned about API and memory bloat, a simpler option could be to
add a frame flag to AVFrame only, although that would prevent users from being
able to e.g. generate premultiplied JPEG-XL or TIFF files. (Both of those
formats, as far as I'm aware, suport tagging the alpha type)

As a last point, I think that "X is a special case for experts only" is too
broad an argument. By that logic, we might well remove or degrade half of
FFmpeg, starting with support for interlaced formats.

What you are proposing ultimately seems, to me, to be more of a hack than a
proper solution.

> Another option would be to treat premultiplied alpha as different pixels
> formats: we have YUVA420P for full range alpha, add YUVM420P for
> premultiplied alpha.

I think we learned from YUVJ what a terrible idea this is in the long run.
You are unnecessarily duplicating all pixel formats. It also doesn't match
precedents - we don't have separate image formats for linear light vs gamma
light formats, for example, even though they affect the behavior of filters
about as much as multiplied vs straight alpha does.

> Last option, the worst one in my opinion: Like you did, but every
> component must explicitly declare if it supports premultiplied alpha; if
> a premultiplied frame arrives to a component that does not support them,
> return an error. The guiding principle is that it is better to fail than
> to silently propagate wrong output.

That is the idea here.

>
> 1: On the other hand, if you have energy to spare, adding FATE coverage
> to the libavfilter negotiation process would be immensely useful. You
> can see an attempt there:
> https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2022-August/299593.html
> It requires a detailed graph output option, which you are working on, so
> that is good.
>
> The process to develop these tests is: find a line in the code that
> looks like it needs coverage; understand what it does; conceive a filter
> graph where it has a consequence and add it as a test; disable that line
> and check the test fails.
>
> Regards,
>
> --
>   Nicolas George
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".