[FFmpeg-devel] [PATCH 01/12] avutil/frame: add AVFrame.alpha_mode

[Nicolas George]

Niklas Haas (HE12025-02-20):
> This is true; I am thinking about adding negotiation to this in libavfilter
> down the line as well, for the same reason. I just want to get the basic
> infrastructure in place first.

Ok, but... see later.

> I don't see it as being worse than the status quo of silently doing the wrong
> thing.

The difference is the wrong thing is made official and can happen
anywhere.

> I think that the bigger issue is that some formats and sources can only deal
> with premultiplied alpha. As a side note, something as simple as scaling an
> image should only ever be done on premultiplied alpha - otherwise the
> background fill color may leak into interpolated edge pixels.

This statement is not true: take something that “must” be done in
premuliplied and a normal input, convert the input to premultiplied, do
the thing, convert back, and voilà. You can even factor the conversion
formulas into the thing to have a more complex process that works
directly on normal frames.

The true statement is: some formats and source are more expensive in
non-premultiplied format. Which… sure, but we can live with that, it is
only a matter of compromise.

> I think what bothers me about this approach is that it means we will need to
> duplicate the options about whether the input is premultiplied or not for
> every filter that can handle premul alpha. For example, vf_overlay, vf_scale,
> vf_libplacebo. In the case of the latter, it's not even possible to handle
> cleanly because the filter may have multiple inputs, some of which are premul
> and others which are not.

Just let the user insert the conversion filter.

> IMO cleaner to have only a single frame flag and defer this to vf_setparams.
> If you are concerned about API and memory bloat, a simpler option could be to
> add a frame flag to AVFrame only, although that would prevent users from being
> able to e.g. generate premultiplied JPEG-XL or TIFF files. (Both of those
> formats, as far as I'm aware, suport tagging the alpha type)

I am concerned about API bloat in the sense of adding yet another
fscing thing the application programmer needs to think about. You do not
address that concern.

> I think we learned from YUVJ what a terrible idea this is in the long run.

The YUVJ do not need a different treatment in the code.

> > Last option, the worst one in my opinion: Like you did, but every
> > component must explicitly declare if it supports premultiplied alpha; if
> > a premultiplied frame arrives to a component that does not support them,
> > return an error. The guiding principle is that it is better to fail than
> > to silently propagate wrong output.
> That is the idea here.

If you insist on doing that way, then the guardrails must be there from
the start: the commit that prevents a decoder or a filter from
outputting a premultiplied frame if the receiver is not ready needs to
be the very same commit that makes premultiplied frames official by
adding a flag.

Regards,

-- 
  Nicolas George