[FFmpeg-devel] [PATCH v2 1/4] avcodec/aom_film_grain: add AOM film grain synthesis

[Mark Thompson]

On 08/03/2024 13:44, Niklas Haas wrote:
> On Fri, 08 Mar 2024 10:31:28 -0300 James Almer <jamrial at gmail.com> wrote:
>> On 3/8/2024 10:21 AM, Niklas Haas wrote:
>>> From: Niklas Haas <git at haasn.dev>
>>>
>>> Implementation copied wholesale from dav1d, sans SIMD, under permissive
>>> license. This implementation was extensively verified to be bit-exact,
>>> so it serves as a much better starting point than trying to re-engineer
>>> this from scratch for no reason. (I also authored the original
>>> implementation in dav1d, so any "clean room" implementation would end up
>>> looking much the same, anyway)
>>>
>>> The notable changes I had to make while adapting this from the dav1d
>>> code-base to the FFmpeg codebase include:
>>>
>>> - reordering variable declarations to avoid triggering warnings
>>> - replacing several inline helpers by avutil equivalents
>>> - changing code that accesses frame metadata
>>> - replacing raw plane copying logic by av_image_copy_plane
>>>
>>> Apart from this, the implementation is basically unmodified.
>>
>> Do we want this to be public? Both as a struct and the decoding functions.
>> It could be used by libavfilter or even outside our libraries. The hevc
>> decoder would export the relevant T.35 SEI in the new struct if told to
>> not apply fg, like we already do in av1.
> 
> I'm not sure if the AFGS1 struct itself needs to be public, since it is
> largely just a codec-internal wrapper for multiple param sets (for
> scalable codecs).

This is not correct.  Along with scalable cases, the multiple param sets are to support applying film grain at the display resolution after scaling, providing a better result than upscaling the grain applied at the decode resolution.

For example, you could have a scalable stream with operating points of 1920x1080, 1280x720 and 640x360.  The AFGS1 metadata associated with the stream would then have film grain parameters for those three resolutions, plus perhaps 2560x1440 and 3840x2160.

In the ideal case you then pick the operating point for decode based on your available bandwidth and decode capabilities, and the resolution for film grain application based on the display.  The decode happens without any film grain, the clean video is upscaled, and then the film grain is applied immediately before display.

A current conforming AV1 implementation which only supports applying film grain as part of the decode process can do so and produce a conforming result, but the quality may not be as good as the ideal case because the presence of noise will affect the upscale quality and also the grain itself will be scaled in a way which may not look correct.

I'm not sure what the best way to expose this is.  For a player application an option to select the intended display resolution and then export an AV1 film grain side data as it is now is sufficient, but that doesn't really work in an application like ffmpeg where the target resolution isn't directly known.

(Also note that a transcode can carry the AFGS1 message from the source to the output without ever touching it, as long as the target resolution satisfies the requirement on the coded resolution being available in the param sets.  It seems desirable to support this possibility.)

Thanks,

- Mark