[FFmpeg-devel] [FFmpeg-devel, RFC] lavfi: add opencl tonemap filter.

[Niklas Haas]

Hello Ruiling,

> Thanks for sharing your idea with me. I basically also noticed some poor quality tone mapping result for some hdr stream.
> I will try your suggestion to see whether I can make it in good state so I can include it in next version.
> In fact I have not thought detecting scene change quite well. A question for your idea is:
> is it possible that your scene detection heuristic may still failed to detect some particular scene change and lead to poor tone mapping quality?

The way my scene change detection heuristic works like this: I trigger a
scene change (and therefore discard the frame averaging buffer) if the
distance between the current frame average brightness and the current
running average exceeds a threshold value, that threshold being (by
default) 20 cd/m².

So we can divide the failures of this algorithm into two categories:

1. False negative (scene change without resetting the buffer): This can
   only happen if there was an actual scene change but the average
   brightness change did not exceed 20 cd/m², i.e. the scenes are
   similar in brightness. I consider this a fairly harmful failure
   because that also means there's no visual discontinuity since the
   scenes are so similar to begin with.

2. False positive (buffer got reset without a scene change). This is the
   more worrying failure of the algorithm, since it can happen in the
   middle of a scene (e.g. as the result of a bright flash of light
   on-screen), which will manifest itself as a sudden decrease in the
   total frame brightness coinciding with the new source of light. (Or
   vice versa, a sudden increase in brightness coinciding with the
   sudden absence of a light source).

The scene change threshold is a trade-off. Lowering the value decreases
the likelihood of #1 but increases the likelihood of #2. Increasing the
value decreases the likelihood of #2, but increases the likelihood (and
apparent effect) of #1.

If you want to optimize or improve the algorithm, the case #2 is the one
I would be most interested in, i.e. reducing the rate of false
positives. This can surely be done in a smarter way, e.g. by comparing
more than just the scene average but also other easily obtained metrics.

If you have access to low-level frame information, you could do
something like increasing the threshold for non-keyframes significantly,
since keyframes are likely to coincide with scene changes or cuts this
might help the algorithm out.

> I just copy this piece of code from vf_tonemap.c. I think we need to fix it there first if this is wrong.
> I guess the code was like this because we think that all video stream that truly use ST. 2084 should
> Include the mastering display metadata, if it is absent, then transfer function should be arib-std-b67.

Yeah, good point. And ideally, maybe those two should share this logic
to avoid code duplication.

> Basically I am targeting full gpu pipeline transcoding(gpu decoding + gpu filtering + gpu encoding),
> Most video streams I have encountered are YUV stream.
> Could you show me what kind of use-case that need RGB support?
> So I will try to see whether adding it in this patch or do it later.

One particular use case I have in mind that would be good to support is
OpenEXR formats, which are typically RGB and high depth or floating point.

> In fact, I have not learned about HLG deeply. I still need some time to add HLG support.

Well the one key thing that's interesting about HLG is that the OOTF
(which forms part of the EOTF) is parametrized by the peak brightness of
the display. So when tone mapping (which is done in display-referred
space) a HLG signal to a lower brightness, the inverse OOTF you need to
apply to go back to HLG afterwards needs to be done on the new peak
brightness, to reflect the changes made to the signal. As said I'm not
sure which component handles the OOTF in FFmpeg land, so if you
implement it yourself (which may be a necessary) that would be a thing
to keep in mind.

> Well, this is very useful reference for me. I need some time to digest it.
> Thanks a lot.

No problem,
Niklas