[FFmpeg-devel] SDR->HDR tone mapping algorithm?

Guo, Yejun yejun.guo at intel.com
Wed Feb 13 09:59:43 EET 2019

> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf
> Of Niklas Haas
> Sent: Saturday, February 09, 2019 1:30 AM
> To: Kps, Harish Krupo <harish.krupo.kps at intel.com>
> Cc: vittorio.giovara at gmail.com; Sharma, Shashank
> <shashank.sharma at intel.com>; Palli, Tapani <tapani.palli at intel.com>;
> FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>;
> Strasser, Kevin <kevin.strasser at intel.com>
> Subject: Re: [FFmpeg-devel] SDR->HDR tone mapping algorithm?
> Hi,
> The important thing to consider is what constraints we are trying to
> solve. And I think the expected behavior is that an SDR signal in SDR
> mode should look identical to an SDR signal in HDR mode, to the end
> user.
> This is, of course, an impossible constraint to solve, since we don't
> know anything about the display, either in HDR or in SDR mode. At best,
> in the absence of this knowledge, we could make a guess (e.g. it's
> roughly described by sRGB in SDR mode, and for HDR mode it roughly
> follows the techniques outlined in ITU-R Report BT.2390). Better yet
> would be to actually obtain this information from somewhere, but where?
> (The user? ICC profile? EDID?).

just for your information. There might be a way (in future) to get this information
via libdrm (at least intel platform), see https://lists.freedesktop.org/archives/dri-devel/2017-May/143135.html.

> But the bottom line is that to solve the "make SDR in HDR mode appear
> identical to SDR in SDR mode" constraint, the curve you are trying to
> invert is not your own tone mapping operator, but the tone mapping
> operator implemented by the display (in HDR mode), which definitely
> depends on what brightness level the display is targeting (in both SDR
> and HDR modes).
> For an ideal HDR display, this would simply be the PQ curve's exact
> definition (i.e. noop tone mapping). But in practice, the display will
> almost surely not be capable of displaying up to 10,000 nits, so it will
> implement a tone mapping operator of some kind (even if it's as simple
> as clipping the extra range). Some colorimetric/reference displays
> actually do the latter, since they prefer clipping out-of-range signals
> over distorting in-range ones. But most consumer displays will probably
> do something similar to the hable curve, most likely in per-channel
> modes.
> For an ideal SDR display, it depends on who you ask (w.r.t what "ideal"
> means). In the ITU-R world, an ideal SDR reference display implements
> the BT.1886 transfer function. In practice, it's probably closer to a
> pure power gamma 2.2 curve. Or maybe sRGB. We really have nothing else
> to do here except either consult an ICC profile or just stick our head
> in the sand and guess randomly.
> --------------------------------------------------------------------------
> I'd also like to comment on your compositor design proposal. A few notes:
> 1. It's always beneficial to do as few color conversion steps as
>    possible, to minimize cumulative errors and optimize performance. If
>    you use a 3DLUT as any step (e.g. for implementing an ICC-profile
>    based mapping), the 3DLUT should be as "wide" as possible and cover
>    as many operations as possible, so that the 3DLUT can be end-to-end
>    optimized (by the CMM).
>    If you insist on doing compositing in linear light, then I would
>    probably composite in display-referred linear light and convert it to
>    non-linear light during scanout (either by implementing the needed
>    OETF + linear tone mapping operator via the VCGTs, or by doing a
>    non-linear tone mapping pass). But I would recommend trying to avoid
>    any second gamut conversion step (e.g. from BT.2020 to the display's
>    space after compositing).
>    Otherwise, I would composite directly in the target color space
>    (saving us one final conversion step), which would obviously be
>    preferable if there are no transparency effects to worry about.
>    Maybe we could even switch dynamically between the two depending on
>    whether any blending needs to occur? Assuming we can update the VCGTs
>    atomically and without meaningful latency.
> 2. Rec 2020 is not (inherently) HDR. Also, the choice of color gamut has
>    nothing to do with the choice of transfer function. I might have Rec
>    709 HDR content. In general, when ingesting a buffer, the user should
>    be responsible for tagging both its color primaries and its transfer
>    function.
> 3. If you're compositing in linear light, then you most likely want to
>    be using at least 16-bit per channel floating point buffers, with 1.0
>    mapping to "SDR white", and HDR values being treated as above 1.0.
>    This is also a good color space to use for ingesting buffers, since
>    it allows treating SDR and HDR inputs "identically", but extreme
>    caution must be applied due to the fact that with floating point
>    buffers, we're left at the mercy of what the client wants to put into
>    them (10^20? NaN? Negative values?). Extra metadata must still be
>    communicated between the client and the compositor to ensure both
>    sides agree on the signal range of the floating point buffer
>    contents.
> 4. Applications need a way to bypass the color pipeline in the
>    compositor, i.e. applications need a way to tag their buffers as
>    "this buffer is in display N's native (SDR|HDR) color space". This of
>    course only makes sense if applications both have a way of knowing
>    what display N's native SDR/HDR color space is, as well as which
>    display N they're being displayed (more) on. Such buffers should be
>    preserved as much as possible end-to-end, ideally being just directly
>    scanned out as-is.
> 5. Implementing a "good" HDR-to-SDR tone mapping operator; and even the
>    question of whether to use the display's HDR or SDR mode, requires
>    knowledge of what brightness range your composited buffer contains.
>    Crucially, I think applications should be allowed to tag their
>    buffers with the brightest value that they "can" contain. If they
>    fail to do so, we should assume the highest possible value permitted
>    by the transfer function specified (e.g. 10,000 nits for PQ). Putting
>    this metadata into the protocol early would allow us to explore
>    better tone mapping functions later on.
> Some final words of advice,
> 1. The protocol suggestions for color management in Wayland have all
>    seemed terribly over-engineered compared to the problem they are
>    trying to solve. I have had some short discussions with Link Mauve on
>    the topic of how to design a protocol that's as simple as possible
>    while still fulfilling its purpose, and we started drafting our own
>    protocol for this, but it's sitting in a WIP state somewhere.
> 2. I see that Graeme Gill has posted a bit in at least some of these
>    threads. I recommend listening to his advice as much as possible.
> On Fri, 08 Feb 2019 22:01:49 +0530, Harish Krupo
> <harish.krupo.kps at intel.com> wrote:
> > Hi Vittorio,
> >
> > Vittorio Giovara <vittorio.giovara at gmail.com> writes:
> >
> > > On Fri, Feb 8, 2019 at 3:22 AM Harish Krupo <harish.krupo.kps at intel.com>
> > > wrote:
> > >
> > >> Hello,
> > >>
> > >> We are in the process of implementing HDR rendering support in the
> > >> Weston display compositor [1] (HDR discussion here [2]). When HDR
> > >> and SDR surfaces like a video buffer and a subtitle buffer are presented
> > >> together, the composition would take place as follows:
> > >> - If the display does not support HDR metadata:
> > >>   in-coming HDR surfaces would be tone mapped using opengl to SDR
> and
> > >>   blended with the other SDR surfaces. We are currently using the Hable
> > >>   operator for tone mapping.
> > >> - If the display supports setting HDR metadata:
> > >>   SDR surfaces would be tone mapped to HDR and blended with HDR
> surfaces.
> > >>
> > >> The literature available for SDR->HDR tone mapping varies from simple
> > >> linear expansion of luminance to CNN based approaches. We wanted to
> know
> > >> your recommendations for an acceptable algorithm for SDR->HDR tone
> mapping.
> > >>
> > >> Any help is greatly appreciated!
> > >>
> > >> [1] https://gitlab.freedesktop.org/wayland/weston
> > >> [2]
> > >> https://lists.freedesktop.org/archives/wayland-devel/2019-
> January/039809.html
> > >>
> > >> Thank you
> > >> Regards
> > >> Harish Krupo
> > >>
> > >
> > > In *theory* the tonemapping functions should be reversible, so if you
> use
> > > vf_tonemap or vf_tonemap_opencl and properly expand the range via
> zimg
> > > (vf_zscale) before compression it should work fine. However I have
> never
> > > tried it myself, so I cannot guarantee that those filters will work as is.
> > > Of course haasn from the libplacebo project might have better
> suggestions,
> > > so you should really reach out to him.
> >
> > Thanks, will try reversing the algorithms. Sure, will contact Haasn.
> >
> > Regards
> > Harish Krupo
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

More information about the ffmpeg-devel mailing list