[FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

[Timo Rothenpieler]

On 10.08.2022 23:43, Andreas Rheinhardt wrote:
> Timo Rothenpieler:
>> On 10.08.2022 23:24, Andreas Rheinhardt wrote:
>>> Timo Rothenpieler:
>>>> IEEE-754 differentiates two different kind of NaNs.
>>>> Quiet and Signaling ones. They are differentiated by the MSB of the
>>>> mantissa.
>>>>
>>>> For whatever reason, actual hardware conversion of half to single always
>>>> sets the signaling bit to 1 if the mantissa is != 0, and to 0 if it's 0.
>>>> So our code has to follow suite or fate-testing hardware float16 will be
>>>> impossible.
>>>
>>> What does the exr spec say about quiet and signaling nans?
>>
>> Not sure how exr would be involved here.
> 
> Your patch changes the output of an exr-test. The output of the exr
> decoder is presumably determined by the exr spec. There is after all the
> possibility that what hardware does in hardware and what this patch does
> in software is incompatible with what exr specifies.

The exr spec just says something along the lines of analogous to 
ieee-754 floats: 
https://openexr.readthedocs.io/en/latest/TechnicalIntroduction.html?highlight=ieee#the-half-data-type
It barely ever mentions NaNs, other than that they exist. Which makes 
sense, given they don't typically appear in images.

The only output changed is that for how NaNs are converted.
And given the cross-validation with multiple hardware implementations, 
I'm confident that it's correct.