[FFmpeg-devel] [PATCH 3/4] avcodec/ppc/fdctdsp: use more accurate constants

[Hendrik Leppkes]

On Mon, Nov 16, 2015 at 6:24 PM, Ganesh Ajjanagadde <gajjanag at mit.edu> wrote:
> On Mon, Nov 16, 2015 at 9:48 AM, Daniel Serpell <dserpell at gmail.com> wrote:
>> Hi!,
>>
>> El Fri, Nov 13, 2015 at 11:42:30AM -0500, Ganesh Ajjanagadde escribio:
>>> Whoever wrote this stuff had a pretty bad libm - digits differ pretty
>>> quickly.
>>
>> They where correctly rounded to 24bit precision. I don't know if that
>> was intentional, so I can't comment on the correctness of the patch.
>
> Unless I am off here, doing things at 24 bits makes very little sense:
> float itself offers 32 bits

Actually floats offer 23-bits of precision for the significand (with 8
for mantissa and one sign bit) - 7-8 digits, so any more precision
than that couldn't be stored accurately anymore, so when using single
precision like this code, more precision is not needed.
Maybe it was even specifically wanted to have values that are
represented accurately, but I do not know the history of this code
either.

- Hendrik