[FFmpeg-devel] [PATCH 2/2] swscale/output: Bias 16bps output calculations to improve non overflowing range for GBRP16/GBRPF32

[Andreas Rheinhardt]

Michael Niedermayer:
> On Wed, Nov 02, 2022 at 10:02:39PM +0100, Michael Niedermayer wrote:
>> Fixes: integer overflow
>> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
>> ---
>>  libswscale/output.c       | 25 +++++++++++--------------
>>  libswscale/x86/output.asm | 16 +++++++++++++++-
>>  2 files changed, 26 insertions(+), 15 deletions(-)
> 
> Note, these corner case overflows could affect some of the similary
> implemented cases that are not depth 16 too
> 
> ill fix them if issues are replicated
> 

The checkasm-sw_gbrp runs into many overflows (when run under UBSan);
e.g. fate.ffmpeg.org tells me of an issue in line 2289. Said line is not
touched in your commits.

- Andreas