[FFmpeg-devel] [PATCH 2/3] swscale/output: Fix integer overflow in yuv2gbrp_full_X_c()

[Michael Niedermayer]

Fixes: signed integer overflow: 1966895953 + 210305024 cannot be represented in type 'int'
Fixes: 391921975/clusterfuzz-testcase-minimized-ffmpeg_SWS_fuzzer-5916798905548800

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
---
 libswscale/output.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/libswscale/output.c b/libswscale/output.c
index 21c3bdc3076..084727edfe0 100644
--- a/libswscale/output.c
+++ b/libswscale/output.c
@@ -2316,9 +2316,9 @@ yuv2gbrp_full_X_c(SwsInternal *c, const int16_t *lumFilter,
         Y -= c->yuv2rgb_y_offset;
         Y *= c->yuv2rgb_y_coeff;
         Y += 1 << (SH-1);
-        R = Y + V * c->yuv2rgb_v2r_coeff;
-        G = Y + V * c->yuv2rgb_v2g_coeff + U * c->yuv2rgb_u2g_coeff;
-        B = Y +                            U * c->yuv2rgb_u2b_coeff;
+        R = Y + V * (unsigned)c->yuv2rgb_v2r_coeff;
+        G = Y + V * (unsigned)c->yuv2rgb_v2g_coeff + U * (unsigned)c->yuv2rgb_u2g_coeff;
+        B = Y +                                      U * (unsigned)c->yuv2rgb_u2b_coeff;
 
         if ((R | G | B) & 0xC0000000) {
             R = av_clip_uintp2(R, 30);
-- 
2.48.1