[FFmpeg-cvslog] avcodec/binkdsp: Fix integer overflows in idct
Michael Niedermayer
git at videolan.org
Wed Jun 26 20:12:36 EEST 2019
ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Tue Jun 18 14:28:17 2019 +0200| [7a072fbcc4c6f8ddbf37b131c2d141589118abcd] | committer: Michael Niedermayer
avcodec/binkdsp: Fix integer overflows in idct
Fixes: signed integer overflow: 3784 * 682038 cannot be represented in type 'int'
Fixes: 15265/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_BINK_fuzzer-5088311799971840
Fixes: 15268/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_BINK_fuzzer-5666502344179712
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Reviewed-by: Peter Ross <pross at xvid.org>
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=7a072fbcc4c6f8ddbf37b131c2d141589118abcd
---
libavcodec/binkdsp.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/libavcodec/binkdsp.c b/libavcodec/binkdsp.c
index 9d70e2326f..a357d31672 100644
--- a/libavcodec/binkdsp.c
+++ b/libavcodec/binkdsp.c
@@ -33,20 +33,22 @@
#define A3 3784
#define A4 -5352
+#define MUL(X,Y) ((int)((unsigned)(X) * (Y)) >> 11)
+
#define IDCT_TRANSFORM(dest,s0,s1,s2,s3,s4,s5,s6,s7,d0,d1,d2,d3,d4,d5,d6,d7,munge,src) {\
const int a0 = (src)[s0] + (src)[s4]; \
const int a1 = (src)[s0] - (src)[s4]; \
const int a2 = (src)[s2] + (src)[s6]; \
- const int a3 = (A1*((src)[s2] - (src)[s6])) >> 11; \
+ const int a3 = MUL(A1, (src)[s2] - (src)[s6]); \
const int a4 = (src)[s5] + (src)[s3]; \
const int a5 = (src)[s5] - (src)[s3]; \
const int a6 = (src)[s1] + (src)[s7]; \
const int a7 = (src)[s1] - (src)[s7]; \
const int b0 = a4 + a6; \
- const int b1 = (A3*(a5 + a7)) >> 11; \
- const int b2 = ((A4*a5) >> 11) - b0 + b1; \
- const int b3 = (A1*(a6 - a4) >> 11) - b2; \
- const int b4 = ((A2*a7) >> 11) + b3 - b1; \
+ const int b1 = MUL(A3, a5 + a7); \
+ const int b2 = MUL(A4, a5) - b0 + b1; \
+ const int b3 = MUL(A1, a6 - a4) - b2; \
+ const int b4 = MUL(A2, a7) + b3 - b1; \
(dest)[d0] = munge(a0+a2 +b0); \
(dest)[d1] = munge(a1+a3-a2+b2); \
(dest)[d2] = munge(a1-a3+a2+b3); \
More information about the ffmpeg-cvslog
mailing list