[FFmpeg-cvslog] avcodec/rscc: Check pixel_size for overflow

[Michael Niedermayer]

ffmpeg | branch: release/3.3 | Michael Niedermayer <michael at niedermayer.cc> | Sat May 13 01:31:19 2017 +0200| [d9b25b392330e8039fffbd918285c7ae917a057e] | committer: Michael Niedermayer

avcodec/rscc: Check pixel_size for overflow

Fixes: 1509/clusterfuzz-testcase-minimized-5129419876204544

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit 934572c5c3592732a30336afdf2df9926a8b4df2)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=d9b25b392330e8039fffbd918285c7ae917a057e
---

 libavcodec/rscc.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/libavcodec/rscc.c b/libavcodec/rscc.c
index ebacd3f62c..f270cd5351 100644
--- a/libavcodec/rscc.c
+++ b/libavcodec/rscc.c
@@ -211,6 +211,12 @@ static int rscc_decode_frame(AVCodecContext *avctx, void *data,
         ctx->tiles[i].y = bytestream2_get_le16(gbc);
         ctx->tiles[i].h = bytestream2_get_le16(gbc);
 
+        if (pixel_size + ctx->tiles[i].w * (int64_t)ctx->tiles[i].h * ctx->component_size > INT_MAX) {
+            av_log(avctx, AV_LOG_ERROR, "Invalid tile dimensions\n");
+            ret = AVERROR_INVALIDDATA;
+            goto end;
+        }
+
         pixel_size += ctx->tiles[i].w * ctx->tiles[i].h * ctx->component_size;
 
         ff_dlog(avctx, "tile %d orig(%d,%d) %dx%d.\n", i,