[FFmpeg-cvslog] avcodec/agm: Fix integer overflow with w/h

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Fri Apr  5 00:20:33 2019 +0200| [2169a3f262beedcb06fdb0e0e5277a680570df13] | committer: Michael Niedermayer

avcodec/agm: Fix integer overflow with w/h

Fixes: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
Fixes: 13999/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_AGM_fuzzer-5644405991538688

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda at gmail.com>
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=2169a3f262beedcb06fdb0e0e5277a680570df13
---

 libavcodec/agm.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/libavcodec/agm.c b/libavcodec/agm.c
index 2d2092222d..cbd45e8095 100644
--- a/libavcodec/agm.c
+++ b/libavcodec/agm.c
@@ -535,11 +535,13 @@ static int decode_frame(AVCodecContext *avctx, void *data,
 
     s->flags = 0;
     w = bytestream2_get_le32(gbyte);
+    h = bytestream2_get_le32(gbyte);
+    if (w == INT32_MIN || h == INT32_MIN)
+        return AVERROR_INVALIDDATA;
     if (w < 0) {
         w = -w;
         s->flags |= 2;
     }
-    h = bytestream2_get_le32(gbyte);
     if (h < 0) {
         h = -h;
         s->flags |= 1;