[FFmpeg-cvslog] avformat/rmdec: Make expected_len 64bit

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Tue Oct 20 21:32:59 2020 +0200| [728330462cadb765307cc132377b6b5d177a225c] | committer: Michael Niedermayer

avformat/rmdec: Make expected_len 64bit

Fixes: signed integer overflow: 1347551268 * 14 cannot be represented in type 'int'
Fixes: 26458/clusterfuzz-testcase-minimized-ffmpeg_dem_RM_fuzzer-5655364324032512

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=728330462cadb765307cc132377b6b5d177a225c
---

 libavformat/rmdec.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/libavformat/rmdec.c b/libavformat/rmdec.c
index 03ddf29226..3606b74a6e 100644
--- a/libavformat/rmdec.c
+++ b/libavformat/rmdec.c
@@ -703,17 +703,19 @@ static int rm_sync(AVFormatContext *s, int64_t *timestamp, int *flags, int *stre
             state= (state<<8) + avio_r8(pb);
 
             if(state == MKBETAG('I', 'N', 'D', 'X')){
-                int n_pkts, expected_len;
+                int n_pkts;
+                int64_t expected_len;
                 len = avio_rb32(pb);
                 avio_skip(pb, 2);
                 n_pkts = avio_rb32(pb);
-                expected_len = 20 + n_pkts * 14;
-                if (len == 20)
+                expected_len = 20 + n_pkts * 14LL;
+
+                if (len == 20 && expected_len <= INT_MAX)
                     /* some files don't add index entries to chunk size... */
                     len = expected_len;
                 else if (len != expected_len)
                     av_log(s, AV_LOG_WARNING,
-                           "Index size %d (%d pkts) is wrong, should be %d.\n",
+                           "Index size %d (%d pkts) is wrong, should be %"PRId64".\n",
                            len, n_pkts, expected_len);
                 len -= 14; // we already read part of the index header
                 if(len<0)