[FFmpeg-cvslog] avformat/rpl: Use 64bit in bitrate computation and check it

[Michael Niedermayer]

ffmpeg | branch: release/4.4 | Michael Niedermayer <michael at niedermayer.cc> | Mon Apr 26 22:43:51 2021 +0200| [5664263561ecd8e2781432bdf99322c7fa195b00] | committer: Michael Niedermayer

avformat/rpl: Use 64bit in bitrate computation and check it

Fixes: signed integer overflow: 777777776 * 4 cannot be represented in type 'int'
Fixes: 29102/clusterfuzz-testcase-minimized-ffmpeg_dem_RPL_fuzzer-6726188921913344

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit 29b244ffc15abe2c24d2145f63048e8b3bdaa303)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=5664263561ecd8e2781432bdf99322c7fa195b00
---

 libavformat/rpl.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/libavformat/rpl.c b/libavformat/rpl.c
index 0f00c03a52..940794606c 100644
--- a/libavformat/rpl.c
+++ b/libavformat/rpl.c
@@ -207,8 +207,10 @@ static int rpl_read_header(AVFormatContext *s)
             ast->codecpar->bits_per_coded_sample = 4;
 
         ast->codecpar->bit_rate = ast->codecpar->sample_rate *
-                                  ast->codecpar->bits_per_coded_sample *
-                                  ast->codecpar->channels;
+                                  (int64_t)ast->codecpar->channels;
+        if (ast->codecpar->bit_rate > INT64_MAX / ast->codecpar->bits_per_coded_sample)
+            return AVERROR_INVALIDDATA;
+        ast->codecpar->bit_rate *= ast->codecpar->bits_per_coded_sample;
 
         ast->codecpar->codec_id = AV_CODEC_ID_NONE;
         switch (audio_format) {