[FFmpeg-cvslog] avcodec/rangecoder: Test for invalid corner case

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Mon May  8 02:58:54 2017 +0200| [5f928c5201c077b9765610bc5304235c3f1d9bd6] | committer: Michael Niedermayer

avcodec/rangecoder: Test for invalid corner case

Fixes runtime error: left shift of 1912602815 by 8 places cannot be represented in type 'int'
Fixes: 1403/clusterfuzz-testcase-minimized-4724820484816896

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=5f928c5201c077b9765610bc5304235c3f1d9bd6
---

 libavcodec/rangecoder.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/libavcodec/rangecoder.c b/libavcodec/rangecoder.c
index 9c6ef75b69..d7a2774d27 100644
--- a/libavcodec/rangecoder.c
+++ b/libavcodec/rangecoder.c
@@ -58,6 +58,10 @@ av_cold void ff_init_range_decoder(RangeCoder *c, const uint8_t *buf,
 
     c->low         = AV_RB16(c->bytestream);
     c->bytestream += 2;
+    if (c->low >= 0xFF00) {
+        c->low = 0xFF00;
+        c->bytestream_end = c->bytestream + 2;
+    }
 }
 
 void ff_build_rac_states(RangeCoder *c, int factor, int max_p)