[FFmpeg-cvslog] avcodec/mpeg4videodec: Clear bits_per_raw_sample if it has originated from a previous instance

[Michael Niedermayer]

ffmpeg | branch: release/4.0 | Michael Niedermayer <michael at niedermayer.cc> | Sat Jun  9 22:25:38 2018 +0200| [897524954be2eba39af71fbb9210236b337270a6] | committer: Michael Niedermayer

avcodec/mpeg4videodec: Clear bits_per_raw_sample if it has originated from a previous instance

Fixes: assertion failure
Fixes: ffmpeg_crash_5.avi

Found-by: Thuan Pham <thuanpv at comp.nus.edu.sg>, Marcel Böhme, Andrew Santosa and Alexandru Razvan Caciulescu with AFLSmart
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit 2fc108f60f98cd00813418a8754a46476b404a3c)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=897524954be2eba39af71fbb9210236b337270a6
---

 libavcodec/mpeg4videodec.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/libavcodec/mpeg4videodec.c b/libavcodec/mpeg4videodec.c
index 51b58fac02..d0ebaac6e8 100644
--- a/libavcodec/mpeg4videodec.c
+++ b/libavcodec/mpeg4videodec.c
@@ -3109,6 +3109,12 @@ int ff_mpeg4_decode_picture_header(Mpeg4DecContext *ctx, GetBitContext *gb)
     /* search next start code */
     align_get_bits(gb);
 
+    // If we have not switched to studio profile than we also did not switch bps
+    // that means something else (like a previous instance) outside set bps which
+    // would be inconsistant with the currect state, thus reset it
+    if (!s->studio_profile && s->avctx->bits_per_raw_sample != 8)
+        s->avctx->bits_per_raw_sample = 0;
+
     if (s->codec_tag == AV_RL32("WV1F") && show_bits(gb, 24) == 0x575630) {
         skip_bits(gb, 24);
         if (get_bits(gb, 8) == 0xF0)