[FFmpeg-cvslog] avcodec/vp3: Error out on invalid num_coeffs in unpack_vlcs()

[Michael Niedermayer]

ffmpeg | branch: release/2.8 | Michael Niedermayer <michael at niedermayer.cc> | Sun Feb 11 03:38:54 2018 +0100| [89668fa8430886823b476c13cb46e766f3cf7547] | committer: Michael Niedermayer

avcodec/vp3: Error out on invalid num_coeffs in unpack_vlcs()

This fixes a hypothetical integer overflow

Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit f2318aee8ca8df1c84092f7d6691a2d0df02c474)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=89668fa8430886823b476c13cb46e766f3cf7547
---

 libavcodec/vp3.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/libavcodec/vp3.c b/libavcodec/vp3.c
index 61e659c2b6..bf56d942e5 100644
--- a/libavcodec/vp3.c
+++ b/libavcodec/vp3.c
@@ -951,9 +951,11 @@ static int unpack_vlcs(Vp3DecodeContext *s, GetBitContext *gb,
     Vp3Fragment *all_fragments = s->all_fragments;
     VLC_TYPE(*vlc_table)[2] = table->table;
 
-    if (num_coeffs < 0)
+    if (num_coeffs < 0) {
         av_log(s->avctx, AV_LOG_ERROR,
                "Invalid number of coefficents at level %d\n", coeff_index);
+        return AVERROR_INVALIDDATA;
+    }
 
     if (eob_run > num_coeffs) {
         coeff_i      =