[FFmpeg-cvslog] avcodec/cfhd: Fix runtime error: signed integer overflow: 65280 * 65288 cannot be represented in type 'int'

[Michael Niedermayer]

ffmpeg | branch: release/3.3 | Michael Niedermayer <michael at niedermayer.cc> | Thu Jun  1 18:32:52 2017 +0200| [4f02447d45681bf7e99f184bc97ee8d68284ffc5] | committer: Michael Niedermayer

avcodec/cfhd: Fix runtime error: signed integer overflow: 65280 * 65288 cannot be represented in type 'int'

Fixes: 1925/clusterfuzz-testcase-minimized-5564569688735744

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit cd6f319a7470394044627d1bd900e21b9aca5f4a)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=4f02447d45681bf7e99f184bc97ee8d68284ffc5
---

 libavcodec/cfhd.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/libavcodec/cfhd.c b/libavcodec/cfhd.c
index ef97b547ab..57a8d2b76e 100644
--- a/libavcodec/cfhd.c
+++ b/libavcodec/cfhd.c
@@ -501,7 +501,7 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame,
             int highpass_a_width = s->plane[s->channel_num].band[s->level][s->subband_num].a_width;
             int highpass_a_height = s->plane[s->channel_num].band[s->level][s->subband_num].a_height;
             int highpass_stride = s->plane[s->channel_num].band[s->level][s->subband_num].stride;
-            int expected = highpass_height * highpass_stride;
+            int expected;
             int a_expected = highpass_a_height * highpass_a_width;
             int level, run, coeff;
             int count = 0, bytes;
@@ -512,11 +512,12 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame,
                 goto end;
             }
 
-            if (highpass_height > highpass_a_height || highpass_width > highpass_a_width || a_expected < expected) {
+            if (highpass_height > highpass_a_height || highpass_width > highpass_a_width || a_expected < highpass_height * (uint64_t)highpass_stride) {
                 av_log(avctx, AV_LOG_ERROR, "Too many highpass coefficients\n");
                 ret = AVERROR(EINVAL);
                 goto end;
             }
+            expected = highpass_height * highpass_stride;
 
             av_log(avctx, AV_LOG_DEBUG, "Start subband coeffs plane %i level %i codebook %i expected %i\n", s->channel_num, s->level, s->codebook, expected);