[FFmpeg-cvslog] avcodec/iff: finetune the palette size check in the mask case

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Sat Jun 22 21:17:52 2019 +0200| [0f9789c8e37eb6d166729e876729beb21b7d5647] | committer: Michael Niedermayer

avcodec/iff: finetune the palette size check in the mask case

Fixes: out of array access
Fixes: 15381/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_IFF_ILBM_fuzzer-5668057826983936

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Peter Ross <pross at xvid.org>
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=0f9789c8e37eb6d166729e876729beb21b7d5647
---

 libavcodec/iff.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/iff.c b/libavcodec/iff.c
index b43bd507b3..4099d118e4 100644
--- a/libavcodec/iff.c
+++ b/libavcodec/iff.c
@@ -1514,7 +1514,7 @@ static int decode_frame(AVCodecContext *avctx,
     buf_size -= bytestream2_tell(gb);
     desc = av_pix_fmt_desc_get(avctx->pix_fmt);
 
-    if (!s->init && avctx->bits_per_coded_sample <= 8 &&
+    if (!s->init && avctx->bits_per_coded_sample <= 8 - (s->masking == MASK_HAS_MASK) &&
         avctx->pix_fmt == AV_PIX_FMT_PAL8) {
         if ((res = cmap_read_palette(avctx, (uint32_t *)frame->data[1])) < 0)
             return res;