[FFmpeg-cvslog] avcodec/hevc_mp4toannexb_bsf: check that nalu size doesnt overflow

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Fri Dec 13 00:50:21 2019 +0100| [a8ceb2a72fa1bef4ab5f1ec6cdc7ce74fffda19d] | committer: Michael Niedermayer

avcodec/hevc_mp4toannexb_bsf: check that nalu size doesnt overflow

Fixes: Out of array access
Fixes: 19299/clusterfuzz-testcase-minimized-ffmpeg_BSF_HEVC_MP4TOANNEXB_fuzzer-5169193398042624

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=a8ceb2a72fa1bef4ab5f1ec6cdc7ce74fffda19d
---

 libavcodec/hevc_mp4toannexb_bsf.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/libavcodec/hevc_mp4toannexb_bsf.c b/libavcodec/hevc_mp4toannexb_bsf.c
index d0f1b94f0e..baa93628ed 100644
--- a/libavcodec/hevc_mp4toannexb_bsf.c
+++ b/libavcodec/hevc_mp4toannexb_bsf.c
@@ -152,8 +152,7 @@ static int hevc_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
         extra_size    = add_extradata * ctx->par_out->extradata_size;
         got_irap     |= is_irap;
 
-        if (SIZE_MAX - nalu_size < 4 ||
-            SIZE_MAX - 4 - nalu_size < extra_size) {
+        if (FFMIN(INT_MAX, SIZE_MAX) < 4ULL + nalu_size + extra_size) {
             ret = AVERROR_INVALIDDATA;
             goto fail;
         }