[FFmpeg-cvslog] avcodec/dxtory: Fix get_raw_size() for YUV

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Fri Sep 25 17:07:24 2020 +0200| [497debb171b866500ef428479139a112ee7e50d8] | committer: Michael Niedermayer

avcodec/dxtory: Fix get_raw_size() for YUV

Fixes: out of array read
Fixes: 25455/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_DXTORY_fuzzer-6327985731534848

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda at gmail.com>
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=497debb171b866500ef428479139a112ee7e50d8
---

 libavcodec/dxtory.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/dxtory.c b/libavcodec/dxtory.c
index 3f3c23ff2a..157e4b3ed2 100644
--- a/libavcodec/dxtory.c
+++ b/libavcodec/dxtory.c
@@ -44,9 +44,9 @@ static int64_t get_raw_size(enum AVPixelFormat fmt, int width, int height)
     case AV_PIX_FMT_YUV444P:
         return width * height * 3LL;
     case AV_PIX_FMT_YUV420P:
-        return (int64_t)(width * height) + AV_CEIL_RSHIFT(width, 1) * AV_CEIL_RSHIFT(height, 1);
+        return (int64_t)(width * height) + 2 * AV_CEIL_RSHIFT(width, 1) * AV_CEIL_RSHIFT(height, 1);
     case AV_PIX_FMT_YUV410P:
-        return (int64_t)(width * height) + AV_CEIL_RSHIFT(width, 2) * AV_CEIL_RSHIFT(height, 2);
+        return (int64_t)(width * height) + 2 * AV_CEIL_RSHIFT(width, 2) * AV_CEIL_RSHIFT(height, 2);
     }
 
     return 0;