[FFmpeg-devel] [PATCH] avcodec/mpegvideoenc: fix undefined negative left shift

[Ganesh Ajjanagadde]

This should fix the first undefined behavior reported in:
https://trac.ffmpeg.org/ticket/4727.

I can't reproduce the runtime behavior reported in the ticket, hence I
can't confirm that this actually fixes the exact issue reported in the
ticket.

Regardless, I can confirm that this is a genuine issue, and that
negative shifts can (and do) occur, fixed by this.

Tested with FATE.

Signed-off-by: Ganesh Ajjanagadde <gajjanagadde at gmail.com>
---
 libavcodec/mpegvideo_enc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c
index 8e565c1..a6af8bd 100644
--- a/libavcodec/mpegvideo_enc.c
+++ b/libavcodec/mpegvideo_enc.c
@@ -148,7 +148,7 @@ void ff_convert_matrix(MpegEncContext *s, int (*qmat)[64],
                     qmat16[qscale][0][i] == 128 * 256)
                     qmat16[qscale][0][i] = 128 * 256 - 1;
                 qmat16[qscale][1][i] =
-                    ROUNDED_DIV(bias << (16 - QUANT_BIAS_SHIFT),
+                    ROUNDED_DIV(bias * (1<<(16 - QUANT_BIAS_SHIFT)),
                                 qmat16[qscale][0][i]);
             }
         }
-- 
2.6.1