[FFmpeg-devel] [PATCH]lavc/motion_est: Fix undefined negative left shifts

[Carl Eugen Hoyos]

Hi!

Attached patch fixes the following ubsan errors:
libavcodec/motion_est.c:959:42: runtime error: left shift of negative 
value -16
libavcodec/motion_est.c:960:42: runtime error: left shift of negative 
value -16

I do not have a testcase for lines 961f.

Please comment, Carl Eugen
-------------- next part --------------
From 4dd1b8d61c5dd2e77b7c855a1d2acf7d90277daf Mon Sep 17 00:00:00 2001
From: Carl Eugen Hoyos <cehoyos at ag.or.at>
Date: Tue, 18 Apr 2017 11:02:30 +0200
Subject: [PATCH] lavc/motion_est: Fix undefined negative left shifts.

---
 libavcodec/motion_est.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/libavcodec/motion_est.c b/libavcodec/motion_est.c
index 25b606f..316d16a 100644
--- a/libavcodec/motion_est.c
+++ b/libavcodec/motion_est.c
@@ -956,10 +956,10 @@ void ff_estimate_p_frame_motion(MpegEncContext * s,
             P_TOPRIGHT[1] = s->current_picture.motion_val[0][mot_xy - mot_stride + 2][1];
             if (P_TOP[1] > (c->ymax << shift))
                 P_TOP[1] =  c->ymax << shift;
-            if (P_TOPRIGHT[0] < (c->xmin << shift))
-                P_TOPRIGHT[0] =  c->xmin << shift;
-            if (P_TOPRIGHT[1] > (c->ymax << shift))
-                P_TOPRIGHT[1] =  c->ymax << shift;
+            if (P_TOPRIGHT[0] < (c->xmin * (1 << shift)))
+                P_TOPRIGHT[0] =  c->xmin * (1 << shift);
+            if (P_TOPRIGHT[1] > (c->ymax * (1 << shift)))
+                P_TOPRIGHT[1] =  c->ymax * (1 << shift);
 
             P_MEDIAN[0] = mid_pred(P_LEFT[0], P_TOP[0], P_TOPRIGHT[0]);
             P_MEDIAN[1] = mid_pred(P_LEFT[1], P_TOP[1], P_TOPRIGHT[1]);
-- 
1.7.10.4