[FFmpeg-cvslog] avcodec/flacdsp: Avoid undefined operations in non debug builds

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Thu Dec 15 01:32:18 2016 +0100| [831274fba4b14c542458ce5a3d0135b49499299a] | committer: Michael Niedermayer

avcodec/flacdsp: Avoid undefined operations in non debug builds

This fixes ubsan warnings in non debug builds by using unsigned operations

in debug builds the correct signed operations are retained so that overflows
(which should not occur in valid files and may indicate problems in the DSP code
or decoder) can be detected.

Alternatively they can be changed to unsigned unconditionally, then its
not possible though to detect overflows easily if someone wants to test
the DSP code for overflows.

The 2nd alternative would be to leave the code as it is and accept that
there are undefined operations in the DSP code and that ubsan output is
full of them in some cases.

Similar changes would be needed in some other DSP routines

Suggested-by: Matt Wolenetz <wolenetz at google.com>
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=831274fba4b14c542458ce5a3d0135b49499299a
---

 libavcodec/flacdsp.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/libavcodec/flacdsp.c b/libavcodec/flacdsp.c
index 30b6648..180d4e7 100644
--- a/libavcodec/flacdsp.c
+++ b/libavcodec/flacdsp.c
@@ -43,14 +43,22 @@
 #define PLANAR 1
 #include "flacdsp_template.c"
 
+// For debuging we use signed operations so overflows can be detected (by ubsan)
+// For production we use unsigned so there are no undefined operations
+#ifdef DEBUG
+#define SUINT   int
+#else
+#define SUINT   unsigned
+#endif
+
 static void flac_lpc_16_c(int32_t *decoded, const int coeffs[32],
                           int pred_order, int qlevel, int len)
 {
     int i, j;
 
     for (i = pred_order; i < len - 1; i += 2, decoded += 2) {
-        int c = coeffs[0];
-        int d = decoded[0];
+        SUINT c = coeffs[0];
+        SUINT d = decoded[0];
         int s0 = 0, s1 = 0;
         for (j = 1; j < pred_order; j++) {
             s0 += c*d;
@@ -66,7 +74,7 @@ static void flac_lpc_16_c(int32_t *decoded, const int coeffs[32],
     if (i < len) {
         int sum = 0;
         for (j = 0; j < pred_order; j++)
-            sum += coeffs[j] * decoded[j];
+            sum += coeffs[j] * (SUINT)decoded[j];
         decoded[j] += sum >> qlevel;
     }
 }