[FFmpeg-devel] [RFC v2 3/3] daaladec: Implement a native Daala decoder

[Andreas Cadhalpun]

On 29.12.2015 16:58, Ronald S. Bultje wrote:
> Hi,
> 
> On Tue, Dec 29, 2015 at 10:55 AM, Andreas Cadhalpun <
> andreas.cadhalpun at googlemail.com> wrote:
> 
>>> +static av_always_inline void idct_1D_8(pixel *x, int xstride, const
>> pixel y[16])
>>> +{
>>> +    int t1h, t4h, t6h, t0 = y[0], t1 = y[1], t2 = y[2], t3 = y[3], t4 =
>> y[4];
>>> +    int t5 = y[5], t6 = y[6], t7 = y[7];
>>> +    t5 -= (t3*2485 + 4096) >> 13;
>>> +    t3 += (t5*18205 + 16384) >> 15;
>>> +    t5 -= (t3*2485 + 4096) >> 13;
>>> +    t7 -= (t1*3227 + 16384) >> 15;
>>> +    t1 += (t7*6393 + 16384) >> 15;
>>> +    t7 -= (t1*3227 + 16384) >> 15;
>>> +    t1 += t3;
>>
>> These seven lines can overflow.
> 
> 
> Why do you believe they can overflow?

Because ubsan told me that.

> Look at range constraints for the input values.

Apparently that is not constrained enough, e.g.:
t3 = -1449866
t3*2485 = -3602917010 < INT32_MIN

Best regards,
Andreas