[FFmpeg-devel] [PATCH] ac3enc: fix 16-bit signed overflow in mdct

[Justin Ruggles]

On 12/21/2010 10:51 AM, Justin Ruggles wrote:

> On 12/20/2010 07:47 PM, Michael Niedermayer wrote:
> 
>> On Mon, Dec 20, 2010 at 11:40:51PM +0000, Mans Rullgard wrote:
>>
>> Welcome back! :)
>>
>>
>>> One of the cos table values is -32768 and must not be negated.
>>> ---
>>>  libavcodec/ac3enc.c |    2 +-
>>>  1 files changed, 1 insertions(+), 1 deletions(-)
>>>
>>> diff --git a/libavcodec/ac3enc.c b/libavcodec/ac3enc.c
>>> index d372e6a..1191d16 100644
>>> --- a/libavcodec/ac3enc.c
>>> +++ b/libavcodec/ac3enc.c
>>> @@ -405,7 +405,7 @@ static void mdct512(AC3MDCTContext *mdct, int32_t *out, int16_t *in)
>>>      for (i = 0; i < n4; i++) {
>>>          re =  ((int)rot[   2*i] - (int)rot[ n-1-2*i]) >> 1;
>>>          im = -((int)rot[n2+2*i] - (int)rot[n2-1-2*i]) >> 1;
>>> -        CMUL(x[i].re, x[i].im, re, im, -mdct->xcos1[i], mdct->xsin1[i]);
>>> +        CMUL(x[i].re, x[i].im, -re, -im, mdct->xcos1[i], -mdct->xsin1[i]);
>>
>> this contains more negates than before and i suspect the amount can be reduced
>> if it is slower
> 
> 
> I slightly prefer to change the table generation back to clipping -32767
> to 32767.  Anyone against?


I'm going to go ahead and change this back.  There is no quality
difference.  Any speed difference doesn't matter because it is only used
in initialization.

-Justin