[FFmpeg-devel] [PATCH] aaccoder: prevent crash of anmr coder

[Andreas Cadhalpun]

On 04.12.2015 23:49, Claudio Freire wrote:
> On Fri, Dec 4, 2015 at 2:23 PM, Andreas Cadhalpun
> <andreas.cadhalpun at googlemail.com> wrote:
>> If minq is negative, the range of sf_idx can be larger than
>> SCALE_MAX_DIFF allows, causing assertion failures later in
>> encode_scale_factors.
>>
>> Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun at googlemail.com>
>> ---
>>  libavcodec/aaccoder.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/libavcodec/aaccoder.c b/libavcodec/aaccoder.c
>> index 2a0cb1f..e8a61ce 100644
>> --- a/libavcodec/aaccoder.c
>> +++ b/libavcodec/aaccoder.c
>> @@ -370,7 +370,7 @@ static void search_for_quantizers_anmr(AVCodecContext *avctx, AACEncContext *s,
>>      }
>>      while (idx) {
>>          sce->sf_idx[bandaddr[idx]] = minq + q0;
>> -        minq = paths[idx][minq].prev;
>> +        minq = FFMAX(paths[idx][minq].prev, 0);
>>          idx--;
>>      }
>>      //set the same quantizers inside window groups
> 
> Actually, a negative .prev signals an impossible path.
> 
> So perhaps the attached patch (well, git diff) would go better? (if
> you confirm it passes fuzzing I'll push it)

That diff doesn't change anything, because in the problematic case
paths[idx][i].cost is always inf, anyway.

Best regards,
Andreas