[FFmpeg-devel] [RFC v2 3/3] daaladec: Implement a native Daala decoder

[Ronald S. Bultje]

Hi,

On Tue, Dec 29, 2015 at 11:49 AM, Andreas Cadhalpun <
andreas.cadhalpun at googlemail.com> wrote:

> On 29.12.2015 17:32, Ronald S. Bultje wrote:
> > On Tue, Dec 29, 2015 at 11:29 AM, Andreas Cadhalpun <
> > andreas.cadhalpun at googlemail.com> wrote:
> >> On 29.12.2015 17:15, Ronald S. Bultje wrote:
> >>> In that case, you likely need 64bit integers for 15bit math precision
> >> (look at
> >>> vp9 code), or daala needs to reduce precision (as does hevc).
> >>
> >> Yes, either the intermediate calculation needs to happen with 64bit
> >> integers,
> >> or the input has to be sanitized in some way.
> >
> >
> > It depends what the purpose and source was. Was this real input, or
> fuzzed,
> > or what?
>
> These were fuzzed samples.
>
> > vp9 decoder can certainly overflow with garbage input and that is
> > specifically defined so in libvpx. "Only input generated from a real
> fdct"
> > is considered sane and has a defined outcome.
>
> Do you have a sample causing overflows in the vp9 decoder?


Nope, but I'm going to assume they're not hard to create, just use a few
high same-sign quantized coefficients (e.g. SHORT_MAX) and a high quantizer
(qidx=255, dq_ac=1828). Both dequant into int16_t (1828*SHORT_MAX doesn't
fit in 16 bits) as well as the idct results themselves (because you're
adding and subtracting from an already near-edge value which should fit in
16bits) will overflow.

> (Overflows in dsp code are typically not a security concern.)
>
> Well, the overflows in the imdct calculation of the aac_fixed decoder
> ultimately
> caused crashes.


I would prefer if for video codecs, unless specifically proven otherwise
for that codec with specific content, we assume by default that overflows
in performance-sensitive dsp code (specifically idct) are OK. ubsan may not
like us on fuzzed content, but so be it.

Ronald