[FFmpeg-devel] [PATCH 2/2] avcodec/pixlet: Fixes: undefined shift in av_mod_uintp2()

[Michael Niedermayer]

On Fri, Aug 18, 2017 at 06:21:56PM +0200, Paul B Mahol wrote:
> On 8/18/17, Michael Niedermayer <michael at niedermayer.cc> wrote:
> > Fixes: runtime error: shift exponent 4294967289 is too large for 32-bit type
> > 'int'
> > Fixes: 3030/clusterfuzz-testcase-minimized-4649809254285312
> >
> > Found-by: continuous fuzzing process
> > https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> > Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> > ---
> >  libavcodec/pixlet.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/libavcodec/pixlet.c b/libavcodec/pixlet.c
> > index 088226bdda..a9cfe085c9 100644
> > --- a/libavcodec/pixlet.c
> > +++ b/libavcodec/pixlet.c
> > @@ -262,7 +262,7 @@ static int read_high_coeffs(AVCodecContext *avctx,
> > uint8_t *src, int16_t *dst, i
> >
> >          flag = 0;
> >
> > -        if (state * 4ULL > 0xFF || i >= size)
> > +        if ((uint64_t)state > 0xFF / 4 || i >= size)
> 
> This is not exact same behaviour.

no, it differs for a small number of cases that have one of the 2
most significant bits (of the 64 bits) set in state.
I think this was a bug i introduced in a previous commit.

If above change is not the correct thing to do, please elaborate.
state is passed into ff_clz() later which has no deterministic
behavior when these MSBs are set as ff_clz() uses unsigned and state
is int64_t and unsigned can be 32 or 64bit.
So this really just change cases that do not work equally
across platforms

looking at this now, i suspect independant of this patch
the 32 and 24 in
pfx = ((state + 8) >> 5) + (state ? ff_clz(state): 32) - 24;
should be CHAR_BIT * sizeof(unsiged) or something similar
or something else should be changed

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Take away the freedom of one citizen and you will be jailed, take away
the freedom of all citizens and you will be congratulated by your peers
in Parliament.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20170819/4d1d0d36/attachment.sig>