[FFmpeg-devel] [PATCH]Fix pnm encoding with bpc set

[Carl Eugen Hoyos]

Christophe Gisquet <christophe.gisquet <at> gmail.com> writes:

> > -        if( avctx->bits_per_raw_sample )
> > +        if (   avctx->bits_per_raw_sample
> > +            && 
> > av_pix_fmt_desc_get(avctx->pix_fmt)->comp[0].depth_minus1 > 7)
> >              maxdepth = (1 << avctx->bits_per_raw_sample) - 1;
> 
> So bits_per_raw_sample can be != 0 but kind of wrong in 
> some 8 bits cases? Do you have an example for this?

No, I was just too lazy to implement 8bit, partly 
because I don't know how to test, partly because I 
wasn't entirely happy about your patch (being applied).

> +        int bps_delta =
> av_pix_fmt_desc_get(avctx->pix_fmt)->comp[0].depth_minus1 + 1 -
> avctx->bits_per_raw_sample;
> +        if (!avctx->bits_per_raw_sample || !bps_delta) {
> 
> Could this be moved out the loop (the compiler should do it, but...) ?

That should be done, yes.

> Second point, can this get negative? I don't think so 
> because it means something would be seriously wrong.

Yes, but it doesn't look like a security issue to me.

> I think "[PATCH 2/5] pnmenc: use bits_per_raw_sample" 
> is wrong then

Definitely, I still wonder how you tested it.

> or there's maybe an endianess issue somewhere.

I don't think so, at least I don't see any indication.

Carl Eugen