[FFmpeg-devel] Fix MMX dct_quantize for non zigzag_direct scans

[Michael Niedermayer]

On Tue, May 13, 2008 at 04:39:39AM +0100, Ramiro Polla wrote:
> Hello,
>
> If I use dct_quantize_c for the mimic encoder, all goes well. But if I
> use MMX dct_quantize, it doesn't work the same. The last_non_zero value
> returned is wrong. There is a note about it in
> libavcodec/mpegvideo_enc.c and it searches again for the last_non_zero
> coeff after all calls to dct_quantize if they're not dct_quantize_c.
>
> MMX dct_quantize uses inv_zigzag_direct16 independing of the scantable,
> so that's why (I think) it returns the wrong value. If I change
> ScanTable to also have an inverse[64] table on MMX and use that instead,
> it returns the right value. But MMX dct_quantize uses custom code to
> permute the block in the end, and with this new last_non_zero value
> (from inverse[]) it doesn't copy the blocks correctly. If I add
> non-optimized block permuting code for the cases of alternate scan, it
> works again.
>
> I thought maybe the current code always returned a larger last_non_zero
> value, but it doesn't happen all the time (although it happens most of
> the time).
>
> But then I got confused... Isn't that custom permute code only good for
> ff_zigzag_direct? Shouldn't it copy the wrong values for alternate
> scans? Or are the values being permuted somewhere else again?
>
> Anyways, attached patch gives same resulting file for tempete.cif
> encoded in mpeg4 with -flags +alt.

what happens with other optimized dct quantizers (non x86)
and try the regression tests with forced mmx quant (by default it uses C)
to ensure that the patch does not change the mmx output.

and id like to see benchmarks as well, so we can be sure this doesnt
slow the code down

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Those who are too smart to engage in politics are punished by being
governed by those who are dumber. -- Plato 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20080513/2fd4f5a9/attachment.pgp>