[FFmpeg-devel] 10bit DNxHD vs dct_quantize_c

[Michael Niedermayer]

On Fri, Mar 18, 2011 at 05:24:57PM +0000, Joseph Artsimovich wrote:
[...]
> Second, what's up with faandct()?  It's definitely not equivalent to a textbook DCT transform, as it produces values in the wrong range.  I saw  the FAAN_POSTSCALE definition, but its effect is not documented anywhere and I can't figure out what it does on my own.

IIRC the values are just multiplied by 8
if FAAN_POSTSCALE is set
else its more complex

the idea behind the *8 was to use more of the available bits to improve
precission with 8bit input. unfortunately this pushes 10bit input over the
edge

it should be very easy to compile the ffandct twice and one of that with
less aggressive scaling

    
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Its not that you shouldnt use gotos but rather that you should write
readable code and code with gotos often but not always is less readable
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20110318/86124a9c/attachment-0001.asc>