[FFmpeg-devel] [PATCH] MMX VP3 Loop Filter

[David Conrad]

On Oct 6, 2008, at 3:42 AM, Jason Garrett-Glaser wrote:

> On Mon, Oct 6, 2008 at 12:35 AM, David Conrad <lessen42 at gmail.com>  
> wrote:
>> On Oct 4, 2008, at 2:18 PM, Michael Niedermayer wrote:
>>> I'm not sure if this can be done with only 8-bit math;  
>>> filter_limit can
>> theoretically be 127 which pushes CONST0/1 to +- 254, and mm2/mm3  
>> can range
>> from -127 to 128.
>
> Are you sure filter_limit can be that large?  I know of no sane
> deblocker that allows such high limit values.

Quoting the Theora spec,

LFLIMS - Integer array - 7 bits - Not signed

This procedure decodes the table of loop ?lter limit values used to  
drive the
loop ?lter, which is described in Section 6.4.1. It is decoded as  
follows:

1. Read a 3-bit unsigned integer as NBITS.
2. For each consecutive value of qi from 0 to 63, inclusive:
(a) Read an NBITS-bit unsigned integer as LFLIMS[qi ].

I can't find anything in the spec that limits it any further.

> Even if it can, you can still do it in byte math; look at x264's
> deblocking code for reference.  H.264 requires 9 or 10 bits of
> precision for various parts of deblocking, and x264 gets away with
> using byte math by emulating the extra bits separately.  This is still
> faster than unpacking and doing 16-bit math.

I'll have a look; at the very least I guess I could only do 16-bit  
math in the unlikely extreme cases.