[FFmpeg-devel] [RFC] Documenting RV40 Loop Filter

[Kostya]

I post this in hope to clarify details of RV40 loop filter, since I don't
understand all details of its work and hope to determine correct comments
and variable names for my loop filter patch.

Please comment. I will try to find to provide an additional information if
possible.

==== Loop filter invoking as it's done in decoder ====

Filtering is performed on the edges of 4x4 subblocks of 16x16 macroblock.
Decoder calculates horizontal and vertical deblock patterns for luma and
chroma separately from a cartload of values (see at the end). If there is
a bit set in the pattern for some macroblock, then its edges (horizontal
or vertical ones depending on the selected pattern) should be filtered.

Coded block pattern for luma:

y0 y1 y2 y3
y4 y5 y6 y7
y8 y9 yA yB -> yF yE yD yC yB yA y9 y8 y7 y6 y5 y3 y3 y2 y1 y0
yC yD yE yF    (yF - MSB, y0 - LSB)

Coded pattern for chroma:

c0 c1
c2 c3 -> c3 c2 c1 c0

There are two modes of operation - ordinary mode and edge mode, strong filtering
may be performed only in the edge mode (this name comes from the fact it is
set only for edges of macroblock and not internal subblock edges).

Different filter parameters (ordinary/edge mode, limits) are calculated using
macroblock filtering strength which is 2 for intra-coded MBs and macroblock
with separately coded DCs, 1 otherwise.

Filter uses clipping parameters - they are zero for non-coded subblocks and
take value of ClipTable[filtering strength][quantiser]. Both clipping value
for current subblock and its neightbour are used.

Filtering order:
1. If horizontal deblock pattern shows that bottom subblock should be filtered,
   then current subblock block edge is filteres as ordinary edge
2. If vertical deblock pattern shows this subblock should be deblocked and
   its left edge should be filtered in ordinary mode, it is filtered.
3. For the top row of subblocks top edge may be filtered if this subblock is
   marked for filtering and edge mode is present.
4. For left column of subblocks, if left edge should be filtered in edge mode.

== Weak loop filter ==

This is a rather simple filter filtering pixels close to the edge first.
(BTW, how to name them? p_n and q_n as H.264 standard does?)
So if we have
  p4 p3 p2 p1 p0 | q0 q1 q2 q3 q4
pair (p0, q0) is filtered first and pair (p2, q2) is the last filtered one.

Filtering proecess consists of calculating pixel difference, if the difference
is not too big (its absolute value is less than Beta for (p1,q1) and (p2,q2);
for (p0,q0) difference value is checked against Alpha parameter) then clip
that difference to the range [-limit; +limit] and remove that difference from
this pixel pair (TODO: good names for limits for all pairs).

== Strong loop filter ==

In strong filtering mode elements are replaced with values obtained from
FIR filter with coefficients {25/128, 26/128, 26/128, 26/128, 25/128}
(i.e. averaging pixel with two neighbours on each side). Filtered value
may be clipped in order to not differ from original value by more than some
limit value.

For luma blocks pair (p3, q3) is also filtered but with filter
{26/128, 51/128, 26/128, 26/128} where coefficient 51/128 is for original
value and other values are for p1, p2, p4 or q1, q2, q4

=== Deblock patterns calculation (aka very black magic) ===

Horizontal deblock pattern for luma is made from:
 CBP for the current MB
 coded block pattern made for the MB shifted one row of subblocks up
 (i.e. its first row is really a bottom row of top neighbour MB, second row is
  the first row of current MB, etc.)
 top row CBP for the bottom neighbour is appended at the end
 motion vectors pattern, i.e. when subblock lies on the edge of 8x8 block with
  significant MV difference from its neighbour, then this subblock is also
  marked in deblock pattern
 some other values (maybe for B-frames)

Vertical deblock pattern for luma is made from:
 CBP for the current MB
 CBP for the MB shifted left by one subblock column (i.e. its left column is
  really right column of the left neighbour MB)
 motion vector pattern
 some other B-frame specific stuff (I hope)

Chroma deblock patterns do not have motion vectors pattern and smaller.