[Ffmpeg-devel] New/extended filter API discussion

Luca Abeni lucabe72
Sat Jan 6 11:46:05 CET 2007

Hi Alexander,

On Sat, 2007-01-06 at 03:57 +0300, Alexander Chemeris wrote:
> > also i suggest that whoever designs the filter system looks at mplayers
> > video filters as they support a large number of the things above
> Take a look into Avisynth (and VirtualDub, maybe) filter API. It runs under
> Windows only, but have very interesting filter API with automatic buffer
> management, based on pull model as opposed to push model.
I'll surely have a look soon, thanks for the hint.

> I have some thoughts about video filter API, because I'm thinking about
> very similar subsystem for sipX media processing library.
Well, I do not know when (and if :) libavfilter will be ready, but it
will be designed to be as generic as possible. So maybe we can work
together on it, and you can use it in sipX.

I have some questions and comments, for clarifying my design ideas and
comparing them with other people who are interested in the design of a
filter layer.
> So, when I thought
> about benefits and drawbacks of "push", "pull" and "process" models of
> opperation, I came to the view, that "process" approach is simplest among
> others while almost as powerful as "push" and "pull". By "process" model
> I mean following -- each filter have 'process()' function which simply take
> input frames, process them and push into output buffer. Most work in this
> approach should be done by flowgraph class itself - it knows how filters
> are connected and could take output frames from preceding filter and pass
> them to subsequent filter.
How does this satisfy the "the number of frames consumed by a filter
does not have to match the number output ((inverse)telecine, ...)"

> One drawback of "push" and "pull" models is that they could not have
> multiple inputs (for "push") or outputs (for "pull"). Lets consider
> "push" model.
> If filter graph have two inputs and one of them pushes frame behaviour
> of the second input is ill defined -- it could be ignored, or pulled, or smth
> else, but I do not see good approach here.
I might be naive here, but I do not see the problem.
The filter knows what to do with its inputs (maybe it can generate a
frame as soon as an input frame arrives, or maybe it needs a frame from
each input before generating an output frame, or... It depends on the
filtering algorithm).
Now, let's consider a "Picture-in-Picture" filter Fpip as an example:
let's assume that it has 2 inputs I1 and I2, and it puts an image coming
from I2 (maybe rescaled) in the upper right corner of an image coming
from I1.
When the first input filter pushes a frame to Fpip, it calls
Fpip->push_buffer(Fpip, f, 0). Fpip checks if a frame from I2 is already
available; if yes it produces an output picture, if not it just store
the received frame f and returns.
When the second input filter pushes a frame to Fpip, it calls
Fpip->push_buffer(Fpip, f, 1). Now Fpip knows that a frame from I1 is
already available, and can render an output frame and push it to the
next filter.
The trick is in the third parameter of push_buffer().

I think "pull" filter can work in a similar way.

> And finally I had an idea of universal "push-pull-process" model. Let all
> processing is done in 'process()' function, but instead of simple input
> and output buffers it have two functions - 'get_input()' and 'set_output()'.
> Then if 'get_input()' could take frame from input buffer or ask preceding
> filter for frame, and 'set_output()' behave similar, we could switch from
> "push" to "pull" or "process" model even at runtime.
Since I am interested in supporting both push and pull, I think this is
interesting... I have to think a little bit more about this approach.
Anyway, in my idea the "push" or "pull" behaviour is a global static
property of the filter chain... I did not think about dynamically
switching between push and pull.

> Why this runtime switching is needed? Because each model is native for
> different cases. "Push" is natural when processing frames as they come,
> e.g. when converting from one format to other with maximum speed. "Pull"
> is well-suited when frames are requested by rendering side (as in Avisynth).
If I understand correctly, this is addressed by my proposal about buffer
allocations (see my previous email). Or maybe I am misunderstanding what
you are saying?


More information about the ffmpeg-devel mailing list