[Ffmpeg-devel] New/extended filter API discussion
Sat Jan 6 11:46:05 CET 2007
On Sat, 2007-01-06 at 03:57 +0300, Alexander Chemeris wrote:
> > also i suggest that whoever designs the filter system looks at mplayers
> > video filters as they support a large number of the things above
> Take a look into Avisynth (and VirtualDub, maybe) filter API. It runs under
> Windows only, but have very interesting filter API with automatic buffer
> management, based on pull model as opposed to push model.
I'll surely have a look soon, thanks for the hint.
> I have some thoughts about video filter API, because I'm thinking about
> very similar subsystem for sipX media processing library.
Well, I do not know when (and if :) libavfilter will be ready, but it
will be designed to be as generic as possible. So maybe we can work
together on it, and you can use it in sipX.
I have some questions and comments, for clarifying my design ideas and
comparing them with other people who are interested in the design of a
> So, when I thought
> about benefits and drawbacks of "push", "pull" and "process" models of
> opperation, I came to the view, that "process" approach is simplest among
> others while almost as powerful as "push" and "pull". By "process" model
> I mean following -- each filter have 'process()' function which simply take
> input frames, process them and push into output buffer. Most work in this
> approach should be done by flowgraph class itself - it knows how filters
> are connected and could take output frames from preceding filter and pass
> them to subsequent filter.
How does this satisfy the "the number of frames consumed by a filter
does not have to match the number output ((inverse)telecine, ...)"
> One drawback of "push" and "pull" models is that they could not have
> multiple inputs (for "push") or outputs (for "pull"). Lets consider
> "push" model.
> If filter graph have two inputs and one of them pushes frame behaviour
> of the second input is ill defined -- it could be ignored, or pulled, or smth
> else, but I do not see good approach here.
I might be naive here, but I do not see the problem.
The filter knows what to do with its inputs (maybe it can generate a
frame as soon as an input frame arrives, or maybe it needs a frame from
each input before generating an output frame, or... It depends on the
Now, let's consider a "Picture-in-Picture" filter Fpip as an example:
let's assume that it has 2 inputs I1 and I2, and it puts an image coming
from I2 (maybe rescaled) in the upper right corner of an image coming
When the first input filter pushes a frame to Fpip, it calls
Fpip->push_buffer(Fpip, f, 0). Fpip checks if a frame from I2 is already
available; if yes it produces an output picture, if not it just store
the received frame f and returns.
When the second input filter pushes a frame to Fpip, it calls
Fpip->push_buffer(Fpip, f, 1). Now Fpip knows that a frame from I1 is
already available, and can render an output frame and push it to the
The trick is in the third parameter of push_buffer().
I think "pull" filter can work in a similar way.
> And finally I had an idea of universal "push-pull-process" model. Let all
> processing is done in 'process()' function, but instead of simple input
> and output buffers it have two functions - 'get_input()' and 'set_output()'.
> Then if 'get_input()' could take frame from input buffer or ask preceding
> filter for frame, and 'set_output()' behave similar, we could switch from
> "push" to "pull" or "process" model even at runtime.
Since I am interested in supporting both push and pull, I think this is
interesting... I have to think a little bit more about this approach.
Anyway, in my idea the "push" or "pull" behaviour is a global static
property of the filter chain... I did not think about dynamically
switching between push and pull.
> Why this runtime switching is needed? Because each model is native for
> different cases. "Push" is natural when processing frames as they come,
> e.g. when converting from one format to other with maximum speed. "Pull"
> is well-suited when frames are requested by rendering side (as in Avisynth).
If I understand correctly, this is addressed by my proposal about buffer
allocations (see my previous email). Or maybe I am misunderstanding what
you are saying?
More information about the ffmpeg-devel