[FFmpeg-devel] [PATCH 3/5] ffmpeg: flush and drain video filters.

[Nicolas George]

Le quartidi 24 ventôse, an CCXX, Michael Niedermayer a écrit :
> This sounds much better indeed

TGood, thanks.

> many
> 
> 1. one could show the video a few frames ago in a smaller picture in
> the big one as an effect like if someone pointed a camera onto a
> monitor displaying the video from the camera
> 
> 2. one could with several simple filters construct a denoise or
> temporal average (IIR) filter, with audio it could be a echo filter.
> 
> 3. if one made a movie one could for example refer in the movie to a
> recording of some earlier part of the movie by having that displayed
> via a picture in picture filter or as a flashback (this maybe isnt
> the most sane way to implement that though)
> 
> 4. iam sure there are various effects one could do with it ...

[ TL;DR: if filters are deterministic, my proposal with EAGAIN and no FIFOs
can handle loops. OTOH, either with poll_frame or EAGAIN, loops will lead to
infinite recursion, and that needs to be detected. ]


I am not sure I can see exactly how it can be done with a loop and not just
a cycle, but let us assume it works.

I believe practical examples will always have an input in the loop: a filter
with two inputs, one coming from the loop and one coming from outside.
Something like that:

    L1   +----+       ,........       +----+  L4
    ---->|    |  L2  :         :  L3  |    |----->
         | F1 |----->: filters :----->| F2 |
      ,->|    |      '.........'      |    |-.
      |  +----+                       +----+ |
      |              L5                      |
      `--------------------------------------'

Then, unless I am mistaken, there is no need to add a FIFO inside the loop
to prevent it from producing an infinite number of frames: the availability
of frames on L1 will regulate the loop.

To make sure of that, I need to state another principle, I hope you will
find it acceptable:

# When a filter has several inputs, its output must not depend on the order
# of arrival of frames on different inputs.

In other words, both these sequences:

push(in[0], buf1); push(in[1], buf2); push(in[0], buf3); push(in[1], buf4);

push(in[0], buf1); push(in[1], buf2); push(in[1], buf4); push(in[0], buf3);

should give the exact same output.

I can grant two exceptions to this principle: First, if a filter has an
internal queue that threatens to overflow, it can decide to takes measures.
Second, if a filter relies on a PRNG, the order of arrival may affect the
exact sequence produced by the PRNG, that is not too big a problem.

Rationale for this principle: the order of arrival on different inputs is an
implementation detail that can change at any time. If we were to try to run
filters in separate threads with message queues for links, the order of
arrival on different inputs would become mostly random.


This principle has a corollary: a filter with several inputs, under normal
circumstances, will eventually stop producing output if one of its inputs
stays empty.

It may take a lot of time: if the filters uses timestamps and one input is
1 hour ahead of the other, it may output one hour worth of frames before
blocking; if the other input has non-monotonic timestamps, it may take an
infinite number of frames, but that is not normal circumstances.


With that corollary, we can be sure that the output of the loop I have
drawn, at L4, will be bounded by (a function of) its input, at L1. No need
to insert a FIFO to block the stream. (Of course, F1 itself needs an
internal FIFO, but that is not a problem, because that is the same code that
gets packets and knows what to do with them.)


That being said, we currently can not handle loops, and my solution does not
address that. The problem is that, for most filters, poll_frame on an output
pad is mostly implemented as a function of poll_frames on all its outputs.

For example, on the loop I drawn, assuming no filter has any buffered frame
(at the very start for example):

poll(L4) = poll(L3)
         = poll(L2) (possibly ×2 or /2 or whatever)
         = min(poll(L1), poll(L5)) (or some other condition)
         = min(poll(L1), poll(L3))

And we have an infinite recursion. The same applies to request_frame, with
or without EAGAIN.

Unless I am mistaken, it can be solved relatively easily by adding a flag to
links to test whether a request_frame is currently pending. If a filter
tries to issue a request_frame on a link where the flag is already set,
avfilter_request_frame returns an error immediately (I would have suggested
ELOOP, but this is not windows-friendly, and the standard message speaks of
symlinks; let us settle for a new AVERROR_LOOP). Therefore, we now get:

request(L4) = request(L3)
            = request(L2)
            = request(L1) && request(L5) (or some other operation)
            = request(L1) && request(L3)
            = request(L1) && AVERROR_LOOP

Regards,

-- 
  Nicolas George
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20120314/ee142718/attachment.asc>