[FFmpeg-devel] [PATCH] lavfi/buffersrc: push the frame deeper if requested.

[Paul B Mahol]

On 7/11/17, Nicolas George <george at nsup.org> wrote:
> Le primidi 21 messidor, an CCXXV, Marton Balint a ecrit :
>> I am not saying they aren't, I'm just showing a case where your revert is
>> breaking another case which was working before. Nicolas also stated there
>> can be cases where the old code (before the revert) worked better. I just
>> hope that I found a simple enough case, so somebody can take a look and
>> find
>> the root cause.
>
> Thanks for stepping in. I will try to dot the `i's in this issue.
>
> From the start, libavfilter has had subtle issues in corner cases. These
> issue reside primarily with filters with several inputs and the handling
> of EOF. We all tried to get rid of them, but soon realized they are like
> lamination bubbles: once you think you are rid of one, another pops up a
> few centimeters away, the only solution is to redo almost the whole
> thing.
>
> I have been doing just that for libavfilter. And since doing it all at
> once is not an option, I have been doing it step by step. The
> consequence is that there is a large stitch between the new and the old
> code, that may be as annoying as the bubbles, and even more so when the
> stitch approaches areas where there are bubbles.
>
> Dropping the metaphor: I have been reworking the scheduling of
> libavfilter to get filter with multiple inputs and EOF working, but
> look, I don't mean to be rude, but this is not as easy as it looks.
>
> Since changing all the filters is not an option, I had to include a
> compatibility wrapper to have the old implementations work in the new
> design. This compatibility wrapper is obviously not perfect. In fact, it
> probably cannot be perfect, because if it could, we would not need it in
> the first place.
>
> It took time, but the new design is there, and as far as I know, it
> works. Filters with several inputs are activated when needed, EOF is
> propagated with its timestamp (but ffmpeg does not inject it because the
> patch series implementing it was blocked by bikeshedding). The only
> thing that does not work is the existing filters. That makes quite a big
> "only", I grant you.
>
> Now, I realize how frustrating to have a feature in a filter that used
> to work and no longer does, especially if it is a feature that you
> implemented yourself lovingly, and I sympathize.
>
> My goal, obviously, is to have everything working. But it takes time,
> especially when alone.
>
> Also, there have been unexpected stuff in my personal life that have
> left me with less time than before to work on ffmpeg.
>
> Now, if a feature has been broken and you want it back, you can have two
> options: you can revert, or you can help move forward.
>
> If you revert, you will possibly fix the feature, but you will also
> bring back all the issues that were fixed by these commits. Plus
> possibly other issues due to code changes that expect the new behaviour.
> And thus, you will have to revert and revert and revert.
>
> Also, you make more work for me, delaying by that much the time where
> the work is done.
>
> If you want to help move forward, on the other hand... IMO, it is the
> only sane option.
>
> Now, I also realize that helping move forward is easier said than done,
> even more so since I have not documented the new design. Documentation
> takes time, documenting in details something that does not work yet is a
> waste of time. Still, there is already quite a lot in the doxygen
> comments, and if somebody wants to help, I will give any information
> needed.
>
> And I will do so for the issue at hand.
>
> The problem is that with the "shortest" option, filters that use the
> "dualinput" system do not report EOF correctly.
>
> I have not yet been able to determine if the problem is in the dualinput
> system, not returning an error when it should, or in the wrapper code,
> not taking the return code into account properly.
>
> Still, I am not sure understanding that and fixing it would be a good
> use of the time, because it may not be possible, and because the filters
> with several inputs need to be rewritten with the new framework in mind:
> only then can they reap the benefits, especially getting rid of the
> "buffer queue overflow" problems.

Nothing can get rid of that issue, except tootal lavfi rewrite.

>
> So, how to write a filter like that with the new framework?
>
> There is a single callback, called "activate". It must, in succession:
>
> - Examine the input links (using:
>
> http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20170615/4490ac33/attachment.patch
>   ):
>
>   - If a link has no queued frames but a status, use
>     ff_inlink_acknowledge_status().
>
>   - If the queued frames (and if relevant status) allow to produce a
>     frame on output, do it and return.
>
>   - If the input status change allow to deduce that one output or
>     several are finished, report it with
>     ff_avfilter_link_set_in_status() and return.
>
> - Examine the output links (using a wrapper, to plan for future
>   evolutions such as mutexes)
>
>   - If frame_wanted_out is set on an output, find out which inputs would
>     require a frame to allow to produce one on that output and request
>     them using ff_inlink_request_frame().
>
> - If nothing could be done, just return FFERROR_NOT_READY (it will need
>   to be moved into filters.h).
>
> I think doing that work for the dualinput / framesync filters is not a
> very complex work, but I am afraid it will require similar changes to
> all filters using it.

I tried this and it did not work at all. And I even contacted Nicolas
privately and never got reply.

>
>
> (A side note about lamination: if what you want to laminate is
> water-resistant, adding a layer of soapy water will help: it will not
> damage the adhesive but slow it down, and the bubbles, filled with water
> instead of air, are much easier to lead to the edges. But do not do that
> to apply a screen protector on a capacitive touchscreen, they do not
> like to get wet!)
>
> Regards,
>
> --
>   Nicolas George
>