[FFmpeg-devel] [PATCH] avdevice/avdevice: Deprecate AVDevice Capabilities API

[Mark Thompson]

On 25/01/2021 23:20, Nicolas George wrote:
> Mark Thompson (12021-01-25):
>> Merging the libraries (in source form, orthogonal to merging the
>> binaries) only makes sense if we are going to continue using the
>> libavformat internals, and that is exactly the thing we are trying to
>> get rid of.
> 
> "We"?
> 
> It seems to me that many people here have an agenda about libavdevice
> while they have no reason to.

A number of the people in the project have expressed this opinion, hence I believe it is something which "we" want to do.  ("we" may also not want to do it; "we" do not need to be consistent.)

>> If we replace the libavdevice API with a proper one oriented towards
>> devices rather than being hacked on to the libavformat API then we
>> actively don't want that merger.
>>
>> So, what do people actually want here?  The situation as I see it is
>> (please correct me on anything you feel is in error here):
>>
>> * The features which libavdevice offers are useful to a lot of people.
> 
> Thank you for acknowledging it.
> 
>> * Many of those are using it via the ffmpeg utility, but not all.
>> * The libavdevice API is the libavformat API because it was originally
>> split out from libavformat, and it has the nice property that devices
>> and files end up being interchangable in some contexts.
>> * The libavdevice API, being the libavformat API for files, is not
>> particularly well-suited in other contexts, because devices may not
>> have the same properties as files.
>> * Some odd things like the completely-unused capabilities API and the
>> almost-never-used message API are hacked on top of that to try to
>> avoid some libavformat issues, but are not actually useful to anyone
>> (hence the lack of use).
>> * To implement devices as AVInputFormat/AVOutputFormat instances,
>> libavdevice currently needs access to the internals of libavformat.
>> * Many developers want to get rid of that dependency on libavformat
>> internals, because it creates a corresponding ugliness on the
>> libavformat side which has to leave those parts exposed in an
>> ABI-constrained way.
>>
>> What might we do about this:
>>
>> * Delete libavdevice, with a statement that its capabilities are
>> outside the scope of ffmpeg.  Obviously this isn't sensible, the
>> ffmpeg utility would then need to depend on some external library to
>> offer the same features it does now.
>> * Merge libavdevice into the ffmpeg utility.  Since it is the primary
>> user this would neatly sidestep the previous objection, but now
>> external users are in the same boat so again unacceptable.
> 
> One of the main reasons the fork failed while original ffmpeg kept going
> was that while ffmpeg has a policy of being useful, the fork had an
> unspoken rule about code purity that lead to removing many useful
> features. IIRC, Anton was one of the most vocal proponent of this
> harmful unspoken rule.
> 
> I find worrisome to see the same attitude start to appear here. We need
> to be very careful.
> 
> Also, if the project actually starts removing features for purity, like
> some people discussed here, I'll stop bothering about this dying
> project.
> 
>> * Merge libavdevice into libavformat.  This works to eliminate the
>> dependency problem, but it only really makes sense if we accept that
>> libavdevice is finished and will never want new API which isn't
>> repurposed libavformat.
>> * Make a new libavdevice-specific API and switch to that.  The
>> eliminates the dependency problem, but there is a nasty transition
>> period and it would break the interchangability of the two libraries.
>> It also would be the most work, but leave libavdevice in the best
>> state for doing more things with in future.
> 
> One of the important features of libavdevice, as you mentioned earlier,
> is that devices can be a drop-in replacement for demuxers and muxers. An
> application that is not designed for devices can be made to use them,
> and that is immensely convenient. It does not work always, but it work
> often enough to be, I repeat, immensely useful.

I'm afraid I disagree with this sentiment.  Each library should be designed around an API which makes sense for the abstraction it is trying to create.  Forcing one library into the ill-suited API of another to work makes it less useful, not more.  (OpenMAX is a less useful API than libavcodec/libavfilter, despite covering more cases.)

> If a solution does not have this feature, we can just skip it.

Nothing here excludes the possibility of making a libavdevice psuedo-format which covers the device use-cases possible within the libavformat API (compare the lavfi pseudo-demuxer in the ffmpeg utility).

>> * Nothing.  People will continue to be annoyed about the problems above.
> 
> Duh. People often are annoyed at the wrong things, let them be annoyed.
> I have had proof in the past that some of the annoyance on this subject
> was actually a deep misunderstanding of how linking works.
> 
> There is another option:
> 
> * Stop imposing ourselves constraints that bring absolutely no benefit.
> Having internals is useful for a library because we can change them
> without bothering about compatibility: if we change the internals, we
> can change all places that use them at the same time, and everything
> keeps working.
> 
> It is a good hygiene that the fftools are written to use no internals,
> because that proves the public API is complete enough to build powerful
> tools.
> 
> But for the libraries, it makes no sense to have this constraint. What
> benefit is there in it? Nobody has been able to answer this question,
> because there are none. Libraries should be able to use each other
> internals (typical example: sharing header decoding and format-specific
> utility functions between lavc and lavf).

Even after such a merge, the libavdevice API is still a problem.

>> Opinions?
>>
>> Personally, I think the right option here is to make a new
>> libavdevice-specific API and cut the connection to libavformat
>> entirely, and I would be prepared to do some work towards that goal.
>> Exactly what form it would take is unclear, but I would be happy to
>> talk about that further if people agree that it is the right approach.
> 
> One of the very important things you missed:
> 
> A lot of devices would also be very useful as libavfilter sources and
> sinks. It allows automatic format negotiation, direct rendering and many
> other benefits.

Direct rendering with devices is not possible without a proper libavdevice API.  Making an internal special case allowing it in libavfilter only does not seem like the most useful step to take in this direction to me.

> But it cannot work for now, because the API to run a filter graph only
> works with buffer sinks, no other kinds of sinks.

I agree that this change makes sense for the use-cases you describe, but I'm unsure why it has to be related (though I suspect I may be missing some subtlety).

I'm also unclear why it would need to mess with libavdevice internals - a pseudo-filter which acts as a source is already possible if you want to write one.

> Therefore, before considering enhancing the API of libavdevice, we need
> a proper API tu run a filter graph.

Why is this a dependency?  libavdevice is useful in a context with no libavfilter at all.

> And since we want inter-filters parallelism, a feature that has been
> wanted for a long time, for that we need a proper event loop and
> parallel task scheduler. Which brings us back to this mail, on which I
> intend to follow-up soon:
> https://ffmpeg.org/pipermail/ffmpeg-devel/2020-December/274172.html

I agree that careful movements towards an event loop are a good idea, but again not sure why it has to be related.

> Yes, libavdevice's API is far from perfect, but it is part of a much
> wider problem: everything in FFmpeg that is not directly codecs and
> formats, and secondarily filters, has been let to slowly rot, because
> "we don't need this", "it's not our core mission", "it's unusual",
> whatever.

Or lack of developer interest, which in the playback case which prompted this discussion originally is I think driven by "other projects do it better, so why should we include it here".

> Continue like that, and FFmpeg will become irrelevant. Or change
> completely, embrace originality, create the necessary infrastructure,
> and FFmpeg can be again a leader in efficient audio-video programming.

- Mark