[FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Niklas Haas]

On Fri, 13 Oct 2023 21:19:34 +0200 Michael Niedermayer <michael at niedermayer.cc> wrote:
> Hi everyone
> 
> I propose using 15k$ from SPI for funding sws cleanup work.
> this is substantially less than what people belive this needs (see IRC logs from yesterday or so)
> So it really is more a small price for a good deed and not proper payment.
> This of course is only available to competent developers. (exact rules or how thats determined
> would still need to be decided unless its a clear case)
> Also the exact outcome and goal would need to be discussed by the community and whoever
> does the work.
> But some goals would probably be to make sws
> * pleasent to work with
> * similar speed or faster
> * proper multithreading
> * proper full colorspace convertion not ignoring gamma, primaries, ...
> * clean / understandable modular design (maybe everything can be a "Filter" inside sws
>   that get build into a chain)
> 
> Proper payment (50k$ maybe) would be too much in relation to what SPI has ATM (150k$)
> 
> Above all, this is just my oppinion, the actual SPI funding also would need to
> be approved by the community. This can happen after a specific volunteer comes forth
> or before, whichever way the community prefers.
> 
> thx

My gut instinct is that the correct path forwards is to first create a
new API which can internally dispatch to swscale, libplacebo or zimg
based on what's compiled/available/preferred (e.g. using GPU filters for
hwframes, CPU filters for swframes, zimg (or vf_colorspace's primitives)
for colorspace conversions ...).

Much of the pain points of my own recent experience with swscale
revolves around libswscale's very low level API, complete lack of
understanding of most AVFrame metadata, and relatively complex
configuration requirements. (vf_scale's comments even acknowledge that
libswscale should handle this stuff internally, so users just need to
point an AVFrame at both ends and let it do the right thing
automatically)

The second pain point of extending libswscale itself is that the
high-level configuration and low-level "scaling" primitives are deeply
entangled, reconfigured in various places, and hard to touch without
fear of breaking edge cases.

A new API would solve both problems. It would allow us to write new,
AVFrame-based, high-level "business logic", which could continue to call
into the low-level swscale implementation details and kernels for the
time being, and also tap into GPU filters (a la libplacebo) where
possible.

As time moves on we could replace those underlying primitives by cleaned
up rewrites where necessary, until the dependency on libswscale itself
is null.

Simultaneously, vf_scale can be rewritten on top of this API, which
would allow e.g. auto-scale filters to start doing scaling on GPU when
conversion between two hwformats is needed.