[FFmpeg-devel] [PATCH] avutil/WIP: add AVAsync API

[Zhang Rui]

2015-11-11 8:27 GMT+08:00 Clément Bœsch <u at pkh.me>:
> From: Clément Bœsch <clement at stupeflix.com>
>
> ---
>
> So here is a first prototype of a higher level API following an
> asynchronous model to (for now) simply use the current synchronous API.
>
> I suggest to look at libavutil/async.h (sorry no doxy yet) and
> doc/examples/async_demuxing_decoding.c for an example of usage.
>
> Basically what the current draft proposes for the user is to instanciate
> an asynchronous context, in which you register Readers (the user will
> generally do a demuxing in the associated callback to get a packet), in
> which you register Decoders (basically just a wrapper for an
> AVCodecContext) to which you associate a push frame callback.
>
> You construct something like this:
>
>   AVAsyncContext(
>     AVAsyncReader(
>       AVAsyncDecoder(...),
>       AVAsyncDecoder(...),
>       AVAsyncDecoder(...),
>       ...
>     ),
>     AVAsyncReader(
>       AVAsyncDecoder(...),
>       AVAsyncDecoder(...),
>       ...
>     ),
>     ....
>   )
>
> You will generally have a Reader per file but that can be anything you
> want as long as you can pull of packets.
>
> Now implementation wise, the interesting part is that every level is
> asynchronous:
>
> - The context will spawn a thread for each reader
> - Then in each reader, you will have a thread for each decoder
> - Then each decoder will have its own queue of packets, and its own
>   queue of output frames
> - Then along with each decoder, you will have Watcher in its dedicated
>   thread: the watcher is monitoring the frame queue of its decoder, and
>   will call the user push frame callback anytime a new one appears in
>   the queue.
>
> The interesting part about the watcher is that it will allow the user to
> transparently do any operation on the frame without blocking the
> decoding or the demuxing: basically, blocking in push_frame for the user
> will just block any subsequent call to the callback, but the packet
> and frame queues of the decoder will continue to fill up, ready to be
> pop'ed.
>
> Note: the size of the packet and frame queue for each decoder is
> configurable
>
> Note2: the packet queue is not set in the reader so that a decoder that
> isn't pulling its packets and decoding fast enough will not block the
> others. As a result, the Reader just reads one packet and is then
> responsible for broadcasting it to the appropriate decoder (by just
> adding it to its internal queue).
>
> Now all of this doesn't require any modification to the current API so
> it doesn't really solve the problem of writing asynchronous
> decoders/hwaccel. The goal for the next iteration will be to adjust the
> VideoToolbox hwaccel to make use of the asynchronous model, and
> basically just make it call the user callback by itself. It will be a
> good candidate for such thing.
>
> Another concerning limitation in the current code is the inability to
> seek: basically the user can not lock the codec and reader context to
> execute a seek, nor flush the queues. And I'd like suggestion of
> direction in that regard:

If we want a fat reader, seek should be handled inside reader,
and let user call AVAsyncReader.seek_async(...),
to request reader handle seek operation at next round before av_read_frame().

If we want a thin reader, just let user push packet into
reader's internal packet queue.

Anyhow, I don't think it's a good idea to put av_read_frame()
and avformat_seek_file() into different thread.

> - first, in threadmessage it seems I can't yet flush the fifo properly
>   (for example by calling av_frame_free() on each entry of the frame
>   queue): should I extend the API? Nicolas, maybe you have a suggestion?

Maybe serial number in frame_queue as ffplay's packet_queue?

> - secondly, how should I allow the user to lock/unlock the reader (and
>   as a result the likely AVFormatContext user side) and decoder? I added
>   a few prototypes in async.h but I'm pretty sure that won't fit many
>   cases.  Comments very welcome.

IMO, we don't need to allow user to lock/unlock reader/decoder,
we just provide thread-safe reader/decoder API.

> I hope this is going in a direction most developers like. If it's not
> the case, I'm of course open to any reworking but please be specific.
>
> Also, I'm really sorry I'm not sending this to libav-devel even so I
> apparently looked forward a colaboration in a previous thread. I was
> planning to but unfortunately the threadmessage API is not there yet and
> it would have require way too much work for an initial prototype. When
> we come up with something that satisfy everyone and this code becomes
> something more than just a draft, I will maybe do the necessary.
> Unfortunately, porting VideoToolbox accel, threadmessage and probably a
> bunch of other ffmpeg specific features (I can thing about
> av_dynarray2_add and maybe more) is going to require a bit too much code
> adjustment, so I might just share the async.h and redirect them to a
> working implementation here....
>
> Anyway, please comment.