[FFmpeg-devel] [PATCH] lavc/vaapi_encode: Async the encoding and output procedure of encoder

[Fu, Linjie]

> Mark Thompson:
> Sent: Monday, November 18, 2019 07:14
> To: ffmpeg-devel at ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH] lavc/vaapi_encode: Async the
> encoding and output procedure of encoder
> 
> On 07/11/2019 16:32, Linjie Fu wrote:
> > Currently, vaapi encodes a pic if all its references are ready,
> > and then outputs it immediately by calling vaapi_encode_output.
> >
> > However, while working on output procedure, hardware is be able to
> > cope with encoding tasks in the meantime to have the better performance.
> >
> > So a more efficient way is to send all the pics with available refs to
> > hardware to allow encoding while output.
> >
> > It's what vaapi originally did before the regression, and the performance
> > could be improved for ~20%.
> >
> > CMD:
> > ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128
> > -hwaccel_output_format vaapi -i
> bbb_sunflower_1080p_30fps_normal.mp4
> > -c:v h264_vaapi -f h264 -y /dev/null
> >
> > Source:
> > https://download.blender.org/demo/movies/BBB/
> >
> > Before:
> >     ~164 fps
> > After:
> >     ~198 fps
> >
> > Fix #7706.
> >
> > Signed-off-by: Linjie Fu <linjie.fu at intel.com>
> > ---
> 
> The sync in the same receive call here is required for correctness because of
> the how VAAPI syncs to the input surface (consider the interactions of a split
> followed by two encoders using the same surfaces).  I didn't realise that for a
> long time, hence the error existing in earlier versions.

Indeed, you're right 1:N encoding procedure suffered from this. And the proper
fix is to provide a new API to sync the coded buffer which is independent for each
encoding task, instead of syncing the shared input surface. (See comments below)

> 
> Relatedly, this change would significantly increase latency on Intel platforms
> because vaEndPicture() is mostly synchronous there, and you're now calling
> it multiple times before returning anything.

As the the latency, how about adding an option like "-async 0/1" to specify the
user requirement:
For live streaming, user may use -async 0 to minimize the latency;
For video on demand, user may use -async 1 to maximize the performance.

> 
> More generally, though, the API definition here is just really stupid.  If you're
> interested it would be much better to fix this in the API - sync-to-output of
> some kind would make far more sense, as would some kind of event-based
> system (extra points if it can interop via fds to normal poll() and Vulkan).
> 
We have a proposal[1] to libva to introduce the new function to make encoder
synchronization by output bitstream., and looking forward to your comments.

- Linjie

[1] <https://github.com/intel/libva/pull/408>