[FFmpeg-devel] [PATCH 0/5] VAAPI support infrastructure, encoders
sw at jkqxz.net
Sun Jan 17 18:33:31 CET 2016
This patch series contains the following:
* Some VAAPI infrastructure in libavutil. Initialisation code, along
with routines to handle mapping and copying to/from VA surfaces.
Doesn't really fit as a public API, but has to go there because it's
used by the parts in both libavfilter and libavcodec.
* VAAPI decode hwaccel helper. This allows use of the VAAPI hwaccel
with ffmpeg itself, rather than only through third-party use of libavcodec.
* VAAPI H.264 encoder. This is mostly a token effort to see things
working - I moved on to the H.265 encoder once it worked at all. The
other patch offered on this list recently (the one that looks
suspiciously like an unacknowleged copy of libva
test/encode/h264encode.c) would certainly offer better quality for now,
though it lacks the integration to use the video processing at the same
* VAAPI H.265 encoder. This is a simple working encoder: it only
supports IDR and P frames at fixed QP so far, so the output is not very
good. I am intending to work on this to make it better, see below.
* VAAPI conversion filter. It exists to allow sensible transforms
(scale, colour-convert from RGB) using hardware in front of the encoder,
and is currently useless for anything else because it can only output
VAAPI surfaces. Adding more outputs would be straightforward, though
not very useful without being able to feed it VAAPI surfaces as input
from a decoder (see below). Another improvement would be to support
more available VPP filters - deinterlacing, other colour transforms, etc.
All of this code was written by me alone. Some inspiration but no code
was taken from libva examples and from libyami (mostly around what codec
options I am allowed to set).
Tested with Intel Quick Sync gen 7 (Haswell/4500U), gen 8
(Braswell/N3700) and gen 9 (Skylake/6300), all with libva 1.6.2 and
Linux 4.3.3 (current Debian stretch/testing).
* The bits in libavutil don't really make a very good public API. Needs
documentation if it does become one. "AVVAAPI" is a terrible namespace
prefix to write everywhere. There should probably be a common way to
select the default VAAPI connection (top-level option on ffmpeg itself?).
* The configure tests need to make a more intelligent decision of when
to enable things. (H.265 encode is not present before libva 1.6.0, say.)
* There may well be memory leaks. I don't fully understand how the
reference counting and auxiliary buffers in AVFrames work, so it would
be very helpful if someone familiar with them could have a look at it
(and suggest a better method which I can implement, if appropriate).
* Timestamps do something funny - I pass the pts values through in what
I think is the correct way, but without '-vsync 0' it perpetually
complains that they are somehow wrong ("Past duration ... too large").
Maybe some of the clocks aren't running at the same rate? Someone
familiar with this can probably instantly explain what I've done wrong.
* The colour conversion isn't quite working as I would hope - it seems
to saturate some colours? This is probably fixable by being clearer
about the colour spaces involved.
* The encoders aren't very good.
Things I intend to work on:
* Clean this up to get it integrated.
* Improve the H.265 encoder: B frames and some sort of GOP
configuration, along with better bitrate control. Not sure what would
be useful to do after that; suggestions welcome.
* Investigate the various copies to/from VAAPI surfaces. It should be
possible to improve this, at least in some cases.
./ffmpeg -hwaccel vaapi -i in.mp4 -f null /dev/null
Decode input using the vaapi hwaccel, throw the output away.
./ffmpeg -vsync 0 -f x11grab -s 1920x1080 -i :0.0+0,0 -an -c:v
vaapi_hevc -qp 18 -idr_interval 120 out.mp4
Capture the X11 screen and encode it to H.265 at low QP. s/hevc/h264/
to get H.264 output instead, which will work on non-Skylake. (Remove
'-vsync 0' to see timestamp problem mentioned above.)
./ffmpeg -vsync 0 -f x11grab -s 1920x1080 -i :0.0+0,0 -an -vf vaapi_conv
-c:v vaapi_hevc -qp 18 -idr_interval 120 out.mp4
Same as previous, but with the colour-conversion from RGB to NV12 now
happening in hardware using the vaapi_conv filter (VAAPI surfaces are
passed between the filter and the encoder).
./ffmpeg -vsync 0 -hwaccel vaapi -i in.mp4 -vf vaapi_conv=size=1280x720
-an -c:v vaapi_hevc -qp 30 -idr_interval 120 out.mp4
Decode input using the vaapi hwaccel, scale to 1280x720 with vaapi_conv
and encode with H.265.
This works, but unfortunately not entirely in hardware. The hwaccel
setup very much wants a normal output from the hwaccel helper, so you
end up copying the surface into a separate buffer, then back into a
different surface on the filter input. I'm not sure how to fix this
without hacking the decoder side horribly (it needs to select the VAAPI
output format very early, before the filter is set up).
Other random notes:
H.265/HEVC naming: what should I be using here? Existing practice isn't
very clear: it seems to be mostly H.264 (ITU naming) and HEVC (ISO/MPEG
naming), but with some instances of the other one in each case. I
mildly prefer H.265, but I've used hevc in all variable names to try to
stay consistent with (some of the) existing practice. It would be easy
to change if there is consensus the other way.
A warning to anyone playing with the H.265 encoder on Skylake: the
drivers are not robust to errors! It is not difficult to hang your GPU
if you fiddle with the low-level parameters. It did always recover for
me (the kernel driver spots the hang and resets everything after a few
seconds), but that sounds like undefined behaviour so YMMV. (Easy
reproduction case if anyone feels like debugging some drivers: set
log2_diff_max_min_luma_coding_block_size to 3 rather than 2.)
All comments welcome.
More information about the ffmpeg-devel