[FFmpeg-user] Support: h264_vaapi unavailable as decoder

[Brian O'Connor]

Hi, apologies if this isn't the right forum for this.  I'm very new to
configuring and tuning ffmpeg, and have been searching for solutions
to this problem for a few days.  I don't know where else to look for
information.

Summary
h264_vaapi is not present as a decoder, but is present as an encoder
and is listed under the available hwaccels.  vainfo, AFAICT, reports
that the GPU is capable of hardware decoding of h264 streams

Context
I am attempting to utilize hardware acceleration for a NVR software
that decodes rtsp streams coming in encoded as h264.  No matter what I
have done in a variety of contexts, I cannot get ffmpeg to utilize
hardware acceleration via VA-API

Hardware + Software
OS: Ubuntu 20.04 server, headless
Graphics card: Asus ATI Radeon HD6450

FFMpeg version:
commit eda2a50c8e9381a357f936a190659bb039e0c038

More info
Every time I try to hardware decode the RTSP stream, it uses the
native FFMpeg decoder instead, resulting in high CPU usage.

I don't understand why h264_vaapi is not available as an h264 decoder.
I pulled the source down from Github today and compiled locally, and I
am seeing the same results as in other environments (namely docker
containers).

What appears notable (to me, at least) is that after running the
configure command, h264_vaapi is not listed as an enabled encoder.
However, h264_vaapi is listed as an available decoder, so it would
seem that this is something that is known at configure time.  I do not
know how to get past this.

Below is output of some various commands that appear helpful in
debugging this.  I have also saved this output to a github gist as
well: https://gist.github.com/btoconnor/096a423c83709e7be1eb89b206ba276f

Any help would be greatly appreciated.  Happy to provide more output /
information if helpful.

$ vainfo
error: XDG_RUNTIME_DIR not set in the environment.
error: can't connect to X server!
libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/r600_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Mesa Gallium driver 21.0.3 for AMD CAICOS (DRM
2.50.0 / 5.11.0-38-generic, LLVM 12.0.0)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

configure command:
$ ./configure --prefix="$HOME/ffmpeg_building/build"
--bindir="$HOME/bin" --extra-libs="-lpthread -lm"   --ld="g++"
--bindir="$HOME/bin"   --enable-gpl   --enable-gnutls
--enable-libaom   --enable-libass   --enable-libfdk-aac
--enable-libfreetype   --enable-libmp3lame   --enable-lilibopus
--enable-libvorbis   --enable-libvpx   --enable-libx264
--enable-nonfree   --enable-vaapi --enable-libvpx
--enable-hwaccel=h264_vaapi --enable-shared --arch=amd64

output of configure (edited for length, full output available in gist above):
<SNIP>
Enabled decoders:
<SNIP>
h264
h264_v4l2m2m
<SNIP>
Enabled encoders:
<SNIP>
h264_v4l2m2m
h264_vaapi
<SNIP>

Enabled hwaccels:
av1_vaapi               h264_vaapi              hevc_vaapi
 mjpeg_vaapi             mpeg2_vaapi             mpeg4_vaapi
  vc1_vaapi               vp8_vaapi               vp9_vdpau
   wmv3_vdpau
h263_vaapi              h264_vdpau              hevc_vdpau
 mpeg1_vdpau             mpeg2_vdpau             mpeg4_vdpau
  vc1_vdpau               vp9_vaapi               wmv3_vaapi

Example output trying to decode RTSP stream:
$ ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device
/dev/dri/renderD128 -i rtsp://redacted//h264Preview_01_sub -f null -
ffmpeg version N-104414-geda2a50c8e Copyright (c) 2000-2021 the FFmpeg
developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/home/brian/ffmpeg_building/build
--bindir=/home/brian/bin --extra-libs='-lpthread -lm' --ld=g++
--bindir=/home/brian/bin --enable-gpl --enable-gnutls --enable-libaom
--enable-libass --enable-libfdk-aac --enable-libfreetype
--enable-libmp3lame --enable-libopus --enable-libvorbis
--enable-libvpx --enable-libx264 --enable-nonfree --enable-vaapi
--enable-libvpx --enable-libx264
  libavutil      57.  7.100 / 57.  7.100
  libavcodec     59. 12.100 / 59. 12.100
  libavformat    59.  6.100 / 59.  6.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8. 15.100 /  8. 15.100
  libswscale      6.  1.100 /  6.  1.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
Input #0, rtsp, from 'rtsp://redacted//h264Preview_01_sub':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h264Preview_01_sub
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 640x352,
29.58 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))

Brian O'Connor