[FFmpeg-trac] #7968(avcodec:new): av_hwdevice_ctx_init fails for AV_HWDEVICE_TYPE_VAAPI with some drivers

[FFmpeg]

#7968: av_hwdevice_ctx_init fails for AV_HWDEVICE_TYPE_VAAPI with some drivers
----------------------------------+--------------------------------------
             Reporter:  swbrowne  |                     Type:  defect
               Status:  new       |                 Priority:  normal
            Component:  avcodec   |                  Version:  git-master
             Keywords:            |               Blocked By:
             Blocking:            |  Reproduced by developer:  0
Analyzed by developer:  0         |
----------------------------------+--------------------------------------
 The problem occurs in `vaapi_decode_find_best_format` because it uses
 `vaQuerySurfaceAttributes` to find the software format. The problem is the
 versions of both `nvidia_drv_video.so` and `vdpau_drv_video.so` on Ubuntu
 18.04 both fail with invalid parameter when specifying a null attribute
 list to get the size required. This causes the following output:
 {{{
 Failed to query surface attributes: 18 (invalid parameter).
 Failed setup for format vaapi_vld: hwaccel initialisation returned error.
 }}}

 In the latest release of libva-vdpau-driver (0.74b) they no longer return
 with invalid parameter, but they only return 2 surface attributes and
 neither of them are a pixel format so it will still fail. Note that this
 hasn't changed in head as of this ticket either. See:
 https://github.com/arekinath/libva-vdpau-
 driver/blob/14b32901a96e5bf135d06e8dcf3d60a3c841fc14/src/vdpau_video.c#L314

 I can work up a patch when I get some time, but my proposed solution would
 be to ignore `vaQuerySurfaceAttribute` failures and if no pixel format is
 found from surface attributes fall back to using `vaMaxNumImageFormats`
 and `vaQueryImageFormats`.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/7968>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker