[FFmpeg-trac] #7650(avcodec:new): FFmpeg MPEG-2 encode fails to "invalid RC mode" when iHD driver is used through VA-API instead of QSV

[FFmpeg]

#7650: FFmpeg MPEG-2 encode fails to "invalid RC mode" when iHD driver is used
through VA-API instead of QSV
-------------------------------------+-----------------------------------
             Reporter:  eero-t       |                    Owner:
                 Type:  defect       |                   Status:  new
             Priority:  normal       |                Component:  avcodec
              Version:  unspecified  |               Resolution:
             Keywords:               |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-----------------------------------

Comment (by eero-t):

 I can verify that either of these will allow FFmpeg to work.

 However, with the patch, I get much worse PSNR / SSIM with VA-API:
 {{{
 ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128
 -hwaccel_output_format vaapi -i 720x480p_30.00_4mb_cabac.264 -c:v
 mpeg2_vaapi -b:v 2M -compression_level 4 -y output.mpg
 ...
 ffmpeg -i 720x480p_30.00_4mb_cabac.264 -i output.mpg -filter_complex
 "ssim;[0:v][1:v]psnr" -frames 2400 -f null -
 ...
 SSIM Y:0.592489 (3.898609) U:0.864354 (8.675915) V:0.905782 (10.258669)
 All:0.690015 (5.086599)
 PSNR y:16.888131 u:27.188619 v:30.218137 average:18.499885 min:13.885003
 max:36.662048
 }}}

 than with QSV:
 {{{
 ffmpeg -hwaccel qsv -c:v h264_qsv -i 720x480p_30.00_4mb_h264_cabac.264
 -c:v mpeg2_qsv -b:v 2M -compression_level 4 -y output.mpg
 ...
 ffmpeg -i 720x480p_30.00_4mb_cabac.264 -i output.mpg -filter_complex
 "ssim;[0:v][1:v]psnr" -frames 2400 -f null -
 ...
 SSIM Y:0.963254 (14.347895) U:0.976697 (16.325860) V:0.976484 (16.286427)
 All:0.967700 (14.907913)
 PSNR y:37.747504 u:44.358034 v:44.510943 average:39.066017 min:31.195431
 max:52.336077
 }}}

 I.e. average PSNR drops to half, from 39 with QSV, to 18.5 with VA-API.

 In case it matters, there are following additional warnings with VA-API:
 {{{
 [mpeg2_vaapi @ 0x56228b35f100] Driver does not support some wanted packed
 headers (wanted 0x3, found 0x10).
 [mpeg2_vaapi @ 0x56228b35f100] Sample aspect ratio 10:11 is not
 representable, signalling square pixels instead.
 }}}

 However, if I drop the "-b:v 2M" option, I still see these extra warnings,
 but PSNR with VA-API is close to one with QSV.

 In few other tests I did, where output is H264 encoded, there was no
 PSNR/SSIM metrics difference between using VA-API and QSV.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/7650#comment:2>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker