[FFmpeg-user] Intel QSV and xstack_qsv: Error while filtering: Internal bug, should not have happened

[Shane Warren]

I'm testing out an Intel Flex 140 card and I am attempting to transcode 4 inputs to 1 output using the xstack_qsv filter.  My command takes in 4 udp multicast ts files (h264 encoded) and outputs one h264_qsv encoded udp transport stream. My command looks like so:

/tmp/ffmpeg_g  -nostats -loglevel verbose -init_hw_device qsv=card1:/dev/dri/card1 -hwaccel_device card1 -filter_hw_device card1 -fflags discardcorrupt  -fflags +genpts -fflags discardcorrupt  \
-hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -i "udp://@225.105.0.1:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" \
-hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -i "udp://@225.105.0.14:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" \
-hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -i "udp://@225.105.0.5:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" \
-hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -i "udp://@225.105.0.4:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" \
-noautoscale -filter_complex "\
[0:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v1]; \
[1:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v2]; \
[2:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v3]; \
[3:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v4]; \
[v1][v2][v3][v4] xstack_qsv=inputs=4:layout=0_0|0_h0|w0_0|w0_h0[mosiac]; \
[mosiac]vpp_qsv=w=1920:h=1080,hwdownload,format=nv12,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,hwupload=extra_hw_frames=64,format=qsv[v]"  \
-map "[v]" -map 0:a:0 -map 1:a:0 -map 2:a:0 -map 3:a:0 \
-c:v h264_qsv -noautoscale -async_depth 1 -scenario livestreaming -r:v 30000/1001 -b:v 6000k -minrate:v 6000k -maxrate:v 6000k -bufsize:v 12000k -threads 1 -profile:v high -bf:v 0 -g:v 15 \
-filter:a "aresample=async=1" -c:a aac -ac:a 2 -ar:a 48000 -b:a 192k -fps_mode auto \
-f mpegts -muxrate 7450599 -pes_payload_size 1528 "udp://@229.100.100.44:10102?pkt_size=1316&fifo_size=90000&bitrate=7450599&burst_bits=10528"

The issue is eventually stream 1 changes resolution from 1280x720 (progressive) to 1920x1080 (interlaced). The stream dies with the following logs coming out:

[h264_qsv @ 0x55a240e68000] Error submitting the frame for encoding.
[vost#0:0/h264_qsv @ 0x55a240e30780] Error submitting video frame to the encoder
Error while filtering: Internal bug, should not have happened

I can transcode the single stream that changes resolution on its own and it can survive a resolution change using this command:

/tmp/ffmpeg_g  -nostats -loglevel verbose -init_hw_device qsv=card1:/dev/dri/card1 -hwaccel_device card1 -filter_hw_device card1 -fflags discardcorrupt  -fflags +genpts -fflags discardcorrupt  -hwaccel qsv -hwaccel_output_format qsv -async_depth 1 -i "udp://@225.105.0.1:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" -vf "vpp_qsv=deinterlace=2:w=1920:h=1080:framerate=30000/1001" -map "0:v" -map 0:a:0 -c:v h264_qsv -noautoscale  -async_depth 1 -scenario livestreaming -r:v 30000/1001 -b:v 5000k -minrate:v 5000k -maxrate:v 5000k -bufsize:v 10000k -a53cc 1 -forced_idr 1 -threads 1 -profile:v high -bf:v 0 -g:v 15 -filter:a "aresample=async=1" -c:a aac -ac:a 2 -ar:a 48000 -b:a 192k -fps_mode auto -f mpegts -muxrate 6450599 -pes_payload_size 1528 "udp://@229.100.100.44:10102?pkt_size=1316&fifo_size=90000&bitrate=6450599&burst_bits=10528"

When that stream changes resolution I see these come out in the log, and the stream carries on without dying:

[h264_qsv @ 0x561bc472f8c0] Video parameter change
[AVHWDeviceContext @ 0x7f313000d1c0] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 23.2.4 ().
[AVHWDeviceContext @ 0x7f313000d1c0] Driver not found in known nonstandard list, using standard behaviour.
[h264_qsv @ 0x561bc472f8c0] Decoder: output is video memory surface

My quesiton is why does the stream die when using xstack_qsv but works fine on its own.