[FFmpeg-trac] #11694(undetermined:new): Multi-GPU: NVDec cant find CUDA device when used inside a docker container and targeting specific GPU ID

[FFmpeg]

#11694: Multi-GPU: NVDec cant find CUDA device when used inside a docker container
and targeting specific GPU ID
-------------------------------------+-------------------------------------
             Reporter:  baudneo      |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:               |                  Version:  7.0
  undetermined                       |
             Keywords:  cuda NVDEC   |               Blocked By:
  docker nvidia-container-toolkit    |
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 What I was trying to accomplish:
 Multi nvidia gpu setup using a frigate tensorrt image and nvidia-
 container-toolkit. Forcing a specific GPU ID (that is not the default
 index: 0) using one of several methods results in an error:

 {{{
 [h264 @ 0x557b6d7f3900] decoder->cvdl->cuvidGetDecoderCaps(&caps) failed
 -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
 }}}

 The issue here is that from what I can tell, only ffmpeg has this issue.
 Other libs/apps in the container that target a specific GPU work: loading
 ONNX models into the GPU works when targeting a specific device_id that is
 not the default index of 0.

 If I target the default index of 0, everything works. It's only when
 targeting the non-default index. I've also tried specifying GPU_UUIDs with
 the same result. In my case, I can't physically move the GPUs around to
 force which device gets index 0, so I am left with forcing it via
 NVIDIA_VISIBLE_DEVICES env var or docker compose deploy options.

 This issue has been reproduced by other frigate users and in ubuntu cuda
 images with ffmpeg 7, so not just localized to my setup.

 Here are a couple issues:
 - [https://github.com/blakeblackshear/frigate/discussions/18018]
 - [https://github.com/blakeblackshear/frigate/discussions/18722]

 Here is an issue I opened in the nvidia-container-toolkit repo:
 [https://github.com/NVIDIA/nvidia-container-toolkit/issues/1209]

 Version:
 {{{
 ffmpeg version n7.0.2-18-g3e6cec1286-20240919 Copyright (c) 2000-2024 the
 FFmpeg developers
   built with gcc 14.2.0 (crosstool-NG 1.26.0.106_ed12fa6)
   configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static
 --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu-
 --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-
 debug --enable-iconv --enable-zlib --enable-libfreetype --enable-
 libfribidi --enable-gmp --enable-libxml2 --enable-openssl --enable-lzma
 --enable-fontconfig --enable-libharfbuzz --enable-libvorbis --enable-
 opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib
 --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth
 --enable-chromaprint --enable-libdav1d --enable-libdavs2 --enable-
 libdvdread --enable-libdvdnav --disable-libfdk-aac --enable-ffnvcodec
 --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar
 --enable-libaribcaption --enable-libass --enable-libbluray --enable-libjxl
 --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh
 --enable-libtheora --enable-libvpx --enable-libwebp --enable-libzmq
 --enable-lv2 --enable-libvpl --enable-openal --enable-libopencore-amrnb
 --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg
 --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-
 schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1
 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi
 --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-
 libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-
 libxvid --enable-libzimg --enable-libzvbi --extra-
 cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-libs='-ldl -lgomp'
 --extra-ldflags=-pthread --extra-ldexeflags=-pie --cc=x86_64-ffbuild-
 linux-gnu-gcc --cxx=x86_64-ffbuild-linux-gnu-g++ --ar=x86_64-ffbuild-
 linux-gnu-gcc-ar --ranlib=x86_64-ffbuild-linux-gnu-gcc-ranlib --nm=x86_64
 -ffbuild-linux-gnu-gcc-nm --extra-version=20240919
   libavutil      59.  8.100 / 59.  8.100
   libavcodec     61.  3.100 / 61.  3.100
   libavformat    61.  1.100 / 61.  1.100
   libavdevice    61.  1.100 / 61.  1.100
   libavfilter    10.  1.100 / 10.  1.100
   libswscale      8.  1.100 /  8.  1.100
   libswresample   5.  1.100 /  5.  1.100
   libpostproc    58.  1.100 / 58.  1.100
 }}}

 Command used:
 {{{
 /usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2
 -hwaccel cuda -hwaccel_output_format cuda -user_agent "FFmpeg
 Frigate/0.16.0-0b7a33d" -rtsp_transport tcp -timeout 10000000 -fflags
 nobuffer -flags low_delay -i rtsp://127.0.0.1:8554/living_room -f segment
 -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v
 copy -c:a aac "/tmp/cache/living_room@%Y%m%d%H%M%S%z.mp4" -r 5 -vf
 "fps=5,scale_cuda=w=1920:h=1080,hwdownload,format=nv12,eq=gamma=1.4:gamma_weight=0.5"
 -threads 2 -f rawvideo -pix_fmt yuv420p
 }}}

 FFREPORT log file: [https://pastebin.com/qKedrL6W]
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/11694>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker