[FFmpeg-trac] #6431(undetermined:closed): Linux ffmpeg always segfaults using NVIDIA hw decoders/encoders

[FFmpeg]

#6431: Linux ffmpeg always segfaults using NVIDIA hw decoders/encoders
-------------------------------------+-------------------------------------
             Reporter:  bubbleguuum  |                    Owner:
                 Type:  defect       |                   Status:  closed
             Priority:  normal       |                Component:
              Version:  unspecified  |  undetermined
             Keywords:  nvidia       |               Resolution:
             Blocking:               |  worksforme
Analyzed by developer:  0            |               Blocked By:
                                     |  Reproduced by developer:  1
-------------------------------------+-------------------------------------

Comment (by bubbleguuum):

 I compiled ffmpeg myself and found the explanation for this crash.

 long story short: nvdec, cuda and friends using dlopen() are not
 compatible with ffmpeg compiled with -static generally used to produce a
 fully static build.

 The reason is that it segfaults dlopen'ing libcuda.so.1, due to this
 warning displayed at compilation:

  warning: Using 'dlopen' in statically linked applications requires at
 runtime the shared libraries from the glibc version used for linking

 After much tinkering, I was able to produce a build with all libraries
 statically linked except the glibc (and thus libdl):

 1. pass --extra-ldexeflags="-Wl,-Bstatic"  to configure. This replaces
 -static and the libraries following this option will be linked statically
 *exept the glibc*. Thus this is not the same thing than -static that force
 static linking for all libraries including the glibc.

 2. pass --extra-libs="-Wl,-Bdynamic -lm -ldl".  This force dynamic linking
 for libm and libdl  . This override is necessary because these libraries
 cannot be linked statically when the glibc is linked dynamically

 3. You must edit the configure script (yuk), function probe_cc():

 change:

 _flags_filter=echo

 to:

 _flags_filter='filter_out -lm|-ldl'

 This will filter out all -lm and -ldl occurences before -Wl,-Bdynamic.
 This is necessary for both these libs to not be linked statically. Finding
 this trick took me forever.


 If all goes well, at the end you have a ffmpeg binary with everything
 statically linked except the glibc and dynamic linker, with working NVIDIA
 support:

 {{{
 ldd ffmpeg
         linux-vdso.so.1 =>  (0x00007ffece9e6000)
         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f19931dc000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
 (0x00007f1992fd8000)
         libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
 (0x00007f1992dc1000)
         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
 (0x00007f1992ba4000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f19927db000)
         /lib64/ld-linux-x86-64.so.2 (0x0000560591227000)

 }}}


 Suggestion: make the configure script issue a big fat warning (or fail
 entirely) if -static is specified and any library using dlopen() is to be
 included in the build.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/6431#comment:7>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker