[FFmpeg-user] Low latency H.264 streaming with ffmpeg

[Greger Burman]

I would like to hear from anyone who has experience with low latency video
streaming and ffmpeg.

In my application I take video from a PCIe capture device and stream live
over network. The end to end latency must be below 200ms otherwise it is
not usable. Let us assume that the network latency is near zero. This
question is about encoder+decoder+buffering latency.

Very low latency can be achieved when using the x264 encoder by using
settings like tune zerolatency. From my own testing with x264 (without
ffmpeg) I can confirm that this works. I have measured approx 90 ms (3
frames) of latency added by encoder+decoder combined. Awesome.

Sadly I have been unable to get anywhere near those numbers when using
ffmpeg and libx264. As player I use either ffplay or a dot-net player built
on the ffmpeg libraries, with about equal results. The lowest end to end
latency I have seen is 400-500ms. That includes capture latency also.

Yes, there are many unknowns here. To help pin point the problem I used
mjpeg streaming for comparison. Same everything except I change ffmpeg
vcodec to mjpeg and format to mjpeg. I change nothing with ffplay. With
mjpeg I now get approx 150ms end to end latency.

Observation:
x264 codec + mpegts format compared to mjpeg codec + mjpeg format adds
about 300 ms (10 frames) latency.

I have tried every ffplay-, ffmpeg- and libx264 setting that I believe
relevant. Most have not made any noticable difference in latency. I will
not list a lot of settings here. Here are the basic settings and commands
that I have used. I can post more details on request.

x264-preset:
vcodec=libx264
thread_type=slice
slices=1
# x264
profile=baseline
level=32
preset=superfast
tune=zerolatency
intra-refresh=1
crf=15
x264-params=vbv-maxrate=5000:vbv-bufsize=1:slice-max-size=1500:keyint=60

$ ffmpeg -r 30 -f dshow -i video="devicename" -pix_fmt yuv420p -an -vpre
x264-preset -f mpegts udp://127.0.0.1:8888

$ ffplay -fflags nobuffer -flags low_delay -framedrop -strict experimental
udp://127.0.0.1:8888

Everything is run on Windows 7 64-bit, with 32-bit executables.
ffmpeg version N-76539-g480bad7 Copyright (c) 2000-2015 the FFmpeg
developers
  built with gcc 5.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads
--enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r
--enable-gnutls --enable-iconv --enable-libass --enable-libbluray
--enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype
--enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug
--enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb
--enable-libopenjpeg --enable-libopus --enable-librtmp
--enable-libschroedinger --enable-libsoxr --enable-libspeex
--enable-libtheora --enable-libtwolame --enable-libvidstab
--enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis
--enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264
--enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg
--enable-lzma --enable-decklink --enable-zlib
  libavutil      55.  5.100 / 55.  5.100
  libavcodec     57. 15.100 / 57. 15.100
  libavformat    57. 14.100 / 57. 14.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 15.100 /  6. 15.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100

cheers
-- 
Greger Burman