<div dir="ltr">Hello all,<div><br></div><div>I'm looking for pointers on how to reduce the latency of h.264-based multimedia stream decoding on Windows platform. I receive a data stream from a scientific instrument via the UDP multicast, so there is very little (mainly GOP size) that I can change on the encoding side.</div><div><br></div><div>On the client side I am using a bare-bones application based on the code from "<a href="https://stackoverflow.com/questions/10715170/receiving-rtsp-stream-using-ffmpeg-library">https://stackoverflow.com/questions/10715170/receiving-rtsp-stream-using-ffmpeg-library</a>". The only change is a replacement of "avcodec_decode_video2" with "avcodec_send_packet/avcodec_receive_frame" pair. I'm also using FFmpeg DLLs compiled on the same machine. My FFmpeg configure options are at the end of this email [1].</div><div><br></div><div>So far I have these settings in my AVFormatContext:</div><div><div>format_ctx->max_delay = 0;</div><div>format_ctx->probesize = 64;</div></div><div><br></div><div>Are there any other options that I can tweak to reduce the latency? I'm fine with having a lower quality picture / higher CPU usage if that reduces the overall latency.</div><div><br></div><div>To test the actual glass-to-glass latency, I'm using an external measuring instrument. So far my best result is ~182 ms. Using a Windows PerformanceCounter method [2] I measure some of the critical parts of the code. For these tests, I'm running the code compiled in x64 Release mode. The values bellow are sliding window averages over 100 measurements:</div><div><br></div><div>1) <span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">avcodec_send_packet/avcodec_receive_frame -> ~6-7 ms;</span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">2) sws_scale (YUV -> RGB) -> ~0.8 ms;</span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">3) Rendering (Creating a compatible bitmap, and then rendering in a Hwnd) -> ~8 ms.</span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div>I'm running my benchmarks against a test application
<span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(FFmpeg based, no access to the source code) </span>provided with an instrument, which consistently delivers a glass-to-glass delay of appx. 138 ms. So even if data conversion
<span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">from the YUV to RGB </span>and rendering in my app was perfect (instantaneous), my application is still appx. 36 ms. behind.</div><div><br></div><div>To answer beforehand why I'm not using the test application provided with the instrument - I need to do extra processing of the video feed before rendering it.</div><div><br></div><div>Thanks for any input!</div><div>~J</div><div><br></div><div>[1] - FFmpeg configure options and output</div><div><div>$ ./configure --toolchain=msvc --enable-shared --prefix=/c/ffmpeg --enable-asm --arch=i686</div><div>install prefix /c/ffmpeg</div><div>source path .</div><div>C compiler cl</div><div>C library msvcrt</div><div>ARCH x86 (generic)</div><div>big-endian no</div><div>runtime cpu detection yes</div><div>standalone assembly yes</div><div>x86 assembler nasm</div><div>MMX enabled yes</div><div>MMXEXT enabled yes</div><div>3DNow! enabled yes</div><div>3DNow! extended enabled yes</div><div>SSE enabled yes</div><div>SSSE3 enabled yes</div><div>AESNI enabled yes</div><div>AVX enabled yes</div><div>AVX2 enabled yes</div><div>AVX-512 enabled yes</div><div>XOP enabled yes</div><div>FMA3 enabled yes</div><div>FMA4 enabled yes</div><div>i686 features enabled yes</div><div>CMOV is fast yes</div><div>EBX available no</div><div>EBP available no</div><div>debug symbols yes</div><div>strip symbols no</div><div>optimize for size no</div><div>optimizations yes</div><div>static no</div><div>shared yes</div><div>postprocessing support no</div><div>network support yes</div><div>threading support w32threads</div><div>safe bitstream reader yes</div><div>texi2html enabled no</div><div>perl enabled no</div><div>pod2man enabled no</div><div>makeinfo enabled no</div><div>makeinfo supports HTML no</div></div><div><br></div><div>[2] - <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ms644904(v=vs.85).aspx">https://msdn.microsoft.com/en-us/library/windows/desktop/ms644904(v=vs.85).aspx</a></div><div><br></div></div>