<div dir="ltr">Hello all,<div><br></div><div>I'm looking for pointers on how to reduce the latency of h.264-based multimedia stream decoding on Windows platform. I receive a data stream from a scientific instrument via the UDP multicast, so there is very little (mainly GOP size) that I can change on the encoding side.</div><div><br></div><div>On the client side I am using a bare-bones application based on the code from "<a href="https://stackoverflow.com/questions/10715170/receiving-rtsp-stream-using-ffmpeg-library">https://stackoverflow.com/questions/10715170/receiving-rtsp-stream-using-ffmpeg-library</a>". The only change is a replacement of "avcodec_decode_video2" with "avcodec_send_packet/avcodec_receive_frame" pair. I'm also using FFmpeg DLLs compiled on the same machine. My FFmpeg configure options are at the end of this email [1].</div><div><br></div><div>So far I have these settings in my AVFormatContext:</div><div><div>format_ctx->max_delay = 0;</div><div>format_ctx->probesize = 64;</div></div><div><br></div><div>Are there any other options that I can tweak to reduce the latency? I'm fine with having a lower quality picture / higher CPU usage if that reduces the overall latency.</div><div><br></div><div>To test the actual glass-to-glass latency, I'm using an external measuring instrument. So far my best result is ~182 ms. Using a Windows PerformanceCounter method [2] I measure some of the critical parts of the code. For these tests, I'm running the code compiled in x64 Release mode. The values bellow are sliding window averages over 100 measurements:</div><div><br></div><div>1)Â <span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">avcodec_send_packet/avcodec_receive_frame -> ~6-7 ms;</span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">2)Â sws_scale (YUV -> RGB) -> ~0.8 ms;</span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">3) Rendering (Creating a compatible bitmap, and then rendering in a Hwnd) -> ~8 ms.</span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div>I'm running my benchmarks against a test application
<span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(FFmpeg based, no access to the source code)Â </span>provided with an instrument, which consistently delivers a glass-to-glass delay of appx. 138 ms. So even if data conversion
<span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">from the YUV to RGB </span>and rendering in my app was perfect (instantaneous), my application is still appx. 36 ms. behind.</div><div><br></div><div>To answer beforehand why I'm not using the test application provided with the instrument - I need to do extra processing of the video feed before rendering it.</div><div><br></div><div>Thanks for any input!</div><div>~J</div><div><br></div><div>[1] - FFmpeg configure options and output</div><div><div>$ ./configure --toolchain=msvc --enable-shared --prefix=/c/ffmpeg --enable-asm --arch=i686</div><div>install prefix      /c/ffmpeg</div><div>source path        .</div><div>C compiler        cl</div><div>C library         msvcrt</div><div>ARCH           x86 (generic)</div><div>big-endian        no</div><div>runtime cpu detection   yes</div><div>standalone assembly    yes</div><div>x86 assembler       nasm</div><div>MMX enabled        yes</div><div>MMXEXT enabled      yes</div><div>3DNow! enabled      yes</div><div>3DNow! extended enabled  yes</div><div>SSE enabled        yes</div><div>SSSE3 enabled       yes</div><div>AESNI enabled       yes</div><div>AVX enabled        yes</div><div>AVX2 enabled       yes</div><div>AVX-512 enabled      yes</div><div>XOP enabled        yes</div><div>FMA3 enabled       yes</div><div>FMA4 enabled       yes</div><div>i686 features enabled   yes</div><div>CMOV is fast       yes</div><div>EBX available       no</div><div>EBP available       no</div><div>debug symbols       yes</div><div>strip symbols       no</div><div>optimize for size     no</div><div>optimizations       yes</div><div>static          no</div><div>shared          yes</div><div>postprocessing support  no</div><div>network support      yes</div><div>threading support     w32threads</div><div>safe bitstream reader   yes</div><div>texi2html enabled     no</div><div>perl enabled       no</div><div>pod2man enabled      no</div><div>makeinfo enabled     no</div><div>makeinfo supports HTML  no</div></div><div><br></div><div>[2] - <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/ms644904(v=vs.85).aspx">https://msdn.microsoft.com/en-us/library/windows/desktop/ms644904(v=vs.85).aspx</a></div><div><br></div></div>