[FFmpeg-trac] #2445(undetermined:new): Live audio/video gets out of sync when the output (file, network, pipe) lags or is too slow.

[FFmpeg]

#2445: Live audio/video gets out of sync when the output (file, network, pipe)
lags or is too slow.
-------------------------------------+-------------------------------------
             Reporter:  Max-P        |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:               |                  Version:  git-
  undetermined                       |  master
             Keywords:               |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 Hello everyone, first please excuse my lenghty bug report. I'm trying to
 be as presise and helpful as possible, please correct me if I miss
 anything here.

 Context: I was trying to stream a game from my desktop with the audio that
 comes with it, but always got a random delay with the audio. I couldn't
 get it working right, so I searched Google a lot and kind of figured the
 problem by taking apart each step of the encoding.

 Summary of the bug: When converting live sources (here x11grab and pulse
 sources), the audio gets out of sync with the video no matter which
 settings are used (-vsync, -af aresample, -async, -isync, -af asetpts=PTS-
 STARTPTS, -vf setpts=PTS-STARTPTS, none of them change anything). I found
 that this is due to the output being unresponsive or too slow at start,
 because it stays in sync when I output to a file, and I found a way to
 reliably tell it's not the tcp:// or rtmp:// protocol of FFmpeg that
 causes problems.


 How to reproduce:

 In order to simulate the buggy or laggy output, I made three C programs
 (attached with the ffmpeg -report log to this ticket), and all examples
 were ran with this command. I understand it's not as good, but it allows
 me to rule out any FFmpeg internal protocol bugs:
 {{{
 % ffmpeg -report \
         -f x11grab -s 1280x720 -r 30 -show_region 1 -i $DISPLAY \
         -f pulse -ac 2 -ar 44100 -i default \
         -vcodec libx264 -b:v 500k -preset veryfast \
         -pix_fmt yuv420p -threads 4 -r 30 -s 1280x720 \
         -acodec libmp3lame -ab 192k \
         -f flv -y - | tee test.flv | ./slow
 }}}

 So, to explain why I do this: I tell the first ffmpeg instance to output
 to stdout, so I can pipe it to any other program I want. I did this,
 because I noticed local files were always in perfect sync without any
 trouble, but as soon as I tried to pipe a ffmpeg to another ffmpeg to get
 it to my server via RTMP, it got out of sync again.

 So I pipe it to tee, which will output the result to a local file on disk,
 and pipe it again to another program.

 1- I pipe it to ./void, which is the program in void.c. It reads STDIN as
 fast as it can. The resulting test.flv file is in perfect sync for as long
 as I tried it (5 minutes). It works the exact same was as just letting
 ffmpeg outputting to test.flv directly. This is the expected behavior.

 2- I pipe it to ./slow, which is a variant of the void.c program, but it
 wait 3 seconds before reading the data, reads it all and continue reading
 it a bit, then wait 3 seconds again and repeat over and over forever. I
 was lazy, so it doesn't do proper rate limiting, but it's good enough for
 the tests. It simulated periodic lags in the output, which seems to happen
 a lot when streaming to slow networks or servers. The resulting test.flv
 file gets progressively out of sync. After only 20 seconds, the audio is
 already a second off from the video, and it gets worse and worse. '''The
 ffmpeg command is left untouched the whole time, only the piped command is
 changed.''' This lets me suppose that when the output buffer is completely
 filled, the input buffers will get filled, but one of the video or audio
 buffers can hold more data than the other, so when the output clears out,
 both sources are now out of sync.

 3- Just for testing sake, I piped it to my last program, idelay.c, which
 simply waits 10 seconds before reading as fast as it can like the void.c
 program. It simulates an initial network setup delay, which is possibly
 what rtmp:// does for me and gets my stream all out of sync. The output
 test.flv file is out of sync by a second or two, but delay seems constant
 afterward.

 4- Of course, piping to "ffmpeg -i - -codec copy -f flv
 rtmp://my.server.tld" causes the same initial delay, which breaks my live
 stream just as much as using rtmp://server directly in the original ffmpeg
 command.



 I seriously hope this is clear enough, I'm doing my best :) If someone can
 give me pointers on where to look at, I could also do some test, C is not
 my best language but I sure can dig into it if needed.

-- 
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/2445>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker