[FFmpeg-devel] Designing a subtitles infrastructure (was: DVBSUBS and TXTSUBS)

Nicolas George nicolas.george at normalesup.org
Thu May 24 13:40:11 CEST 2012

Le quartidi 4 prairial, an CCXX, Clément Bœsch a écrit :
> Now that things seem to calm down with the audio in lavfi, we might want
> to consider working again on subtitles to do the burning in lavfi (and
> various other changes).

That would be nice. As I said in the Trac ticket you were pointing at, the
hard part with subtitles is that they are a sparse stream. For example, take
the following SRT snippet:

00:00:01,000 --> 00:00:02,000
This is the beginning.

00:01:01,000 --> 00:01:02,000
This is the end.

Mux it with a big video, and then:

./ffmpeg_g -i video+sub.mkv -i audio.ogg \
  -f matroska -vcodec libx264 -preset ultrafast -crf 0 -y /dev/null

... and watch ffmpeg eating all the memory.

The reason for that is this: ffmpeg always want a packet from the oldest
stream. After two seconds of encoding, the oldest stream is the subtitles
stream, so it will read from the video+sub file, and get a video packet. The
video packets will be stuck in the muxing infrastructure waiting for audio
packet for proper interleaving on output. Unfortunately, the sub stream is
still the oldest, so ffmpeg does not try to read on the audio file.

We will have the same issue with subtitles filtering:

	[video] [sub] rendersub

will request_frame() on its "sub" input; if the inputs are connected to the
same file, it will get a frame on its "video" input instead, and need to
queue it. And this time, this is decoded video, it is even larger than

IMHO, we will need to assume that files with subtitles in them are properly
interleaved, with a maximum delta: if we get a video or audio packet at PTS,
and there was no subtitle packet recently, we can emit a dummy "nothing new"
subtitles packet at PTS-epsilon.


  Nicolas George
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20120524/4c8e802e/attachment.asc>

More information about the ffmpeg-devel mailing list