[FFmpeg-devel] [RFC] Talk about subtitles
ubitux at gmail.com
Tue Nov 22 19:56:41 CET 2011
[TL;DR: I want to improve/rewrite a real subtitles support in FFmpeg and need
Today again someone asked how to burn subtitles in a video. This is a recurrent
question which I think need to be solved now: some people still stick to the
unmaintained mencoder, try some pipes with MPlayer, or some lost code in
mplayer2, etc. And still nothing user friendly for this. Considering this, I
really wish to improve the situation with FFmpeg.
But before we make the same mistakes we already did, I'd like to discuss the
matter here, and share various issues with subtitles we have to considerate
So first, what are the "required" features?
- decode subtitles: we need a bitmap we can blit on a surface at a given
position, and eventually a purified text version (so it could be printed to
stdout) for a given pts.
- convert subtitles: ATM we have mixed demuxer/decoder/encoder/muxer for
SubRip, MicroDVD (and ASS right?). The question about how we will keep or
make use of this will be discussed later.
- we need to be able to support subtitles from muxed files and external files.
What could go wrong? Actually, a lot of things. Let me enumerate a few issues we
can have (and already have):
- ASS/SSA: AFAIK we use internally a slightly modified representation of ASS to
store all the different subtitles tracks. While now ASS is used to render
almost everything, it might not be the perfect approach because it can't
store the information of all the subtitles format (such as pts precision).
See next points for various related headaches.
- VOBSUB: this is basically a stream of bitmap subtitles + another file for
indexes; we need to be able to support multiple inputs. Also note such
subtitles are sometimes store in an archive...
- Jacosub: this is somewhat an ancestor of ASS/SSA, and it's also a crazy
format. If we do things properly, we might want to support it (so mplayer
could for instance use FFmpeg internals to replace its own old subtitles
support, which includes a partial jacosub support). Note that this format
supports the *include* of external *images*. This is also something we need
to care about in the internal struct. I already wrote a demuxer a while ago,
and it could be used.
- SubRip and MicroDVD: the *text* of these formats (excluding time
representation then) can easily be converted to ASS markup. This has already
been done by Aurelien in FFmpeg, and I also did it a while ago for MPlayer.
This can be re-used, but writing them raises an issue: some crazy people mix
the format markup with the ASS markup (you have ASS tags in the middle of
SubRip specific tags for instance) because some players just pass them to the
ASS library without any escaping. The current behaviour in mplayer2 is to
escape those ASS tags while converting so they are displayed verbatim, but
this might need to be configurable. So this is another thing to care about:
we need to be able to pass options to the new API.
- SAMI subtitles: this is HTML-only-valid-with-iexplorer5 crap, with CSS and
such. Converting them to ASS might be difficult but possible. Though, parsing
HTML is a pain: since it's not the only existing XML-like subtitles, we might
need to have subtitles with a dependency to a random HTML/XML library...
- Closed caption and teletext: I don't know them enough, but we also need to
support them. If anyone can comment on this...
This is all I can think about ATM. So if we want to support this properly, I can
only think of this: libavsubtitles (with a dependency to libass to render a lot
of crazy markup in subtitles).
This library will be used to demux/decode/etc (for conversion purpose too) and
render the subtitles. And more importantly, we would be able to use it to burn
them through the libavfilter with something like vf_subtitles.
There is ATM a pending patch (vf_ass) that can workaround the situation for most
of the people: a lot of software allow the convert from srt to ass, so it won't
be needed to duplicate the srt to ass convert code yet another time for the
filter. Users will be satisfied enough for a while I think with a simple ASS
burning filter. It will give us some time to think about a proper way to write
and integrate libavsubtitles.
I already see people asking why we should support subtitles burning, so before
there are some flames about this:
- some old divx players doesn't support them
- some old config can not render ASS fast enough
- most of the config can not render ASS fast enough with complicated karaoke
- a lot of portable device don't have a subtitle support good enough
Those situations are common and need to be solved.
If anyone wants to work on this, please raise your hand. Also, if people working
on video players (Uoti? Reimar?) could comment on the way they would like to use
the API, please speak up! Nothing is really started ATM, and I'd like to work on
something where everyone could agree.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 490 bytes
Desc: not available
More information about the ffmpeg-devel