[FFmpeg-devel] GSoC 2016 own ideas

[Gerion Entrup]

On Donnerstag, 18. Februar 2016 21:02:09 CET wm4 wrote:
> On Thu, 18 Feb 2016 20:41:29 +0100
> 
> Gerion Entrup <gerion.entrup.ffdev at flump.de> wrote:
> > On Donnerstag, 18. Februar 2016 20:10:47 CET wm4 wrote:
> > > On Thu, 18 Feb 2016 18:27:45 +0100
> > > 
> > > Gerion Entrup <gerion.entrup.ffdev at flump.de> wrote:
> > > > Good day,
> > > > 
> > > > I'm a master student and long term FFmpeg-user. I want to participate
> > > > in
> > 
> > the GSoC 2016 for FFmpeg. The reason, I write this, is that I want to
> > suggest some own ideas. It could be, that some of the mentioned things
> > are wrong (because FFmpeg could do this already or it it much more
> > difficult than I think). Please correct me if so.
> > 
> > > > I'm excited to hear, what do you think about this ideas, if you think,
> > 
> > they are suitable for GSoC and if you maybe are willing to mentor some of
> > this.
> > 
> > > > 1. *disc support
> > > > 
> > > > What current FFmpeg could do:
> > > > - Bluray support through protocol support with prefix "bluray:" and
> > 
> > libbluray. The implemention is some kind of limited. There is missing
> > chapter support. The "main movie" selection is simply the longest one.
> > Menus are not supported. Afaik no metadata are transfered.
> > 
> > > > - CD support through libavdevice and libcdio. Last time I tried it, it
> > 
> > only works under some strange circumstances (I have to set the parameter
> > "-ss 0"). I don't know if it is fixed right now. It recognizes a CD as
> > one track with multiple chapters. Afaik CD-Text is not supported.
> > 
> > > > - No DVD support at all. I saw, that Stefano ones write a patch [1]
> > 
> > (protocol and libdvdnav based), which seems to work in principal, but was
> > probably not ready to merge.
> > 
> > > > Goal:
> > > > - Implement an uniform solution for *disc support, that also support
> > 
> > metadata.
> > 
> > > > - Maybe (?): Add some kind of Menu Support, FFmpeg is not made for
> > > > this
> > 
> > atm, I think. Not sure how difficult it is.
> > 
> > > > - Maybe (?): Try to change something on libdvdnav/libdvdread to make
> > > > it
> > 
> > more consistent (to libbluray ?). Similar idea from Flameeyes [2].
> > 
> > > Hm, I'm not sure if that's even possible in a reasonable way. Full
> > > DVD/Bluray support requires menus, which are hard to get right, and
> > > which would require non-trivial interactions between applications and
> > > libavformat.
> > > 
> > > Improving support for merely transcoding a DVD to a file might be not
> > > so hard and potentially a good idea, though.
> > > 
> > > > Qualification task:
> > > > - Maybe (?): Add chapter support for the current Bluray implementation
> > 
> > (get the chapters out of libbluray is fairly simple, I'm not sure how
> > difficult it is to get them into the stream).
> > 
> > > This sounds like a hard design issue: currently bluray is implemented
> > > as avio protocol, which can not pass things like chapters to the higher
> > > level (demuxer). This would require coming up with a mechanism for
> > > transferring this metadata, which sounds hard. Just reading the bluray
> > > chapters should indeed be simple.
> > 
> > Just reading the chapters and print them is too simple to be a
> > qualification task.
> 
> Indeed.
> 
> By the way, your mail client is making it very hard to distinguish
> quoted text from your replies. It doesn't add a ">" on the next line
> when line-breaking the quoted text.
I see it myself in the received mail. I found out, what causes it. It is 
clearly a bug, but I hopefully can circumvent it, sorry.

> 
> > > > Comments:
> > > > - I would try to orientate on VLC, mplayer and mpv implementation.
> > > > - libdvdread/nav seem to do much less than libbluray, so the main work
> > 
> > would be the dvd support, I guess.
> > 
> > > > - All kinds of discs normally contains more than one track. My idea
> > > > for a
> > 
> > solution would be, that ffmpeg recognize the disc as multiinput and choose
> > the mainmovie as default.
> > 
> > > > 2. treelike metadata
> > > > 
> > > > What current FFmpeg could do:
> > > > - Metadata is always an AVDictionary (string based afaik).
> > > > - Some metadata systems like the one of Matroska use a treelike
> > > > structure
> > 
> > (xml in mkv), so you can specify the charatername given an actor or define
> > more than on actor, see example at the end of the mail.
> > 
> > > > Goal:
> > > > Rewrite the FFmpeg metadata system, to support extended tags. In
> > > > Detail:
> > > > - Support for more than one value per key.
> > > > - Support for more key/value pairs as value.
> > > > 
> > > > Qualification task:
> > > > No idea.
> > > > 
> > > > Comments:
> > > > Not sure, if this task is GSoC suitable.
> > > 
> > > Includes API design, which is better avoided for a gsoc. I'm also not
> > > sure if we really need such metadata, but I guess this is debatable.
> > > 
> > > > 3. Vector graphic support
> > > > 
> > > > What current FFmpeg could do:
> > > > - No support at all, afaik.
> > > > 
> > > > Goal:
> > > > - Extend FFmpeg to support vector graphics (without converting to
> > > > pixel
> > 
> > graphics).
> > 
> > > > - Write an SVG decoder for this.
> > > > - Write an vector to pixel image converter/filter.
> > > 
> > > That sounds not only very hard, but also very out of scope for ffmpeg.
> > > The problem with SVG is that nothing seems to support it correctly
> > :
> > :)
> 
> It's the truth. Support for standard SVG features is very inconsistent.
> 
> > > and
> > > entirely, so it's a good way to get hurt.
> > 
> > Ok, thank you for the evaluation.
> > 
> > > > Qualification task:
> > > > No idea.
> > > > 
> > > > Comments:
> > > > I think, this could be a very hard task. Could you comment, whether
> > > > you
> > 
> > find this suitable? I have no experience with SVG. Maybe it could be based
> > on cairo.
> > 
> > > > 4. Usable analyse filter output for programs
> > > > Not sure, if this is a real problem at all. Not sure if this is
> > > > suitable
> > 
> > for GSoC at all.
> > 
> > > > What current FFmpeg could do:
> > > > - Receive (multiple) streams in filter and write back other streams.
> > > > - The problem I see here, is that the output of analyse filters is
> > > > hard to
> > 
> > process (needs some kind of subprocess and extended usage of grep/sed).
> > 
> > > >     Examples: All filters that analyse some thing and then "print" it
> > > >     to
> > 
> > stdout, like cropdetect, volumedetect, blackframe, .... Some (not
> > implemented filter) like tonality, bpm detection, waveform analysis, ...
> > 
> > > > Goal:
> > > > - Rewrite the filtersystem, so that some function "receive_output"
> > > > exists,
> > 
> > that could be called from the external API or another filter to get non
> > multimedia data as the filter output, like an String for tonality, or an
> > index array for the wavform, etc.
> > 
> > > I'm not sure if I'm following. Currently, filters like cropdetect
> > > pass the metadata via AVFrame (as separate field besides audio/video
> > > data), and other filters (or the API user) can read it.
> > 
> > Hmm, ok I see it. If you look at my example from my other mail with
> > volumedetect, is it correct to say it it possible but depends on the
> > filter, whether it print "just" something out or include it in some tags?
> > If so, this whole idea is not necessary.
> 
> Yes, it depends on the filter. But maybe there's indeed a need to pass
> certain non-audio/video information between filters.
> 
> > > > Comments:
> > > > - Has the side effect, that a following filter in the chain could use
> > > > the
> > 
> > output to do some other stuff.
> > 
> > > > - Not sure, how hard (and wanted) this is.
> > > > 
> > > > 
> > > > 
> > > > Other ideas, but imho not suitable for GSoC:
> > > > - Jack output support (too simple for GSoC)
> > > > - Python bindings
> > > 
> > > Bindings for what?
> > 
> > For the libraries. Some python code, so that one can use the libraries
> > directly with python. It is questionable whether this belongs here or is a
> > seperate project.
> > Moreover such a project exists, but it is outdated and I have no idea
> > about
> > the quality:
> > https://mhaller.github.io/pyffmpeg/
> 
> Hm, not sure if this is enough or appropriate. The ffmpeg API is also
> moving quickly. It also requires deep knowledge of the API to get it
> right. Not a beginner task maybe.
I think, this is the reason, why the project is unmaintained. Missing 
knowledge was the reason I write it in the "probably not suitable" list.

> 
> > > > - merge of ffmpeg and ffplay (call ffmpeg without output reacts like
> > 
> > ffplay). Maybe this idea is fairly naive.
> > 
> > > That sounds hard, and maybe not really what we'd want anyway. (Updating
> > > ffplay with SDL2 or something sounds more realistic.)
> > 
> > A question for that. If I get it right, ffplay use SDL directly. It is
> > wanted to "swap it out" to some kind of libavdevice and then use this
> > device out of ffplay?
> 
> In my personal opinion, audio/video output in libavdevice is an
> abomination. It tries to force a muxer API to be used as API for
> audio/video output, which is ridiculous and obviously has a bunch of
> disadvantages. A real approach would realize such things as a separate
> library with a proper API.
You mean a library inside FFmpeg but not as connected at the one atm?
I guess, this would be far too hard for GSoC, right?

> 
> Coming up with good ideas is hard, sorry for not being able to provide
> any.
> 
> What kind of topics would you be interested in? Just broadly speaking.
This is not a simple question. The mentioned things are things, I directly 
miss in ffmpeg.

Generally my interests are maybe connect more things to FFmpeg (DVD, jack, 
python, SVG are such kind of features).

And another aspect in the mentioned proposals are that with my current 
unterstanding most of them do not fit completely in the current design of 
FFmpeg, so implementing such a thing would maybe help to understand the design 
much better. But maybe this is not a suitable thing in the scope of GSoC.