[FFmpeg-devel] [PATCH][RFC] avcodec/avutil: Add timeline side data

Michael Niedermayer michael at niedermayer.cc
Mon Apr 2 00:38:33 EEST 2018

On Sun, Apr 01, 2018 at 03:39:07PM +0100, Derek Buitenhuis wrote:
> On 4/1/2018 12:44 AM, Michael Niedermayer wrote:
> >> Not sure I follow what this has to do with timelines? There is no format that
> >> exists that store timeline data interleaved, as far as I know - it is a
> >> purely theoretical scenario.
> > 
> > that surprises me. But if this case never occurs (or will occur) thats a good
> > thing. Fewer cases to consider.
> At least as far as I'm aware, yes.
> >> What do you have in mind with regards to 'sharing'? I do not particularly
> >> agree that chapters and edits should share stuff because they both happen to have
> >> start and end times. They're quite different things, even if some formats have
> >> conflated them (design flaw IMO).
> > 
> > I did not think deeply about this but the obvious 3 options are to either
> > give each its own struct and align the APIs used to access them
> > 
> > to show this just with one function:
> > AVTimelineList         *av_timeline_list_alloc(size_t timeline_count);
> > AVChapterList          *av_chapterlist_alloc(size_t chapter_count);
> > AVTimelineMetadataList *av_timeline_metadata_list_alloc(size_t metadata_count);
> > AVTimelineCodecList    *av_timeline_codec_list_alloc(size_t codec_count);
> > 
> > or to have some more common struct maybe similar to side data. It is after all
> > some kind of side/meta data that is specific to timespans.
> > 
> > or to put all in AVFormatContext as normal allocated arrays like chapters are
> > currently.
> > 
> > What i find a bit ugly is that each API uses the style popular at the time
> > at which it is added and they are so far each completetly different even though
> > they are semantically handling the same "unit" of data. that is information
> > about "timespans".
> > 
> > We have AVChapters which are a public field in AVFormatContext
> > There is AV_PKT_DATA_STRINGS_METADATA which is a binary specified side data and
> > there is the newly proposed timeline which is opaque sidedata with accessor functions
> > 
> > Its not the inherent differences of the data that i want to push into a square hole
> > but all the common parts that could be the same and are very different.
> > If 2 things dont fit together iam certainly not intending to suggest to put them
> > together. Nor that 2 things that have reason to be behind differnt APIs should be
> > not.
> > 
> > I mainly wanted to point out that our APIs for these things are quite
> > different and that this should be thought about when adding a new API.
> > Its not a specific cleanup or change i have in mind, more so i wanted to
> > call attention to the inconsistency that there is between APIs
> > so it is considered when adding a new API.
> > If you determine that theres nothing that should be changed from what you
> > intended previously, thats perfectly fine with me.
> I agree that the APIs are annoyingly different, but that they should remain
> separate APIs. The suggestion of aligned (same names, args, etc.) seems the
> most reasonable to me - consistency.
> I don't really think they can share some common structs, but they can certainly
> be aligned.

> However, some of the stuff you listed doesn't make sense to me:
> * Codec changes are not part of timeline data; they're entirely separate things
>   in every format I'm aware of. In MPEG-TS it just changes, in MOV/MP4 it is done
>   via multiple 'stsd' atoms/boxes, which don't really carry any time related
>   data - there is another atom/box that maps samples to a particular 'stsd',
>   and that's it.

yes but in an abstract way each timespan has one codec
its the same kind of "data structure" even if its not stored this way in 

> * What is AVTimelineMetadataList meant to be?


> Lastly, what is the intent here? Is it to align all of these before timeline
> support, or to deprecate and align after? Right now the API is align with
> other side data APIs. I need to know which scenario this is before I go
> ahead and implement this.

Part of the intend is to have a easier to understand API where things follow
similar design.
Another part is "compatibility". Ideally a format like mpegts that just
changes codecs should fill whatever data structure we use for codec changes
in the future. And simply passing these structures over to a mov/mp4 muxer
would make it write stsd accordingly.
It would be suboptimal for example if these 2 would use different ways to
export this codec change data at least unless we also provide code to
I didnt now check what exactly is stored in it currently but,
We do with AV_PKT_DATA_STRINGS_METADATA, the chapter array (which also has
metadata in it) possibly already something that goes a bit in that direction

Its absolutely not my intend to suggest that other APIs would need to
be changed before. Much more to bring the current incosistent APIs to people
attention so it is given a thought when a new API is added.
deprecating APIs and replacing them also has a cost for our users ...


Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Frequently ignored answer#1 FFmpeg bugs should be sent to our bugtracker. User
questions about the command line tools should be sent to the ffmpeg-user ML.
And questions about how to use libav* should be sent to the libav-user ML.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180401/e78554c0/attachment.sig>

More information about the ffmpeg-devel mailing list