[FFmpeg-user] MSE Implementation Using ffmpeg

[Jacob Trimble]

I am trying to implement the MSE[1] API for a project.  This is a
JavaScript API that is being backed by C++ code.  The basic idea is
to create a buffer that is appended opaque (from JS perspective) data
that should be decoded and rendered in a <video> element.  I would
like to use ffmpeg (specifically libavformat) to handle demuxing of the
content.  I intend to use other methods to handle decoding frames.

[1]: https://w3c.github.io/media-source/

The end goal being to use ffmpeg to give us a bunch of demuxed frames.
It is fine if these are out of order, we can handle reordering frames.
We need to pass media segments in arbitrary order and turn that into
demuxed frames that define buffered ranges of playable regions.  When
we try to play into a region that is not buffered, we will pause the
video, waiting for more data.

I tried using a custom AVIOContext with a custom read callback to read
data from the appended buffers. This is able to demux frames so long
as the segments appear in order.  If I try to give a segment from the
past, there are errors.  First, the "current segment" doesn't get
updated when adding segments to the index; this has a simple fix.  The
other problem is that we have no way to see the new frames or to even
know if the frames exist.

We could use the AVStream::index_entries to find frames we have already
demuxed and to use seeks to get the frames that we have not received
yet; however, since we don't want to store the original data for long,
seeking will be hard to implement.  This also seems like using internals
and I would prefer a more helpful API.

First, if there is a better way to do this, I would love to hear it.
If not, we would like a way to have ffmpeg demux frames in file order,
rather than in DTS order.  The goal being to just use the MP4 parser
to pull out frames, not to play the resulting frames directly.  I
would suggest adding a flag that would allow av_read_frame to output
frames in the order they appear in the files.  This would probably
disable seeking since there is no way to predict where to seek to.  Or
maybe a different function that would act like av_read_frame which
would return in file order.

I would be happy to implement this (since I will probably need to do
this in a fork anyway).  I am wondering if there is a better way to
do it now and whether a feature like this would be accepted.

Thanks.