[FFmpeg-devel] [PATCH v4 6/6] decklink_enc: add support for playout of 608 captions in MOV files

[Lance Wang]

On Tue, May 2, 2023 at 10:48 PM Devin Heitmueller <
devin.heitmueller at ltnglobal.com> wrote:

> Hi Lance,
>
> On Sun, Apr 30, 2023 at 7:01 PM Lance Wang <lance.lmwang at gmail.com> wrote:
> > This implementation  is limited to decklink SDI output only,  If
> possible,
> > can we implement the function from demuxer layer,  and then passthrough
> > by SEI side data? By this way,  we can convert  such stream in streaming
> > to  embedded CC to video stream easily also.
>
> I did consider this approach, and it does raise the more fundamental
> issue about trying to minimize the number of ways we have to process
> CC data depending on whether it originated in SEI metadata or in
> separate packets.  There are a number of problems with what you are
> proposing though:
>
> 1.  There could be multiple CC streams within an MOV file but only a
> single CC stream can be embedded into AVFrame side data.  Hence you
> would have to specify some sort of argument to the demux to decide
> which stream to embed.  This makes it much more difficult to do things
> like ingest a stream with multiple CC streams and have separate
> outputs with different CC streams.  Performing the work on the output
> side allows you to use the standard "-map" mechanism to dictate which
> CC streams are routed to which outputs, and to deliver the content to
> different outputs with different CC streams.
>


> 2.  I have use cases in mind where the captions originate from sources
> other than MOV files, where the video framerate is not known (or there
> is no video at all in the source).  For example, I want to be able to
> consume video from a TS source while simultaneously demuxing an SCC or
> MCC file and sending the result in the output.  In such cases the
> correct rate control for the captions can only be implemented on the
> output side, since in such cases the SCC/MCC demux doesn't have access
> to the corresponding video stream (it won't know the video framerate,
> nor is it able to embed the captions into the AVFrame side data).
>
> I can indeed imagine there are use cases where doing it further up the
> pipeline could be useful.  For example, if you were taking in an MOV
> file and wanting to produce a TS where the captions need to be
> embedded as SEI metadata (hence you would need the e608 packets
> converted to AVFrame side data prior to reaching the encoder).
> However I don't see this as a substitute for being able to do it on
> the output side when that is the most flexible approach for those
> other use cases described above.


> Much of this comes down to the fundamental limitations of the ffmpeg
> framework related to being able to move data back/forth between data
> packets and side data.  You can't feed data packets into
> AVFilterGraphs.  You can't easily combine data from data packets into
> AVFrames carrying video (or extract side data from AVFrames to
> generate data packets), etc.  You can't use BSF filters to combine
> data from multiple inputs such as compressed video streams and data
> streams after encoding.  I've run across all these limitations over
> the years, and at this point I'm trying to take the least invasive
> approach possible that doesn't require changes to the fundamental
> frameworks for handling data packets.
>
It's worth noting that nothing you have suggested is an "either/or"
> situation.  Because caption processing is inexpensive, there isn't any
> significant overhead in having multiple AvCCFifo instances in the
> pipeline.  In other words, if you added a feature to the MOV demuxer,
> it wouldn't prevent us from running the packets through an AvCCFifo
> instance on the output side.  The patch proposed doesn't preclude you
> adding such a feature on the demux side in the future.
>
>
Thanks for the answer.  It's acceptable from my side.


Devin
>
> --
> Devin Heitmueller, Senior Software Engineer
> LTN Global Communications
> o: +1 (301) 363-1001
> w: https://ltnglobal.com  e: devin.heitmueller at ltnglobal.com
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>