[FFmpeg-devel] [PATCH v20 02/20] avutil/frame: Prepare AVFrame for subtitle handling

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Hendrik
> Leppkes
> Sent: Monday, December 6, 2021 1:37 AM
> To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH v20 02/20] avutil/frame: Prepare AVFrame
> for subtitle handling
> 
> On Mon, Dec 6, 2021 at 1:24 AM Soft Works <softworkz at hotmail.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Hendrik
> > > Leppkes
> > > Sent: Monday, December 6, 2021 12:37 AM
> > > To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> > > Subject: Re: [FFmpeg-devel] [PATCH v20 02/20] avutil/frame: Prepare
> AVFrame
> > > for subtitle handling
> > >
> > > On Mon, Dec 6, 2021 at 12:23 AM Soft Works <softworkz at hotmail.com> wrote:
> > > >
> > > > It can only work like that.
> > > >
> > >
> > > You literally designed subtitle filtering. You defined how it works.
> > > All your explanations come down to some fancy form of "because I made
> > > it required".
> > >
> > > All I'm hearing is that some internal mechanisms in avfilter need it.
> > > Why is it a public field in one of the most key structures in our
> > > entire codebase? You understand this concern, right? Any public field
> > > should first (and only) answer the question, what information does
> > > this convey to the user? Does the user need this information? I'm not
> > > seeing any of that in your explanations. All I'm seeing is internal
> > > avfilter workings - cemented in public API.
> > >
> > > You are not integrating into an existing infrastructure, you are
> > > literally trying to add the whole thing, so you can change it to not
> > > need this redundant field.
> > >
> > > - Hendrik
> >
> > OK, maybe this is better to understand. Very simple burn-down:
> >
> > I send a subtitle frame into a filtergraph.
> >
> > pts = 10
> > subtitle_pts = 10
> >
> > Now I need to re-send a duplicate of that frame into the graph
> > (=> heartbeat - to keep the graph flowing)
> >
> > Now, it's not possible (by lavf design, not mine) to send another frame
> > without increasing the pts of the frame. So we send it like this:
> >
> > pts = 11
> > subtitle_pts = 10
> >
> > The pts needs to be increased. But the start-display time of the
> > subtitle event must not be changed - it's still the same frame like
> > before, just a duplicate.
> >
> 
> You just keep re-iterating that avfilter internals need it. And I'll
> just keep saying that avfilter internals shall not define public API
> design. I have not heard anything that justifies yet another field
> that has no benefit to the external public API - to the contrary even,
> a user would assume pts is pts, and he would be wrong.

About which use cases of the public API are we talking about actually?

Applications similar to ffmpeg CLI?
Applications that want to do subtitle filtering?
Applications that work with subtitle AVFrames?

=> The heartbeat mechanism is an intrinsic requirement for performing
   subtitle filtering. Whether you want to send frames into a filtergraph
   or process subtitle frames coming out of a filtergraph: such applications
   will often need to deal with both fields, and when just one, then it might
   be the one or the other, depending on use case.

ffmpeg cli needs to deal with both of these fields. That's already a good
indication that other consumers will need the same.
Essentially, this is not an implementation detail that needs to be hidden, 
it's an important element of subtitle filtering functionality.

I have tried to explain the functionality and importance of having these
fields separate in detail and by examples. And it's not only the use of 
those fields in the current code. I have quite a number of future use
cases in mind, and many of them wouldn't be possible with the duality 
of those fields.

I admit that the meaning of those fields in a subtitle context and their 
relation to each other might not be easy to understand immediately, but 
that's just one tiny bit among many others in case of ffmpeg and surely
not among the worst.

Maybe some improvement could be achieved by things like renaming or clarifying
doc texts, I'm open to suggestions.

Kind regards,
softworkz