[FFmpeg-user] ffmpeg architecture question

[Mark Filipak]

On 04/17/2020 02:35 PM, Ted Park wrote:
> Hi,
> 
>> "split[A][B],[A]select='eq(mod((n+1)\,5)\,3)'[C],[B]datascope,null[D],interleave"
>>
>> Though the [B][D] branch passes every frame that is presented at [B], datascope does not appear for frames 2 7 12 17 etc.
>>
>> That reveals that traversal of ffmpeg filter complexes is not recursive.
> 
> I'm pretty sure filter_complex came from "filtergraph that is not simple (complex)" rather than a complex of filters.

I think I remember reading that a filter complex is called a "filter complex" because it contains 
pads that support multiple (complex) filters to be constructed with multiple paths.

> There's no nesting filters, what context are you referring to recursion  in??

Good question, Ted. It would be very much the same as code recursion but using video frames instead 
of execution state.

With recursion, a frame that succeeds to the output (or to the input queue of a multiple input 
'collector', e.g., interleave) would be taken back to the input and tested again with other paths in 
order to (possibly) generate more successes. Essentially, with recursion, a frame could be cloned 
and 'appear' simultaneously in differing parts of the filter complex.

Without recursion, a frame that succeeds becomes essentially 'frozen' and is not 'tested' further.

It appears that frames passing through ffmpeg filter complexes 'freeze' on success and do not recurse.

> I've been trying to get to understand the 55 telecine filter script you came up with ...

The 55 telecine filter complex has only one split. The upper split, [A], passes all frames unaltered 
except 2 7 12 17 etc. It passes them unaltered because they're not combed. In contrast, the lower 
split, [B], passes frames 2 7 12 17 etc. through a 'pp=linblenddeint' filter in order to reduce 
their combing.

> ... and eliminate as many splits as possible, do you mean how the datascope wouldn't appear for the frames selected? ...

In the process of designing the filter complex, I discovered a most important property of how ffmpeg 
operates. I'm attempting to get a developer (Paul?) to confirm whether what I've seen is 
non-recursion or whether I'm mistaken.

> ... Same timestamps might be the issue again, maybe setpts=PTS+1 would make them show up? ...

Again, you have zoomed in on a very important point: When (and where in the filter complex) does 
ffmpeg assign frame numbers? I would guess that ffmpeg assigns frame numbers (0 1 2 3...) at the 
input, based on the PTSs of the arriving frames. The alternative would be that ffmpeg defers 
assigning the 'next' frame number (n+1) to a frame only when the frame succeeds to the output (or a 
queue), but I don't think that's what ffmpeg does. Again, this is a very important architectural 
detail, and I seek confirmation from a developer (Paul?). The reason I think that ffmpeg *does not* 
defer is that deferring would turn fixed modulo 'n' frame calculations into variables (that would be 
insanely difficult to determine in advance).

> ... Or does interleave take identical dts input and order them according to some rule?

Yes, there are rules regarding PTSs & DTSs regarding whether the frame is a B-frame or not, and, I 
think, whether PTS & DTS are within certain timing margins (I recall 100 micro-seconds but that may 
be wrong ...it's not really important in any event).