[FFmpeg-user] Programmatically detecting 'busiest' parts of a video

[Rob Hallam]

Thanks for chiming in, inhahe.

On Tue, 6 Aug 2024 at 16:20, inhahe <inhahe at gmail.com> wrote:
>
> On Tue, Aug 6, 2024 at 8:41 AM Rob Hallam <ffmpeg at roberthallam.com> wrote:
> > I'd like to programmatically detect the 'busiest' parts of a video- ie
> > the most visually active areas. I am leaving audio aside for the
> > purposes of considering this.
> >
> > I figured it might be possible by looking at one / more of:
> >
> >  - the bitrate for VBR videos -- a higher bitrate for a given segment
> > will tend to be associated with more activity (other things being
> > equal)
> >  - frame differences -- count pixels/blocks which differ, average over
> > a time segment
> >  - optical flow? (higher flow values = more activity) I'm not too
> > familiar with this

> The second thing I thought of was your second suggestion,
> but ... [snip] ... I wouldn't try this way.

Agreed.

> I just looked up "optical flow," and it seems to be more or less a
> description of why the second approach wouldn't work,

Yeah that's fair enough.

I am hoping there might be some relatively quick way of working out
changes, motion, flow (etc), since encoders would have to do this
anyway.

I mean, it is possible to re-encode a CBR video as VBR+constant
perceptual quality and then do the same bitrate segmentation, but that
approach seems very inefficient!

Part of the trouble is I know only a little about the techniques of
working with videos in this way, so I'm not sure of the unknown
unknowns / vulnerable to the anosognosic's dilemma [1] [2].

I suspect there's a better approach that I'm not seeing.

Cheers,
Rob

[1]: see eg https://archive.nytimes.com/opinionator.blogs.nytimes.com/2010/06/20/the-anosognosics-dilemma-1/
(thanks Leo Butler)
[2]: mentioned on-list here
https://ffmpeg.org/pipermail/ffmpeg-user/2023-November/057183.html