[FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image classification using CLIP models

[Guo, Yejun]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> m.kaindl0208 at gmail.com
> Sent: Tuesday, February 18, 2025 8:17 PM
> To: 'FFmpeg development discussions and patches' <ffmpeg-
> devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image
> classification using CLIP models
> 
> The new backend is an extension of the existing Torch backend rather than a
> separate implementation.
In general, we don't need to add a new backend for libtorch for another use case.

> 
> Inference in CLIP differs from other models as it encodes (embeds) both
> images and tokenized text labels, then calculates the similarity between the
> encoded vectors. As a result, its forward pass takes two inputs and produces
> two outputs.
Is the tokenizer expected to be used by other dnn filters? We may put it in files
like libavfilter / dnn_filter_common.c

> 
> To ensure clarity and modularity, I have created a separate
> dnn_torch_backend_clip file instead of expanding dnn_torch_backend. This
> keeps the main file manageable and allows for easy exclusion from the build
> when the tokenizer-cpp library is not included.
> 
> If preferred, I can implement a standalone tokenizer class, integrate it into e.g.
> libavutil, and move the remaining code to the backend.
> 
> -----Ursprüngliche Nachricht-----
> Von: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> Im Auftrag von Guo,
> Yejun
> Gesendet: Tuesday, 18 February 2025 11:09
> An: FFmpeg development discussions and patches <ffmpeg-
> devel at ffmpeg.org>
> Betreff: Re: [FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image
> classification using CLIP models
> 
> 
> 
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> > Maximilian Kaindl
> > Sent: Tuesday, February 18, 2025 12:29 AM
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel at ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image
> > classification using CLIP models
> >
> > Hello Yejun Guo,
> >
> > yes i can do that and submit it in another patch. Do you also have
> > some feedback for the clip backend? I have already made some small
> > changes (cuda accel and new preprocessing) that i will submit along
> > with the other patch, but i would like to hear your thoughts.
> >
> Could you share why we need a new backend?
> 
> > Thanks
> >
> > ________________________________
> > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> on behalf of Guo,
> > Yejun <yejun.guo-at-intel.com at ffmpeg.org>
> > Sent: Sunday, February 16, 2025 7:09 AM
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel at ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image
> > classification using CLIP models
> >
> >
> >
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> > > m.kaindl0208 at gmail.com
> > > Sent: Thursday, January 30, 2025 4:33 AM
> > > To: ffmpeg-devel at ffmpeg.org
> > > Subject: [FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image
> > > classification using CLIP models
> > >
> > > Add a new filter 'dnn_clip' that performs zero-shot image
> > > classification using CLIP (Contrastive Language-Image Pre-Training)
> models.
> > The filter supports:
> >
> > For image classification with new dnn models, we'd better add the new
> > model support with dnn_classify at https://ffmpeg.org/ffmpeg-
> > filters.html#dnn_005fclassify
> >
> >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email ffmpeg-devel-request at ffmpeg.org
> with subject "unsubscribe".
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email ffmpeg-devel-request at ffmpeg.org
> with subject "unsubscribe".