[FFmpeg-devel] [PATCH] ffmpeg: set user-set rotation for encoded streams too

[Jan Ekström]

On Sun, May 15, 2022 at 3:21 PM Jan Ekström <jeebjp at gmail.com> wrote:
>
> On Sun, May 15, 2022 at 2:58 PM Gyan Doshi <ffmpeg at gyani.pro> wrote:
> >
> >
> >
> > On 2022-05-15 04:29 pm, Jan Ekström wrote:
> > > On Thu, May 12, 2022 at 11:19 AM Anton Khirnov <anton at khirnov.net> wrote:
> > >> Quoting Gyan Doshi (2022-05-12 09:01:31)
> > >>> A substitute option for metadata rotate would accept a single rotation
> > >>> value.
> > >>> If adding an option for matrix, we should allow flips too.
> > >> I see no reason not to have both. You can have a simple option for just
> > >> the rotation and an advanced option for specifying full matrix.
> > >>
> > > FYI I had thought about arguments-to-side data earlier, and as I saw
> > > this thread I decided to make a proof-of-concept that has one layer
> > > which constructs an AVBufferRef of the side data itself based on an
> > > AVDictionary (constructor, basically), and then another which then
> > > goes over type-arguments sort of dictionary and attaches things (to an
> > > AVFrame in this case). AVPacket and AVFrame side data is slightly
> > > different, so I guess you'd have to split one layer which generates an
> > > AVBufferRef, and then that'd be utilized (discarded or not) by the
> > > utilizing function.
> > >
> > > I do wonder if AVOptions wouldn't be better for something like this,
> > > but at least it lets you play around with the idea. My initial API
> > > user ended up being the setparams filter.
> > >
> > > Link to PoC branch:
> > > https://github.com/jeeb/ffmpeg/commits/arguments_to_side_data
> > >
> > > Usage example:
> > > ffmpeg -v verbose -i INPUT -map 0:v:0 -vf
> > > setparams=side_data=displaymatrix=angle=34.34,showinfo -vframes 1 -f
> > > null -
> >
> > The metadata hack uses avutil/display API to set the matrix and then
> > attach it to the *stream* as side data  A filter will force encoding.
> > Since this is usability/display info for the consumer , it should remain
> > available for streamcopy.
>
> Yes, that is why I noted that there probably needs to be a separation
> of APIs between the AVBufferRef and the "final" one. For re-encoding a
> video filter would lead to the first AVFrame having it, and then that
> could be utilized in the encoder initialization - just like my changes
> pushing the video encoding later enabled color space information to be
> passed on. For stream copy, it should be set either to the AVPacket(s)
> or AVStream, yes. Anton talked about a bit stream filter but I have
> not looked into that side of things.
>
> This was just a proof-of-concept so people can see how string
> arguments to side data might look, since IMHO we are badly lacking
> such an interface (while the C APIs might be OK, we are clearly
> lacking stuff to do string based object generation, which is more or
> less required for CLI etc usage).

What I missed to note earlier, as we still don't have things settled
completely regarding arguments/AVOptions for constructors for side
data (not to mention the fun bits of the side data related structures
being slightly different between packets/streams and frames), it
completely makes sense to have options for specific side data to be
added to streams (or otherwise) for now.

I pushed an initial PoC of such a thing to
https://github.com/jeeb/ffmpeg/commits/ffmpeg_c_display_matrix
You can override the display matrix on both input side (which would
lead to autorotate logic etc working with it - although the side data
indeed does not get applied to the AVFrames themselves looking at the
showinfo filter's output), as well as on the output side, which only
affects the output stream specifically.

Jan