[FFmpeg-devel] [PATCH] ffmpeg: pass encoder init AVFrame side data to output AVStream

[Jan Ekström]

On Sun, Mar 5, 2023 at 1:17 PM Jan Ekström <jeebjp at gmail.com> wrote:
>
> This enables passing through various side data during encoding,
> which is not yet in AVCodecContext/AVCodecParameters, but is read
> from AVStream side data in muxers.
>
> Additionally, add a FATE test that demonstrates PNG->J2K MP4
> transcoding with the ICC profile getting passed through.
> ---

Just tested locally and going PNG->MP4 (RGB H.264)->PNG actually does
work with ffmpeg.c now with regards to ICC side data :) So something
like this does actually help with round-tripping (most likely since
the side data goes from the AVStream to the first AVPacket and thus
into the first AVFrame, which then gets picked up by pngenc),

Currently this is only limited to video, as audio cannot work with the
avfilter peek API, as the peeked frame gets buffered, and thus any
further adjustment of audio frame size will fail, as the buffered
frame gets output first (with whatever the original requested size
was).

>  fftools/ffmpeg.c                             |  6 ++
>  fftools/ffmpeg.h                             |  1 +
>  fftools/ffmpeg_mux.c                         | 47 +++++++++++++
>  tests/fate/ffmpeg.mak                        |  4 ++
>  tests/ref/fate/ffmpeg-side-data-to-avstreams | 72 ++++++++++++++++++++
>  5 files changed, 130 insertions(+)
>  create mode 100644 tests/ref/fate/ffmpeg-side-data-to-avstreams
>
> diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c
> index d721a5e721..134258b825 100644
> --- a/fftools/ffmpeg.c
> +++ b/fftools/ffmpeg.c
> @@ -3113,6 +3113,12 @@ static int init_output_stream_encode(OutputStream *ost, AVFrame *frame)
>                                                   av_pix_fmt_desc_get(enc_ctx->pix_fmt)->comp[0].depth);
>
>          if (frame) {
> +            if (!(ost->side_data_frame = av_frame_alloc()))
> +                return AVERROR(ENOMEM);
> +
> +            if ((ret = av_frame_copy_props(ost->side_data_frame, frame)) < 0)
> +                return ret;
> +
>              enc_ctx->color_range            = frame->color_range;
>              enc_ctx->color_primaries        = frame->color_primaries;
>              enc_ctx->color_trc              = frame->color_trc;
> diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h
> index 4d4433f5ba..21dd0f3a9e 100644
> --- a/fftools/ffmpeg.h
> +++ b/fftools/ffmpeg.h
> @@ -593,6 +593,7 @@ typedef struct OutputStream {
>      AVFrame *filtered_frame;
>      AVFrame *last_frame;
>      AVFrame *sq_frame;
> +    AVFrame *side_data_frame;
>      AVPacket *pkt;
>      int64_t last_dropped;
>      int64_t last_nb0_frames[3];
> diff --git a/fftools/ffmpeg_mux.c b/fftools/ffmpeg_mux.c
> index cf58051949..13da9ab8da 100644
> --- a/fftools/ffmpeg_mux.c
> +++ b/fftools/ffmpeg_mux.c
> @@ -580,6 +580,49 @@ static int bsf_init(MuxStream *ms)
>      return 0;
>  }
>
> +static int avframe_side_data_to_avstream(AVStream *stream, const AVFrame *frame)
> +{
> +    static const struct sd_mapping {
> +        enum AVPacketSideDataType packet;
> +        enum AVFrameSideDataType frame;
> +    } sd_list[] = {
> +        { AV_PKT_DATA_REPLAYGAIN ,                AV_FRAME_DATA_REPLAYGAIN },
> +        { AV_PKT_DATA_DISPLAYMATRIX,              AV_FRAME_DATA_DISPLAYMATRIX },
> +        { AV_PKT_DATA_SPHERICAL,                  AV_FRAME_DATA_SPHERICAL },
> +        { AV_PKT_DATA_STEREO3D,                   AV_FRAME_DATA_STEREO3D },
> +        { AV_PKT_DATA_AUDIO_SERVICE_TYPE,         AV_FRAME_DATA_AUDIO_SERVICE_TYPE },
> +        { AV_PKT_DATA_MASTERING_DISPLAY_METADATA, AV_FRAME_DATA_MASTERING_DISPLAY_METADATA },
> +        { AV_PKT_DATA_CONTENT_LIGHT_LEVEL,        AV_FRAME_DATA_CONTENT_LIGHT_LEVEL },
> +        { AV_PKT_DATA_A53_CC,                     AV_FRAME_DATA_A53_CC },
> +        { AV_PKT_DATA_ICC_PROFILE,                AV_FRAME_DATA_ICC_PROFILE },
> +        { AV_PKT_DATA_S12M_TIMECODE,              AV_FRAME_DATA_S12M_TIMECODE },
> +        { AV_PKT_DATA_DYNAMIC_HDR10_PLUS,         AV_FRAME_DATA_DYNAMIC_HDR_PLUS },
> +    };

For the record, this listing was taken from
ff_decode_frame_props_from_pkt, and probably at the end of the day
should be in a "get one from the other" or "iterate me all the
mappings" sort of API in avcodec.

Then this function adding stuff to AVStream could be in avformat
without the list being duplicated.

This thing is here in this patch mostly to prove that this kind of
passing through additional side data would help.

Jan