[FFmpeg-devel] [PATCH] [RFC]avformat: introduce AVStreamGroup

[Tomas Härdin]

ons 2023-09-06 klockan 11:38 -0300 skrev James Almer:
> Signed-off-by: James Almer <jamrial at gmail.com>
> ---
> This is an initial proof of concept for AVStream groups, something
> that's
> needed for quite a few existing and upcoming formats that lavf has no
> way to
> currently export. Said formats define a single video or audio stream
> composed
> by merging several individualy multiplexed streams within a media
> file.
> This is the case of HEIF, a format defining a tiled image where each
> tile is a
> separate image (either hevc, av1, etc) all of which need to be
> decoded
> individualy and then stitched together for presentation using
> container level
> information; 

I remember this blocking HEIF as a GSoC project. Honestly the way that
format is designed is immensely horrible.

> MPEG-TS programs, currently exported as
> AVProgram, which this new general purpose API would replace.

I can foresee this being a nuisance for users accustomed to AVProgram.
Also this feature borders on NLE territory. Not necessarily a bad
thing, but FFmpeg is overall poorly architectured for NLE stuff. I
believe I raised this issue back when lavfi was proposed, it being
wholly unsuitable for NLE work.


> +typedef struct AVStreamGroup {
> +    /**
> +     * A class for @ref avoptions. Set on stream creation.
> +     */
> +    const AVClass *av_class;
> +
> +    /**
> +     * Group index in AVFormatContext.
> +     */
> +    int index;
> +
> +    /**
> +     * Format-specific group ID.
> +     * decoding: set by libavformat
> +     * encoding: set by the user, replaced by libavformat if left
> unset
> +     */
> +    int id;
> +
> +    /**
> +     * Codec parameters associated with this stream group. Allocated
> and freed
> +     * by libavformat in avformat_new_stream_group() and
> avformat_free_context()
> +     * respectively.
> +     *
> +     * - demuxing: filled by libavformat on stream group creation or
> in
> +     *             avformat_find_stream_info()
> +     * - muxing: filled by the caller before avformat_write_header()
> +     */
> +    AVCodecParameters *codecpar;
> +
> +    void *priv_data;
> +
> +    /**
> +     * Number of elements in AVStreamGroup.stream_index.
> +     *
> +     * Set by av_stream_group_add_stream() and
> av_stream_group_new_stream(), must not
> +     * be modified by any other code.
> +     */
> +    int nb_stream_indexes;
> +
> +    /**
> +     * A list of indexes of streams in the group. New entries are
> created with
> +     * av_stream_group_add_stream() and
> av_stream_group_new_stream().
> +     *
> +     * - demuxing: entries are created by libavformat in
> avformat_open_input().
> +     *             If AVFMTCTX_NOHEADER is set in ctx_flags, then
> new entries may also
> +     *             appear in av_read_frame().
> +     * - muxing: entries are created by the user before
> avformat_write_header().
> +     *
> +     * Freed by libavformat in avformat_free_context().
> +     */
> +    int *stream_index;
> +} AVStreamGroup;

I see no provisions for attaching metadata, for example HEIF stitching.
Putting it in coderpar seems wrong, since it is container-level
metadata. We could just have an HEIF specific struct as container
metadata.

/Tomas