[FFmpeg-devel] [PATCH] [RFC]avformat: introduce AVStreamGroup

[James Almer]

On 9/13/2023 6:34 AM, Tomas Härdin wrote:
> ons 2023-09-06 klockan 16:16 -0300 skrev James Almer:
>> On 9/6/2023 2:53 PM, Tomas Härdin wrote:
>>> ons 2023-09-06 klockan 11:38 -0300 skrev James Almer:
>>>> Signed-off-by: James Almer <jamrial at gmail.com>
>>>> ---
>>>> This is an initial proof of concept for AVStream groups,
>>>> something
>>>> that's
>>>> needed for quite a few existing and upcoming formats that lavf
>>>> has no
>>>> way to
>>>> currently export. Said formats define a single video or audio
>>>> stream
>>>> composed
>>>> by merging several individualy multiplexed streams within a media
>>>> file.
>>>> This is the case of HEIF, a format defining a tiled image where
>>>> each
>>>> tile is a
>>>> separate image (either hevc, av1, etc) all of which need to be
>>>> decoded
>>>> individualy and then stitched together for presentation using
>>>> container level
>>>> information;
>>>
>>> I remember this blocking HEIF as a GSoC project. Honestly the way
>>> that
>>> format is designed is immensely horrible.
>>>
>>>> MPEG-TS programs, currently exported as
>>>> AVProgram, which this new general purpose API would replace.
>>>
>>> I can foresee this being a nuisance for users accustomed to
>>> AVProgram.
>>> Also this feature borders on NLE territory. Not necessarily a bad
>>> thing, but FFmpeg is overall poorly architectured for NLE stuff. I
>>> believe I raised this issue back when lavfi was proposed, it being
>>> wholly unsuitable for NLE work.
>>>
>>>
>>>> +typedef struct AVStreamGroup {
>>>> +    /**
>>>> +     * A class for @ref avoptions. Set on stream creation.
>>>> +     */
>>>> +    const AVClass *av_class;
>>>> +
>>>> +    /**
>>>> +     * Group index in AVFormatContext.
>>>> +     */
>>>> +    int index;
>>>> +
>>>> +    /**
>>>> +     * Format-specific group ID.
>>>> +     * decoding: set by libavformat
>>>> +     * encoding: set by the user, replaced by libavformat if
>>>> left
>>>> unset
>>>> +     */
>>>> +    int id;
>>>> +
>>>> +    /**
>>>> +     * Codec parameters associated with this stream group.
>>>> Allocated
>>>> and freed
>>>> +     * by libavformat in avformat_new_stream_group() and
>>>> avformat_free_context()
>>>> +     * respectively.
>>>> +     *
>>>> +     * - demuxing: filled by libavformat on stream group
>>>> creation or
>>>> in
>>>> +     *             avformat_find_stream_info()
>>>> +     * - muxing: filled by the caller before
>>>> avformat_write_header()
>>>> +     */
>>>> +    AVCodecParameters *codecpar;
>>>> +
>>>> +    void *priv_data;
>>>> +
>>>> +    /**
>>>> +     * Number of elements in AVStreamGroup.stream_index.
>>>> +     *
>>>> +     * Set by av_stream_group_add_stream() and
>>>> av_stream_group_new_stream(), must not
>>>> +     * be modified by any other code.
>>>> +     */
>>>> +    int nb_stream_indexes;
>>>> +
>>>> +    /**
>>>> +     * A list of indexes of streams in the group. New entries
>>>> are
>>>> created with
>>>> +     * av_stream_group_add_stream() and
>>>> av_stream_group_new_stream().
>>>> +     *
>>>> +     * - demuxing: entries are created by libavformat in
>>>> avformat_open_input().
>>>> +     *             If AVFMTCTX_NOHEADER is set in ctx_flags,
>>>> then
>>>> new entries may also
>>>> +     *             appear in av_read_frame().
>>>> +     * - muxing: entries are created by the user before
>>>> avformat_write_header().
>>>> +     *
>>>> +     * Freed by libavformat in avformat_free_context().
>>>> +     */
>>>> +    int *stream_index;
>>>> +} AVStreamGroup;
>>>
>>> I see no provisions for attaching metadata, for example HEIF
>>> stitching.
>>> Putting it in coderpar seems wrong, since it is container-level
>>> metadata. We could just have an HEIF specific struct as container
>>> metadata.
>>
>> The doxy for AVCodecParameters says "This struct describes the
>> properties of an encoded stream.", so It's not about container level
>> props.
> 
> It *is* container level props. The underlying codecs have no concept of
> this kind of stitching. The closest you're going to get is tiles in
> JPEG2000, but I doubt HEIF support JPEG2000.
> 
> We might say "well the resulting stream group has resolution so it's
> like a codec" but see below.
> 
>> Although codecpar will be used to export the merged/stitched stream
>> props like dimensions and channel layout, maybe you're right about
>> the
>> metadata because there would be a clash between actual
>> HEVC/Opus/AAC/AV1
>> extradata and the HEIF/IAMF/etc specific info if both use
>> codecpar.extradata, even if one will be in AVStream and the other in
>> AVStreamGroup.
> 
> Yes, pretty much. But it's more that codecpar is pressed into service
> where it probably doesn't belong. It might be more appropriate to call
> these "essence parameters". I'm going to stick my neck out further and
> say that picture and sound essence should be handled with different
> structs, not smushed together into one struct like AVCodecParameters.

Can you suggest how to approach this then, if not with 
AVCodecParameters? For the resulting merged/stitched stream, we need at 
the very least dimensions, and for audio we need channel layout, and for 
each different kind of group type (HEIF, IAMF, TS programs) we need 
specific parameters, like order of tiles for HEIF, mixing parameters for 
IAMF, and these parameters should be in a form easy for the user (like 
our CLI) to feed to lavfi, where the actual merging/stitching would take 
place with new or existing filters for this specific purpose.

Maybe something like:

----
enum AVStreamGroupParamsType {
     AV_STREAM_GROUP_PARAMS_NONE,
     AV_STREAM_GROUP_PARAMS_TS,
     AV_STREAM_GROUP_PARAMS_HEIF,
     AV_STREAM_GROUP_PARAMS_IAMF,
};

typedef struct AVStreamGroupTSParams {
     // Basically AVProgram
} AVStreamGroupTSParams;

typedef struct AVStreamGroupHEIFParams {
     // dimensions, tile order, etc
} AVStreamGroupHEIFParams;

typedef struct AVStreamGroupIAMFParams {
     // channel layout, mixing params
} AVStreamGroupIAMFParams;

typedef struct AVStreamGroup {
     [...]
     enum AVStreamGroupParamsType type;
     union {
         AVStreamGroupTSParams ts;
         AVStreamGroupHEIFParams heif;
         AVStreamGroupIAMFParams iamf;
     } essence;
     [...]
} AVStreamGroup;
----