[FFmpeg-devel] [PATCH 01/17] avcodec/ac3enc: Don't presume ch_layout to be AV_CHANNEL_ORDER_NATIVE

[Andreas Rheinhardt]

Andreas Rheinhardt:
> It is perfectly legal for users to use a custom layout
> that is equivalent to a supported native one.
> In this case the union in AVChannelLayout is not an uint64_t mask,
> but a pointer to a custom map.
> 
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>
> ---
>  libavcodec/ac3enc.c | 6 +-----
>  1 file changed, 1 insertion(+), 5 deletions(-)
> 
> diff --git a/libavcodec/ac3enc.c b/libavcodec/ac3enc.c
> index 7a6bcf7900..dc197c1517 100644
> --- a/libavcodec/ac3enc.c
> +++ b/libavcodec/ac3enc.c
> @@ -2192,18 +2192,14 @@ av_cold int ff_ac3_encode_close(AVCodecContext *avctx)
>  static av_cold int set_channel_info(AVCodecContext *avctx)
>  {
>      AC3EncodeContext *s = avctx->priv_data;
> +    uint64_t mask = av_channel_layout_subset(&avctx->ch_layout, ~(uint64_t)0);
>      int channels = avctx->ch_layout.nb_channels;
> -    uint64_t mask = avctx->ch_layout.u.mask;
>  
>      if (channels < 1 || channels > AC3_MAX_CHANNELS)
>          return AVERROR(EINVAL);
>      if (mask > 0x7FF)
>          return AVERROR(EINVAL);
>  
> -    if (!mask)
> -        av_channel_layout_default(&avctx->ch_layout, channels);
> -    mask = avctx->ch_layout.u.mask;
> -
>      s->lfe_on       = !!(mask & AV_CH_LOW_FREQUENCY);
>      s->channels     = channels;
>      s->fbw_channels = channels - s->lfe_on;

Will apply this patchset tomorrow unless there are objections.

- Andreas