[FFmpeg-devel] [RFC] AAC Encoder

Kostya kostya.shishkov
Sat Aug 16 17:00:39 CEST 2008


On Sat, Aug 16, 2008 at 03:57:56AM +0200, Michael Niedermayer wrote:
> On Fri, Aug 15, 2008 at 07:59:52PM +0300, Kostya wrote:
> [...]
> 
> > /**
> >  * Set window sequence and related parameters for channel element.
> >  *
> >  * @param ctx   model context
> >  * @param audio samples for the current frame
> >  * @param la    lookahead samples (NULL when unavailable)
> >  * @param tag   number of channel element to analyze
> >  * @param type  channel element type (e.g. ID_SCE or ID_CPE)
> >  * @param cpe   pointer to the current channel element
> >  */
> > void ff_aac_psy_suggest_window(AACPsyContext *ctx, int16_t *audio, int16_t *la, int tag, int type, ChannelElement *cpe);
> 
> I really think this should return the window size as int
> 0 could mean "dont know try both" possibly, of course it can be done
> differently.

It also set window shape (sine/KBD) and does that for up to two channels.
As for "don't know" window type, it will require some radical changes in
encoder and model and may be considered only as a future thing.
 
> [...]
> > /**
> >  * Quantize one coefficient.
> >  * @return absolute value of the quantized coefficient
> >  * @see 3GPP TS26.403 5.6.2 "Scalefactor determination"
> >  */
> > static av_always_inline int quant(float coef, const float Q)
> > {
> >     return av_clip((int)(pow(fabsf(coef) * Q, 0.75) + 0.4054), 0, 8191);
> > }
> 
> converting float to int by casting is rather slow on x86
> also i do not see why the cliping against 0 is done
> 
> and where does the 0.4054 come from? How has this value been selected?

ask 3GPP folks, in their spec (there's a reference in the comment above)
it's also called MAGIC_NUMBER.

as for clipping, it seemed more logical than applying FFMIN()

> > 
> > /**
> >  * Convert coefficients to integers.
> 
> >  * @return sum of coefficients
> 
> this is not true, its the sum of their absolute values

corrected 
 
> [...]
> > 
> > /**
> >  * constants for 3GPP AAC psychoacoustic model
> >  * @{
> >  */
> > #define PSY_3GPP_C1 3.0f                    // log2(8.0)
> > #define PSY_3GPP_C2 1.32192809488736234787f // log2(2.5)
> > #define PSY_3GPP_C3 0.55935730170421255071f // 1 - C2/C1
> > 
> > #define PSY_3GPP_SPREAD_LOW  1.5f // spreading factor for ascending threshold spreading  (15 dB/Bark)
> > #define PSY_3GPP_SPREAD_HI   3.0f // spreading factor for descending threshold spreading (30 dB/Bark)
> > 
> 
> > #define PSY_3GPP_RPEMIN      0.01f
> > #define PSY_3GPP_RPELEV      2.0f
> 
> RPE ?
> please document what that means, iam no psychoacoustic guru
> Its easier to review code when one knows what something is.

Oh, please become one :)
Do you see any comments about them? That's because it's hard to say
anything about them. All I know is that they have some relation to
pre-echo control.
 
> > /**
> >  * @}
> >  */
> > 
> > /**
> >  * information for single band used by 3GPP TS26.403-inspired psychoacoustic model
> >  */
> > typedef struct Psy3gppBand{
> >     float energy;    ///< band energy
> >     float ffac;      ///< form factor
> >     float thr;       ///< energy threshold
> >     float pe;        ///< perceptual entropy
> 
> >     float a;         ///< constant part in perceptual entropy
> >     float b;         ///< variable part in perceptual entropy
> 
> no, single letter variable names are not good for structs
> iam also not truly happy about pe and nl either but they at least have names
> matching the initial letters of what the abbreviate.

taken from the spec, it's also the case when it's hard to give any
sane name to it

> >     float nl;        ///< predicted number of lines left after quantization
> >     float min_snr;   ///< minimal SNR
> >     float thr_quiet; ///< threshold in quiet
> > }Psy3gppBand;
> > 
> > /**
> >  * single/pair channel context for psychoacoustic model
> >  */
> > typedef struct Psy3gppChannel{
> >     float       a[2];                       ///< parameter used for perceptual entropy - constant part
> >     float       b[2];                       ///< parameter used for perceptual entropy - variable part
> >     float       pe[2];                      ///< channel perceptual entropy
> >     float       thr[2];                     ///< channel thresholds sum
> 
> this feels somewhat duplicated from above, iam not saying i want it to
> be changed though, i first need to better understand the code

those are simply sums from all bands for each channel,
they are used in final calculation. 
 
> >     Psy3gppBand band[2][128];               ///< bands information
> >     Psy3gppBand prev_band[2][128];          ///< bands information from the previous frame
> 
> no next_band? yes it may be a naive question but my naive mind would consider
> the next to be usefull as well when the previous is.

previous frame band data is used for pre-echo control,
while next frame band data is nor known yet (but it may
be used in the same way indeed)
 
> > 
> >     float       win_nrg[2];                 ///< sliding average of channel energy
> 
> nrg is not an acceptable abbreviation for energy

renamed

> [...]
> > /**
> >  * Tell encoder which window types to use.
> >  * @see 3GPP TS26.403 5.4.1 "Blockswitching"
> >  */
> > static void psy_3gpp_window(AACPsyContext *apc, int16_t *audio, int16_t *la, int tag, int type, ChannelElement *cpe)
> > {
> >     int ch;
> >     int chans = type == ID_CPE ? 2 : 1;
> >     int i, j;
> >     int br = apc->avctx->bit_rate / apc->avctx->channels;
> >     int attack_ratio = (br <= 16000 + 8000*chans) ? 18 : 10;
> >     Psy3gppContext *pctx = (Psy3gppContext*) apc->model_priv_data;
> >     Psy3gppChannel *pch = &pctx->ch[tag];
> >     uint8_t grouping[2];
> >     enum WindowSequence win[2];
> > 
> 
> >     if(la && !(apc->flags & PSY_MODEL_NO_SWITCH)){
> >         float s[8], v;
> >         for(ch = 0; ch < chans; ch++){
> >             enum WindowSequence last_window_sequence = cpe->ch[ch].ics.window_sequence[0];
> >             int switch_to_eight = 0;
> >             float sum = 0.0, sum2 = 0.0;
> >             int attack_n = 0;
> >             for(i = 0; i < 8; i++){
> >                 for(j = 0; j < 128; j++){
> >                     v = iir_filter(audio[(i*128+j)*apc->avctx->channels+ch], pch->iir_state[ch]);
> 
> this filter can be unrolled by a factor of 2 to avoid some moves in its
> state (like the low pass one)

that's useless. It's order-1 filter after all. 
 
> >                     sum += v*v;
> >                 }
> >                 s[i] = sum;
> >                 sum2 += sum;
> >             }
> >             for(i = 0; i < 8; i++){
> >                 if(s[i] > pch->win_nrg[ch] * attack_ratio){
> >                     attack_n = i + 1;
> >                     switch_to_eight = 1;
> >                     break;
> >                 }
> >             }
> >             pch->win_nrg[ch] = pch->win_nrg[ch]*7/8 + sum2/64;
> > 
> >             switch(last_window_sequence){
> >             case ONLY_LONG_SEQUENCE:
> >                 win[ch] = switch_to_eight ? LONG_START_SEQUENCE : ONLY_LONG_SEQUENCE;
> >                 grouping[ch] = 0;
> >                 break;
> >             case LONG_START_SEQUENCE:
> >                 win[ch] = EIGHT_SHORT_SEQUENCE;
> >                 grouping[ch] = pch->next_grouping[ch];
> >                 break;
> >             case LONG_STOP_SEQUENCE:
> >                 win[ch] = ONLY_LONG_SEQUENCE;
> >                 grouping[ch] = 0;
> >                 break;
> >             case EIGHT_SHORT_SEQUENCE:
> >                 win[ch] = switch_to_eight ? EIGHT_SHORT_SEQUENCE : LONG_STOP_SEQUENCE;
> >                 grouping[ch] = switch_to_eight ? pch->next_grouping[ch] : 0;
> >                 break;
> >             }
> >             pch->next_grouping[ch] = window_grouping[attack_n];
> >         }
> 
> How much quality is lost by using this compared to RD optimal switching?

err, is there such thing as quality?
Per my understanding, 8 short windows sequence is used for better preserving
of sudden change of signal. And the change decision is implemented after
the same 3GPP spec.
 
> [...]
> 
> > /**
> >  * Modify threshold by adding some value in loudness domain.
> >  * @see 3GPP TS26.403 5.6.1.1.1 "Addition of noise with equal loudness"
> >  */
> > static inline float modify_thr(float thr, float r){
> >     float t;
> >     t = pow(thr, 0.25) + r;
> >     return t*t*t*t;
> 
> (t*t)*(t*t)
> might help the compiler optimize it

ok

> [...]
> > /**
> >  * Determine scalefactors and prepare coefficients for encoding.
> >  * @see 3GPP TS26.403 5.4 "Psychoacoustic model"
> >  */
> > static void psy_3gpp_process(AACPsyContext *apc, int tag, int type, ChannelElement *cpe)
> > {
> >     int start;
> >     int ch, w, wg, g, g2, i;
> >     int prev_scale;
> >     Psy3gppContext *pctx = (Psy3gppContext*) apc->model_priv_data;
> >     float pe_target;
> >     int bits_avail;
> >     int chans = type == ID_CPE ? 2 : 1;
> >     Psy3gppChannel *pch = &pctx->ch[tag];
> > 
> >     //calculate energies, initial thresholds and related values - 5.4.2 "Threshold Calculation"
> >     memset(pch->band, 0, sizeof(pch->band));
> >     for(ch = 0; ch < chans; ch++){
> >         start = 0;
> >         for(w = 0; w < cpe->ch[ch].ics.num_windows; w++){
> >             for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
> >                 g2 = w*16 + g;
> >                 for(i = 0; i < cpe->ch[ch].ics.swb_sizes[g]; i++)
> >                     pch->band[ch][g2].energy +=  cpe->ch[ch].coeffs[start+i] *  cpe->ch[ch].coeffs[start+i];
> 
> >                 pch->band[ch][g2].energy /= 262144.0f;
> 
> and this factor does what?
> besides it should be *= 1.0/...

I hope gcc will optimize division by constant.
And it's used for compensate effect of input coefficients
being 512 times larger because of throwing out division
from windowing+MDCT operation. The same thing is done
to form factor calculation.

[...]
> >         //determine scalefactors - 5.6.2 "Scalefactor determination"
> >         for(ch = 0; ch < chans; ch++){
> >             prev_scale = -1;
> >             for(w = 0; w < cpe->ch[ch].ics.num_windows; w++){
> >                 for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
> >                     g2 = w*16 + g;
> 
> >                     cpe->ch[ch].zeroes[w][g] = pch->band[ch][g2].thr >= pch->band[ch][g2].energy;
> 
> how much quality is lost compared to full RD decission ? its just a matter of
> checking how many bits this would need which is likely negligible speed wise.
> (assuming you can unentangle the threshold check into a distortion
>  computation)
 
well, energy < threshold means resulting band will be zero anyway,
and without that check weird values for perceptual entropy start
to appear
 
> >                     if(cpe->ch[ch].zeroes[w][g]) continue;
> >                     //spec gives constant for lg() but we scaled it for log2()
> >                     cpe->ch[ch].sf_idx[w][g] = (int)(2.66667 * (log2(6.75*pch->band[ch][g2].thr) - log2(pch->band[ch][g2].ffac)));
> 
> 2.666... * log2(6.75*pch->band[ch][g2].thr / pch->band[ch][g2].ffac)
> 
> 
> 
> >                     if(prev_scale != -1)
> >                         cpe->ch[ch].sf_idx[w][g] = av_clip(cpe->ch[ch].sf_idx[w][g], prev_scale - SCALE_MAX_DIFF, prev_scale + SCALE_MAX_DIFF);
> >                     prev_scale = cpe->ch[ch].sf_idx[w][g];
> >                 }
> >             }
> >         }
> >         break;
> >     case PSY_MODE_QUALITY:
> >         for(ch = 0; ch < chans; ch++){
> >             start = 0;
> >             for(w = 0; w < cpe->ch[ch].ics.num_windows; w++){
> >                 for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
> >                     g2 = w*16 + g;
> >                     if(pch->band[ch][g2].thr >= pch->band[ch][g2].energy){
> >                         cpe->ch[ch].sf_idx[w][g] = 0;
> >                         cpe->ch[ch].zeroes[w][g] = 1;
> >                     }else{
> >                         cpe->ch[ch].zeroes[w][g] = 0;
> >                         cpe->ch[ch].sf_idx[w][g] = (int)(2.66667 * (log2(6.75*pch->band[ch][g2].thr) - log2(pch->band[ch][g2].ffac)));
> >                         while(cpe->ch[ch].sf_idx[ch][g] > 3){
> >                             float dist = calc_distortion(cpe->ch[ch].coeffs + start, cpe->ch[ch].ics.swb_sizes[g], SCALE_ONE_POS + cpe->ch[ch].sf_idx[ch][g]);
> >                             if(dist < pch->band[ch][g2].thr) break;
> >                             cpe->ch[ch].sf_idx[ch][g] -= 3;
> >                         }
> >                     }
> 
> looks rather similar to the previous cases

partially - it determines primary scalefactor index from threshold
in the same way, what it does to thresholds before that and index
after that differs. 
 
> >                     start += cpe->ch[ch].ics.swb_sizes[g];
> >                 }
> >             }
> >         }
> >         break;
> >     }
> > 
> 
> >     //limit scalefactors
> 
> why? bitstream limits or is this related to psy?

both
 
> [...]
> > static const AACPsyModel psy_models[AAC_NB_PSY_MODELS] =
> > {
> >     {
> >        "Null model",
> >         NULL,
> >         psy_null_window,
> >         psy_null_process,
> >         NULL,
> >     },
> >     {
> >        "Null model - short windows",
> >         NULL,
> >         psy_null8_window,
> >         psy_null8_process,
> >         NULL,
> >     },
> >     {
> >        "3GPP TS 26.403-inspired model",
> >         psy_3gpp_init,
> >         psy_3gpp_window,
> >         psy_3gpp_process,
> >         psy_3gpp_end,
> >     },
> > };
> 
> Iam tempted to suggest that you drop the 2 null models. I see no real
> sense in them. But if you like to keep them thats perfectly fine with me.
> Adding a 4th model for experimenation would possibly make sense
> Also you could keep the 3gpp exactly as the spec says and implement all my
> suggestions rather in a 4th model that would make comparissions easier and
> would give some protection against mistakes ...

ok, dropped one model by now 
 
> > 
> > int av_cold ff_aac_psy_init(AACPsyContext *ctx, AVCodecContext *avctx,
> >                             enum AACPsyModelType model, int elements, int flags,
> >                             const uint8_t *bands1024, int num_bands1024,
> >                             const uint8_t *bands128,  int num_bands128)
> > {
> >     int i;
> > 
> 
> >     if(model >= AAC_NB_PSY_MODELS || !psy_models[model].window || !psy_models[model].process){
> >          av_log(avctx, AV_LOG_ERROR, "Invalid psy model\n");
> >          return -1;
> >     }
> 
> is there really any sense in checking .window and .process of a const static
> array?

no 
 
> > 
> 
> > #ifndef CONFIG_HARDCODED_TABLES
> >    for (i = 0; i < 316; i++)
> >         ff_aac_pow2sf_tab[i] = pow(2, (i - 200)/4.);
> > #endif /* CONFIG_HARDCODED_TABLES */
> 
> this is likely duplicated

it is not. When table is not hardcoded, it should be initialized. 
 
> [...]
> > void ff_aac_psy_preprocess(AACPsyContext *ctx, int16_t *audio, int16_t *dest, int tag, int type)
> > {
> >     int chans = type == ID_CPE ? 2 : 1;
> >     const int chstride = ctx->avctx->channels;
> >     int i, ch;
> >     float t[2];
> > 
> >     if(chans == 1){
> 
> >         for(ch = 0; ch < chans; ch++){
> >             for(i = 0; i < 1024; i++){
> >                 dest[i * chstride + ch] = audio[i * chstride + ch];
> >             }
> >         }
> 
> memcpy

no, it copies with gaps 
 
> >     }else{
> >         for(i = 0; i < 1024; i++){
> >             if(ctx->flags & PSY_MODEL_NO_ST_ATT){
> >                 for(ch = 0; ch < 2; ch++)
> >                     t[ch] = audio[i * chstride + ch];
> >             }else{
> >                 t[0] = audio[i * chstride + 0] * (0.5 + ctx->stereo_att) + audio[i * chstride + 1] * (0.5 - ctx->stereo_att);
> >                 t[1] = audio[i * chstride + 0] * (0.5 - ctx->stereo_att) + audio[i * chstride + 1] * (0.5 + ctx->stereo_att);
> >             }
> >             if(!(ctx->flags & PSY_MODEL_NO_LOWPASS)){
> >                 LPFilterState *is = (LPFilterState*)ctx->lp_state + tag*2;
> 
> >                 for(ch = 0; ch < 2; ch++)
> >                     t[ch] = ff_lowpass_filter(&ctx->lp_coeffs, is + ch, t[ch]);
> 
> i do not think this is compatible with the filter you posted

it is not by now, I will deal with it when commit my filter stuff,
ignore that for now 
 
> >             }
> >             for(ch = 0; ch < 2; ch++)
> >                 dest[i * chstride + ch] = av_clip_int16(t[ch]);
> 
> we have optimized code for converting float to int16, please use it
 

I may have overlooked some stuff for now but that's because it's too
hot here to think properly.
 
> [...]
> -- 
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
> 
> It is not what we do, but why we do it that matters.
-------------- next part --------------
/*
 * AAC encoder psychoacoustic model
 * Copyright (C) 2008 Konstantin Shishkov
 *
 * This file is part of FFmpeg.
 *
 * FFmpeg is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License as published by the Free Software Foundation; either
 * version 2.1 of the License, or (at your option) any later version.
 *
 * FFmpeg is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with FFmpeg; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
 */

#ifndef FFMPEG_AACPSY_H
#define FFMPEG_AACPSY_H

#include "avcodec.h"
#include "aac.h"
#include "lowpass.h"

enum AACPsyModelType{
    AAC_PSY_TEST,              ///< a sample model to exercise encoder
    AAC_PSY_3GPP,              ///< model following recommendations from 3GPP TS 26.403

    AAC_NB_PSY_MODELS          ///< total number of psychoacoustic models, since it's not a part of the ABI new models can be added freely
};

enum AACPsyModelMode{
    PSY_MODE_CBR,              ///< follow bitrate as closely as possible
    PSY_MODE_ABR,              ///< try to achieve bitrate but actual bitrate may differ significantly
    PSY_MODE_QUALITY,          ///< try to achieve set quality instead of bitrate
};

#define PSY_MODEL_MODE_MASK  0x0000000F ///< bit fields for storing mode (CBR, ABR, VBR)
#define PSY_MODEL_NO_SWITCH  0x00000020 ///< disable window switching
#define PSY_MODEL_NO_ST_ATT  0x00000040 ///< disable stereo attenuation
#define PSY_MODEL_NO_LOWPASS 0x00000080 ///< disable low-pass filtering

#define PSY_MODEL_NO_PREPROC (PSY_MODEL_NO_ST_ATT | PSY_MODEL_NO_LOWPASS)

#define PSY_MODEL_MODE(a)  ((a) & PSY_MODEL_MODE_MASK)

/**
 * context used by psychoacoustic model
 */
typedef struct AACPsyContext {
    AVCodecContext *avctx;            ///< encoder context

    int flags;                        ///< model flags
    const uint8_t *bands1024;         ///< scalefactor band sizes for long (1024 samples) frame
    int num_bands1024;                ///< number of scalefactor bands for long frame
    const uint8_t *bands128;          ///< scalefactor band sizes for short (128 samples) frame
    int num_bands128;                 ///< number of scalefactor bands for short frame

    const struct AACPsyModel *model;  ///< pointer to the psychoacoustic model implementation
    void* model_priv_data;            ///< psychoacoustic model implementation private data

    float stereo_att;                 ///< stereo attenuation factor
    LPFilterCoeffs lp_coeffs;         ///< lowpass filter coefficients
    LPFilterState *lp_state;          ///< lowpass filter state
}AACPsyContext;

typedef struct AACPsyModel {
    const char *name;
    int   (*init)   (AACPsyContext *apc, int elements);
    void  (*window) (AACPsyContext *apc, int16_t *audio, int16_t *la, int tag, int type, ChannelElement *cpe);
    void  (*process)(AACPsyContext *apc, int tag, int type, ChannelElement *cpe);
    void  (*end)    (AACPsyContext *apc);
}AACPsyModel;

/**
 * Initialize psychoacoustic model.
 *
 * @param ctx           model context
 * @param avctx         codec context
 * @param model         model implementation that will be used
 * @param elements      number of channel elements (single channel or channel pair) to handle by  model
 * @param flags         model flags, may be ignored by model if unsupported
 * @param bands1024     scalefactor band lengths for long (1024 samples) frame
 * @param num_bands1024 number of scalefactor bands for long frame
 * @param bands128      scalefactor band lengths for short (128 samples) frame
 * @param num_bands128  number of scalefactor bands for short frame
 *
 * @return zero if successful, a negative value if not
 */
int ff_aac_psy_init(AACPsyContext *ctx, AVCodecContext *avctx,
                    enum AACPsyModelType model, int elements, int flags,
                    const uint8_t *bands1024, int num_bands1024,
                    const uint8_t *bands128,  int num_bands128);

/**
 * Preprocess audio frame in order to compress it better.
 *
 * @param ctx   model context
 * @param audio samples to preprocess
 * @param dest  place to put filtered samples
 * @param tag   number of channel element to analyze
 * @param type  channel element type (e.g. ID_SCE or ID_CPE)
 */
void ff_aac_psy_preprocess(AACPsyContext *ctx, int16_t *audio, int16_t *dest, int tag, int type);

/**
 * Set window sequence and related parameters for channel element.
 *
 * @param ctx   model context
 * @param audio samples for the current frame
 * @param la    lookahead samples (NULL when unavailable)
 * @param tag   number of channel element to analyze
 * @param type  channel element type (e.g. ID_SCE or ID_CPE)
 * @param cpe   pointer to the current channel element
 */
void ff_aac_psy_suggest_window(AACPsyContext *ctx, int16_t *audio, int16_t *la, int tag, int type, ChannelElement *cpe);

/**
 * Perform psychoacoustic analysis and output coefficients in integer form
 * along with scalefactors, M/S flags, etc.
 *
 * @param ctx   model context
 * @param tag   number of channel element to analyze
 * @param type  channel element type (e.g. ID_SCE or ID_CPE)
 * @param cpe   pointer to the current channel element
 */
void ff_aac_psy_analyze(AACPsyContext *ctx, int tag, int type, ChannelElement *cpe);

/**
 * Cleanup model context at the end.
 *
 * @param ctx model context
 */
void ff_aac_psy_end(AACPsyContext *ctx);

#endif /* FFMPEG_AACPSY_H */

-------------- next part --------------
/*
 * AAC encoder psychoacoustic model
 * Copyright (C) 2008 Konstantin Shishkov
 *
 * This file is part of FFmpeg.
 *
 * FFmpeg is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License as published by the Free Software Foundation; either
 * version 2.1 of the License, or (at your option) any later version.
 *
 * FFmpeg is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with FFmpeg; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
 */

/**
 * @file aacpsy.c
 * AAC encoder psychoacoustic model
 */

#include "avcodec.h"
#include "aacpsy.h"
#include "aactab.h"

/***********************************
 *              TODOs:
 * General:
 * better audio preprocessing (add DC highpass filter?)
 * more psy models
 *
 * 3GPP-based psy model:
 * thresholds linearization after their modifications for attaining given bitrate
 * try other bitrate controlling mechanism (maybe use ratecontrol.c?)
 * control quality for quality-based output
 **********************************/

/**
 * Quantize one coefficient.
 * @return absolute value of the quantized coefficient
 * @see 3GPP TS26.403 5.6.2 "Scalefactor determination"
 */
static av_always_inline int quant(float coef, const float Q)
{
    return av_clip((int)(pow(fabsf(coef) * Q, 0.75) + 0.4054), 0, 8191);
}

/**
 * Convert coefficients to integers.
 * @return sum of coefficient absolute values
 */
static inline int quantize_coeffs(float *in, int *out, int size, int scale_idx)
{
    int i, sign, sum = 0;
    const float Q = ff_aac_pow2sf_tab[200 - scale_idx + SCALE_ONE_POS - SCALE_DIV_512];
    for(i = 0; i < size; i++){
        sign = in[i] > 0.0;
        out[i] = quant(in[i], Q);
        sum += out[i];
        if(sign) out[i] = -out[i];
    }
    return sum;
}

static inline float calc_distortion(float *c, int size, int scale_idx)
{
    int i;
    int q;
    float coef, unquant, sum = 0.0f;
    const float Q  = ff_aac_pow2sf_tab[200 - scale_idx + SCALE_ONE_POS - SCALE_DIV_512];
    const float IQ = ff_aac_pow2sf_tab[200 + scale_idx - SCALE_ONE_POS + SCALE_DIV_512];
    for(i = 0; i < size; i++){
        coef = fabs(c[i]);
        q = quant(c[i], Q);
        unquant = (q * cbrt(q)) * IQ;
        sum += (coef - unquant) * (coef - unquant);
    }
    return sum;
}

/**
 * Produce integer coefficients from scalefactors provided by the model.
 */
static void psy_create_output(AACPsyContext *apc, ChannelElement *cpe, int chans)
{
    int i, w, w2, wg, g, ch;
    int start, sum, maxsfb, cmaxsfb;

    for(ch = 0; ch < chans; ch++){
        start = 0;
        maxsfb = 0;
        cpe->ch[ch].pulse.num_pulse = 0;
        for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16){
            for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                sum = 0;
                //apply M/S
                if(!ch && cpe->ms_mask[w + g]){
                    for(i = 0; i < cpe->ch[ch].ics.swb_sizes[g]; i++){
                        cpe->ch[0].coeffs[start+i] = (cpe->ch[0].coeffs[start+i] + cpe->ch[1].coeffs[start+i]) / 2.0;
                        cpe->ch[1].coeffs[start+i] =  cpe->ch[0].coeffs[start+i] - cpe->ch[1].coeffs[start+i];
                    }
                }
                if(!cpe->ch[ch].zeroes[w + g])
                    sum = quantize_coeffs(cpe->ch[ch].coeffs + start, cpe->ch[ch].icoefs + start, cpe->ch[ch].ics.swb_sizes[g], cpe->ch[ch].sf_idx[w + g]);
                else
                    memset(cpe->ch[ch].icoefs + start, 0, cpe->ch[ch].ics.swb_sizes[g] * sizeof(cpe->ch[0].icoefs[0]));
                cpe->ch[ch].zeroes[w + g] = !sum;
                start += cpe->ch[ch].ics.swb_sizes[g];
            }
            for(cmaxsfb = cpe->ch[ch].ics.num_swb; cmaxsfb > 0 && cpe->ch[ch].zeroes[w+cmaxsfb-1]; cmaxsfb--);
            maxsfb = FFMAX(maxsfb, cmaxsfb);
        }
        cpe->ch[ch].ics.max_sfb = maxsfb;

        //adjust zero bands for window groups
        w = 0;
        for(wg = 0; wg < cpe->ch[ch].ics.num_window_groups; wg++){
            for(g = 0; g < cpe->ch[ch].ics.max_sfb; g++){
                i = 1;
                for(w2 = 0; w2 < cpe->ch[ch].ics.group_len[wg]*16; w2 += 16){
                    if(!cpe->ch[ch].zeroes[w + w2 + g]){
                        i = 0;
                        break;
                    }
                }
                cpe->ch[ch].zeroes[w + g] = i;
            }
            w += cpe->ch[ch].ics.group_len[wg] * 16;
        }
    }

    if(chans > 1 && cpe->common_window){
        int msc = 0;
        cpe->ch[0].ics.max_sfb = FFMAX(cpe->ch[0].ics.max_sfb, cpe->ch[1].ics.max_sfb);
        cpe->ch[1].ics.max_sfb = cpe->ch[0].ics.max_sfb;
        for(w = 0; w < cpe->ch[0].ics.num_windows*16; w += 16)
            for(i = 0; i < cpe->ch[0].ics.max_sfb; i++)
                if(cpe->ms_mask[w+i]) msc++;
        if(msc == 0 || cpe->ch[0].ics.max_sfb == 0) cpe->ms_mode = 0;
        else cpe->ms_mode = msc < cpe->ch[0].ics.max_sfb ? 1 : 2;
    }
}

static void psy_test_window(AACPsyContext *apc, int16_t *audio, int16_t *la, int tag, int type, ChannelElement *cpe)
{
    int ch, i;
    int chans = type == TYPE_CPE ? 2 : 1;

    for(ch = 0; ch < chans; ch++){
        int prev_seq = cpe->ch[ch].ics.window_sequence[1];
        cpe->ch[ch].ics.use_kb_window[1] = cpe->ch[ch].ics.use_kb_window[0];
        cpe->ch[ch].ics.window_sequence[1] = cpe->ch[ch].ics.window_sequence[0];
        switch(cpe->ch[ch].ics.window_sequence[0]){
        case ONLY_LONG_SEQUENCE:   if(prev_seq == ONLY_LONG_SEQUENCE)cpe->ch[ch].ics.window_sequence[0] = LONG_START_SEQUENCE;   break;
        case LONG_START_SEQUENCE:  cpe->ch[ch].ics.window_sequence[0] = EIGHT_SHORT_SEQUENCE; break;
        case EIGHT_SHORT_SEQUENCE: if(prev_seq == EIGHT_SHORT_SEQUENCE)cpe->ch[ch].ics.window_sequence[0] = LONG_STOP_SEQUENCE;  break;
        case LONG_STOP_SEQUENCE:   cpe->ch[ch].ics.window_sequence[0] = ONLY_LONG_SEQUENCE;   break;
        }

        if(cpe->ch[ch].ics.window_sequence[0] != EIGHT_SHORT_SEQUENCE){
            cpe->ch[ch].ics.use_kb_window[0] = 1;
            cpe->ch[ch].ics.num_windows = 1;
            cpe->ch[ch].ics.swb_sizes = apc->bands1024;
            cpe->ch[ch].ics.num_swb = apc->num_bands1024;
            cpe->ch[ch].ics.num_window_groups = 1;
            cpe->ch[ch].ics.group_len[0] = 1;
        }else{
            cpe->ch[ch].ics.use_kb_window[0] = 1;
            cpe->ch[ch].ics.num_windows = 8;
            cpe->ch[ch].ics.swb_sizes = apc->bands128;
            cpe->ch[ch].ics.num_swb = apc->num_bands128;
            cpe->ch[ch].ics.num_window_groups = 4;
            for(i = 0; i < 4; i++)
                cpe->ch[ch].ics.group_len[i] = 2;
        }
    }
    cpe->common_window = cpe->ch[0].ics.use_kb_window[0] == cpe->ch[1].ics.use_kb_window[0];
}

static void psy_test_process(AACPsyContext *apc, int tag, int type, ChannelElement *cpe)
{
    int start;
    int w, ch, g, i;
    int chans = type == TYPE_CPE ? 2 : 1;

    //detect M/S
    if(chans > 1 && cpe->common_window){
        start = 0;
        for(w = 0; w < cpe->ch[0].ics.num_windows*16; w += 16){
            for(g = 0; g < cpe->ch[0].ics.num_swb; g++){
                float diff = 0.0f;

                for(i = 0; i < cpe->ch[0].ics.swb_sizes[g]; i++)
                    diff += fabs(cpe->ch[0].coeffs[start+i] - cpe->ch[1].coeffs[start+i]);
                cpe->ms_mask[w + g] = diff == 0.0;
            }
        }
    }
    for(ch = 0; ch < chans; ch++){
        for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16){
            for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                cpe->ch[ch].sf_idx[w + g] = SCALE_ONE_POS;
                cpe->ch[ch].zeroes[w + g] = 0;
            }
        }
    }
    psy_create_output(apc, cpe, chans);
}

/**
 * constants for 3GPP AAC psychoacoustic model
 * @{
 */
#define PSY_3GPP_C1 3.0f                    // log2(8.0)
#define PSY_3GPP_C2 1.32192809488736234787f // log2(2.5)
#define PSY_3GPP_C3 0.55935730170421255071f // 1 - C2/C1

#define PSY_3GPP_SPREAD_LOW  1.5f // spreading factor for ascending threshold spreading  (15 dB/Bark)
#define PSY_3GPP_SPREAD_HI   3.0f // spreading factor for descending threshold spreading (30 dB/Bark)

#define PSY_3GPP_RPEMIN      0.01f
#define PSY_3GPP_RPELEV      2.0f
/**
 * @}
 */

/**
 * information for single band used by 3GPP TS26.403-inspired psychoacoustic model
 */
typedef struct Psy3gppBand{
    float energy;    ///< band energy
    float ffac;      ///< form factor
    float thr;       ///< energy threshold
    float pe;        ///< perceptual entropy
    float a;         ///< constant part in perceptual entropy
    float b;         ///< variable part in perceptual entropy
    float nl;        ///< predicted number of lines left after quantization
    float min_snr;   ///< minimal SNR
    float thr_quiet; ///< threshold in quiet
}Psy3gppBand;

/**
 * single/pair channel context for psychoacoustic model
 */
typedef struct Psy3gppChannel{
    float       a[2];                       ///< parameter used for perceptual entropy - constant part
    float       b[2];                       ///< parameter used for perceptual entropy - variable part
    float       pe[2];                      ///< channel perceptual entropy
    float       thr[2];                     ///< channel thresholds sum
    Psy3gppBand band[2][128];               ///< bands information
    Psy3gppBand prev_band[2][128];          ///< bands information from the previous frame

    float       win_energy[2];              ///< sliding average of channel energy
    float       iir_state[2][2];            ///< hi-pass IIR filter state
    uint8_t     next_grouping[2];           ///< stored grouping scheme for the next frame (in case of 8 short window sequence)
    enum WindowSequence next_window_seq[2]; ///< window sequence to be used in the next frame
}Psy3gppChannel;

/**
 * 3GPP TS26.403-inspired psychoacoustic model specific data
 */
typedef struct Psy3gppContext{
    float       barks [1024]; ///< Bark value for each spectral line
    float       bark_l[64];   ///< Bark value for each spectral band in long frame
    float       bark_s[16];   ///< Bark value for each spectral band in short frame
    float       s_low_l[64];  ///< spreading factor for low-to-high threshold spreading in long frame
    float       s_low_s[16];  ///< spreading factor for low-to-high threshold spreading in short frame
    float       s_hi_l [64];  ///< spreading factor for high-to-low threshold spreading in long frame
    float       s_hi_s [16];  ///< spreading factor for high-to-low threshold spreading in short frame
    int         reservoir;    ///< bit reservoir fullness
    int         avg_bits;     ///< average frame size of bits for CBR
    float       ath_l[64];    ///< absolute threshold of hearing per bands in long frame
    float       ath_s[16];    ///< absolute threshold of hearing per bands in short frame
    Psy3gppChannel *ch;
}Psy3gppContext;

/**
 * Calculate Bark value for given line.
 */
static inline float calc_bark(float f)
{
    return 13.3f * atanf(0.00076f * f) + 3.5f * atanf((f / 7500.0f) * (f / 7500.0f));
}

#define ATH_ADD 4
/**
 * Calculate ATH value for given frequency.
 * Borrowed from Lame.
 */
static inline float ath(float f, float add)
{
    f /= 1000.0f;
    return   3.64 * pow(f, -0.8)
            - 6.8  * exp(-0.6  * (f - 3.4) * (f - 3.4))
            + 6.0  * exp(-0.15 * (f - 8.7) * (f - 8.7))
            + (0.6 + 0.04 * add) * 0.001 * f * f * f * f;
}

static av_cold int psy_3gpp_init(AACPsyContext *apc, int elements)
{
    Psy3gppContext *pctx;
    int i, g, start;
    float prev, minscale, minath;
    apc->model_priv_data = av_mallocz(sizeof(Psy3gppContext));
    pctx = (Psy3gppContext*) apc->model_priv_data;

    for(i = 0; i < 1024; i++)
        pctx->barks[i] = calc_bark(i * apc->avctx->sample_rate / 2048.0);
    i = 0;
    prev = 0.0;
    for(g = 0; g < apc->num_bands1024; g++){
        i += apc->bands1024[g];
        pctx->bark_l[g] = (pctx->barks[i - 1] + prev) / 2.0;
        prev = pctx->barks[i - 1];
    }
    for(g = 0; g < apc->num_bands1024 - 1; g++){
        pctx->s_low_l[g] = pow(10.0, -(pctx->bark_l[g+1] - pctx->bark_l[g]) * PSY_3GPP_SPREAD_LOW);
        pctx->s_hi_l [g] = pow(10.0, -(pctx->bark_l[g+1] - pctx->bark_l[g]) * PSY_3GPP_SPREAD_HI);
    }
    i = 0;
    prev = 0.0;
    for(g = 0; g < apc->num_bands128; g++){
        i += apc->bands128[g];
        pctx->bark_s[g] = (pctx->barks[i - 1] + prev) / 2.0;
        prev = pctx->barks[i - 1];
    }
    for(g = 0; g < apc->num_bands128 - 1; g++){
        pctx->s_low_s[g] = pow(10.0, -(pctx->bark_s[g+1] - pctx->bark_s[g]) * PSY_3GPP_SPREAD_LOW);
        pctx->s_hi_s [g] = pow(10.0, -(pctx->bark_s[g+1] - pctx->bark_s[g]) * PSY_3GPP_SPREAD_HI);
    }
    start = 0;
    minath = ath(3410, ATH_ADD);
    for(g = 0; g < apc->num_bands1024; g++){
        minscale = ath(apc->avctx->sample_rate * start / 1024.0, ATH_ADD);
        for(i = 1; i < apc->bands1024[g]; i++){
            minscale = fminf(minscale, ath(apc->avctx->sample_rate * (start + i) / 1024.0 / 2.0, ATH_ADD));
        }
        pctx->ath_l[g] = minscale - minath;
        start += apc->bands1024[g];
    }
    start = 0;
    for(g = 0; g < apc->num_bands128; g++){
        minscale = ath(apc->avctx->sample_rate * start / 1024.0, ATH_ADD);
        for(i = 1; i < apc->bands128[g]; i++){
            minscale = fminf(minscale, ath(apc->avctx->sample_rate * (start + i) / 1024.0 / 2.0, ATH_ADD));
        }
        pctx->ath_s[g] = minscale - minath;
        start += apc->bands128[g];
    }

    pctx->avg_bits = apc->avctx->bit_rate * 1024 / apc->avctx->sample_rate;
    pctx->ch = av_mallocz(sizeof(Psy3gppChannel) * elements);
    return 0;
}

/**
 * IIR filter used in block switching decision
 */
static float iir_filter(int in, float state[2])
{
    float ret;

    ret = 0.7548f * (in - state[0]) + 0.5095f * state[1];
    state[0] = in;
    state[1] = ret;
    return ret;
}

/**
 * window grouping information stored as bits (0 - new group, 1 - group continues)
 */
static const uint8_t window_grouping[9] = {
    0xB6, 0x6C, 0xD8, 0xB2, 0x66, 0xC6, 0x96, 0x36, 0x36
};

/**
 * Tell encoder which window types to use.
 * @see 3GPP TS26.403 5.4.1 "Blockswitching"
 */
static void psy_3gpp_window(AACPsyContext *apc, int16_t *audio, int16_t *la, int tag, int type, ChannelElement *cpe)
{
    int ch;
    int chans = type == TYPE_CPE ? 2 : 1;
    int i, j;
    int br = apc->avctx->bit_rate / apc->avctx->channels;
    int attack_ratio = (br <= 16000 + 8000*chans) ? 18 : 10;
    Psy3gppContext *pctx = (Psy3gppContext*) apc->model_priv_data;
    Psy3gppChannel *pch = &pctx->ch[tag];
    uint8_t grouping[2];
    enum WindowSequence win[2];

    if(la && !(apc->flags & PSY_MODEL_NO_SWITCH)){
        float s[8], v;
        for(ch = 0; ch < chans; ch++){
            enum WindowSequence last_window_sequence = cpe->ch[ch].ics.window_sequence[0];
            int switch_to_eight = 0;
            float sum = 0.0, sum2 = 0.0;
            int attack_n = 0;
            for(i = 0; i < 8; i++){
                for(j = 0; j < 128; j++){
                    v = iir_filter(audio[(i*128+j)*apc->avctx->channels+ch], pch->iir_state[ch]);
                    sum += v*v;
                }
                s[i] = sum;
                sum2 += sum;
            }
            for(i = 0; i < 8; i++){
                if(s[i] > pch->win_energy[ch] * attack_ratio){
                    attack_n = i + 1;
                    switch_to_eight = 1;
                    break;
                }
            }
            pch->win_energy[ch] = pch->win_energy[ch]*7/8 + sum2/64;

            switch(last_window_sequence){
            case ONLY_LONG_SEQUENCE:
                win[ch] = switch_to_eight ? LONG_START_SEQUENCE : ONLY_LONG_SEQUENCE;
                grouping[ch] = 0;
                break;
            case LONG_START_SEQUENCE:
                win[ch] = EIGHT_SHORT_SEQUENCE;
                grouping[ch] = pch->next_grouping[ch];
                break;
            case LONG_STOP_SEQUENCE:
                win[ch] = ONLY_LONG_SEQUENCE;
                grouping[ch] = 0;
                break;
            case EIGHT_SHORT_SEQUENCE:
                win[ch] = switch_to_eight ? EIGHT_SHORT_SEQUENCE : LONG_STOP_SEQUENCE;
                grouping[ch] = switch_to_eight ? pch->next_grouping[ch] : 0;
                break;
            }
            pch->next_grouping[ch] = window_grouping[attack_n];
        }
    }else{
        for(ch = 0; ch < chans; ch++){
            win[ch] = (cpe->ch[ch].ics.window_sequence[0] == EIGHT_SHORT_SEQUENCE) ? EIGHT_SHORT_SEQUENCE : ONLY_LONG_SEQUENCE;
            grouping[ch] = (cpe->ch[ch].ics.window_sequence[0] == EIGHT_SHORT_SEQUENCE) ? window_grouping[0] : 0;
        }
    }

    for(ch = 0; ch < chans; ch++){
        cpe->ch[ch].ics.window_sequence[0] = win[ch];
        cpe->ch[ch].ics.use_kb_window[0] = 1;
        if(win[ch] != EIGHT_SHORT_SEQUENCE){
            cpe->ch[ch].ics.num_windows = 1;
            cpe->ch[ch].ics.swb_sizes = apc->bands1024;
            cpe->ch[ch].ics.num_swb = apc->num_bands1024;
            cpe->ch[ch].ics.num_window_groups = 1;
            cpe->ch[ch].ics.group_len[0] = 1;
        }else{
            cpe->ch[ch].ics.num_windows = 8;
            cpe->ch[ch].ics.swb_sizes = apc->bands128;
            cpe->ch[ch].ics.num_swb = apc->num_bands128;
            cpe->ch[ch].ics.num_window_groups = 0;
            cpe->ch[ch].ics.group_len[0] = 1;
            for(i = 0; i < 8; i++){
                if((grouping[ch] >> i) & 1){
                    cpe->ch[ch].ics.group_len[cpe->ch[ch].ics.num_window_groups - 1]++;
                }else{
                    cpe->ch[ch].ics.num_window_groups++;
                    cpe->ch[ch].ics.group_len[cpe->ch[ch].ics.num_window_groups - 1] = 1;
                }
            }
        }
    }
    cpe->common_window = chans > 1 && cpe->ch[0].ics.window_sequence[0] == cpe->ch[1].ics.window_sequence[0] && cpe->ch[0].ics.use_kb_window[0] == cpe->ch[1].ics.use_kb_window[0];
    if(cpe->common_window && cpe->ch[0].ics.window_sequence[0] == EIGHT_SHORT_SEQUENCE && grouping[0] != grouping[1])
        cpe->common_window = 0;
    if(PSY_MODEL_MODE(apc->flags) > PSY_MODE_QUALITY){
        av_log(apc->avctx, AV_LOG_ERROR, "Unknown mode %d, defaulting to CBR\n", PSY_MODEL_MODE(apc->flags));
    }
}

/**
 * Modify threshold by adding some value in loudness domain.
 * @see 3GPP TS26.403 5.6.1.1.1 "Addition of noise with equal loudness"
 */
static inline float modify_thr(float thr, float r){
    float t;
    t = pow(thr, 0.25) + r;
    return (t*t)*(t*t);
}

/**
 * Calculate perceptual entropy and its corresponding values for one band.
 * @see 3GPP TS26.403 5.6.1.3 "Calculation of the reduction value"
 */
static void calc_pe(Psy3gppBand *band, int band_width)
{
    if(band->energy <= band->thr){
        band->a  = 0.0f;
        band->b  = 0.0f;
        band->nl = 0.0f;
        return;
    }
    band->nl = band->ffac / pow(band->energy/band_width, 0.25);
    if(band->energy >= band->thr * 8.0){
        band->a = band->nl * log2(band->energy);
        band->b = band->nl;
    }else{
        band->a = band->nl * (PSY_3GPP_C2 + PSY_3GPP_C3 * log2(band->energy));
        band->b = band->nl * PSY_3GPP_C3;
    }
    band->pe = band->a - band->b * log2(band->thr);
    band->min_snr = 1.0 / (pow(2.0, band->pe / band_width) - 1.5);
    if(band->min_snr < 1.26f)     band->min_snr = 1.26f;
    if(band->min_snr > 316.2277f) band->min_snr = 316.2277f;
}

/**
 * Determine scalefactors and prepare coefficients for encoding.
 * @see 3GPP TS26.403 5.4 "Psychoacoustic model"
 */
static void psy_3gpp_process(AACPsyContext *apc, int tag, int type, ChannelElement *cpe)
{
    int start;
    int ch, w, wg, g, i;
    int prev_scale;
    Psy3gppContext *pctx = (Psy3gppContext*) apc->model_priv_data;
    float pe_target;
    int bits_avail;
    int chans = type == TYPE_CPE ? 2 : 1;
    Psy3gppChannel *pch = &pctx->ch[tag];

    //calculate energies, initial thresholds and related values - 5.4.2 "Threshold Calculation"
    memset(pch->band, 0, sizeof(pch->band));
    for(ch = 0; ch < chans; ch++){
        start = 0;
        for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16){
            for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                for(i = 0; i < cpe->ch[ch].ics.swb_sizes[g]; i++)
                    pch->band[ch][w+g].energy +=  cpe->ch[ch].coeffs[start+i] *  cpe->ch[ch].coeffs[start+i];
                pch->band[ch][w+g].energy /= 262144.0f;
                pch->band[ch][w+g].thr = pch->band[ch][w+g].energy * 0.001258925f;
                start += cpe->ch[ch].ics.swb_sizes[g];
                if(pch->band[ch][w+g].energy != 0.0){
                    float ffac = 0.0;

                    for(i = 0; i < cpe->ch[ch].ics.swb_sizes[g]; i++)
                        ffac += sqrt(FFABS(cpe->ch[ch].coeffs[start+i]));
                    pch->band[ch][w+g].ffac = ffac / sqrt(512.0);
                }
            }
        }
    }

    //modify thresholds - spread, threshold in quiet - 5.4.3 "Spreaded Energy Calculation"
    for(ch = 0; ch < chans; ch++){
        for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16){
            for(g = 1; g < cpe->ch[ch].ics.num_swb; g++){
                if(cpe->ch[ch].ics.num_swb == apc->num_bands1024)
                    pch->band[ch][w+g].thr = FFMAX(pch->band[ch][w+g].thr, pch->band[ch][w+g-1].thr * pctx->s_low_l[g-1]);
                else
                    pch->band[ch][w+g].thr = FFMAX(pch->band[ch][w+g].thr, pch->band[ch][w+g-1].thr * pctx->s_low_s[g-1]);
            }
            for(g = cpe->ch[ch].ics.num_swb - 2; g >= 0; g--){
                if(cpe->ch[ch].ics.num_swb == apc->num_bands1024)
                    pch->band[ch][w+g].thr = FFMAX(pch->band[ch][w+g].thr, pch->band[ch][w+g+1].thr * pctx->s_hi_l[g+1]);
                else
                    pch->band[ch][w+g].thr = FFMAX(pch->band[ch][w+g].thr, pch->band[ch][w+g+1].thr * pctx->s_hi_s[g+1]);
            }
            for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                if(cpe->ch[ch].ics.num_swb == apc->num_bands1024)
                    pch->band[ch][w+g].thr_quiet = FFMAX(pch->band[ch][w+g].thr, pctx->ath_l[g]);
                else
                    pch->band[ch][w+g].thr_quiet = FFMAX(pch->band[ch][w+g].thr, pctx->ath_s[g]);
                pch->band[ch][w+g].thr_quiet = fmaxf(PSY_3GPP_RPEMIN*pch->band[ch][w+g].thr_quiet, fminf(pch->band[ch][w+g].thr_quiet, PSY_3GPP_RPELEV*pch->prev_band[ch][w+g].thr_quiet));
                pch->band[ch][w+g].thr = FFMAX(pch->band[ch][w+g].thr, pch->band[ch][w+g].thr_quiet * 0.25);
            }
        }
    }

    // M/S detection - 5.5.2 "Mid/Side Stereo"
    if(chans > 1 && cpe->common_window){
        start = 0;
        for(w = 0; w < cpe->ch[0].ics.num_windows*16; w += 16){
            for(g = 0; g < cpe->ch[0].ics.num_swb; g++){
                double en_m = 0.0, en_s = 0.0, ff_m = 0.0, ff_s = 0.0, minthr;
                float m, s;

                cpe->ms_mask[w+g] = 0;
                if(pch->band[0][w+g].energy == 0.0 || pch->band[1][w+g].energy == 0.0)
                    continue;
                for(i = 0; i < cpe->ch[0].ics.swb_sizes[g]; i++){
                    m = cpe->ch[0].coeffs[start+i] + cpe->ch[1].coeffs[start+i];
                    s = cpe->ch[0].coeffs[start+i] - cpe->ch[1].coeffs[start+i];
                    en_m += m*m;
                    en_s += s*s;
                }
                en_m /= 262144.0*4.0;
                en_s /= 262144.0*4.0;
                minthr = FFMIN(pch->band[0][w+g].thr, pch->band[1][w+g].thr);
                if(minthr * minthr * pch->band[0][w+g].energy * pch->band[1][w+g].energy  >= (pch->band[0][w+g].thr * pch->band[1][w+g].thr * en_m * en_s)){
                    cpe->ms_mask[w+g] = 1;
                    pch->band[0][w+g].energy = en_m;
                    pch->band[1][w+g].energy = en_s;
                    pch->band[0][w+g].thr = en_m * 0.001258925f;
                    pch->band[1][w+g].thr = en_s * 0.001258925f;
                    for(i = 0; i < cpe->ch[0].ics.swb_sizes[g]; i++){
                        m = cpe->ch[0].coeffs[start+i] + cpe->ch[1].coeffs[start+i];
                        s = cpe->ch[0].coeffs[start+i] - cpe->ch[1].coeffs[start+i];
                        ff_m += sqrt(fabs(m));
                        ff_s += sqrt(fabs(s));
                    }
                    pch->band[0][w+g].ffac = ff_m / 32.0;
                    pch->band[1][w+g].ffac = ff_s / 32.0;
                }
            }
        }
    }

    for(ch = 0; ch < chans; ch++){
        pch->a[ch] = pch->b[ch] = pch->pe[ch] = pch->thr[ch] = 0.0f;
        for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16){
            for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                if(pch->band[ch][w+g].energy != 0.0)
                    calc_pe(&pch->band[ch][w+g], cpe->ch[ch].ics.swb_sizes[g]);
                if(pch->band[ch][w+g].thr < pch->band[ch][w+g].energy){
                    pch->a[ch]   += pch->band[ch][w+g].a;
                    pch->b[ch]   += pch->band[ch][w+g].b;
                    pch->pe[ch]  += pch->band[ch][w+g].pe;
                    pch->thr[ch] += pch->band[ch][w+g].thr;
                }
            }
        }
    }

    switch(PSY_MODEL_MODE(apc->flags)){
    case PSY_MODE_CBR:
    case PSY_MODE_ABR:
        //bitrate reduction - 5.6.1 "Reduction of psychoacoustic requirements"
        if(PSY_MODEL_MODE(apc->flags) != PSY_MODE_ABR){
            pctx->reservoir += pctx->avg_bits - apc->avctx->frame_bits;
            bits_avail = pctx->avg_bits + pctx->reservoir;
            bits_avail = FFMIN(bits_avail, pctx->avg_bits * 1.5);
            pe_target = 1.18f * bits_avail / apc->avctx->channels * chans;
        }else{
            pe_target = pctx->avg_bits / apc->avctx->channels * chans;
        }
        for(i = 0; i < 2; i++){
            float t0, pe, r, a0 = 0.0f, pe0 = 0.0f, b0 = 0.0f;
            for(ch = 0; ch < chans; ch++){
                a0  += pch->a[ch];
                b0  += pch->b[ch];
                pe0 += pch->pe[ch];
            }
            if(pe0 == 0.0f) break;
            t0 = pow(2.0, (a0 - pe0)       / (4.0 * b0));
            r  = pow(2.0, (a0 - pe_target) / (4.0 * b0)) - t0;

            //add correction factor to thresholds and recalculate perceptual entropy
            for(ch = 0; ch < chans; ch++){
                pch->a[ch] = pch->b[ch] = pch->pe[ch] = pch->thr[ch] = 0.0;
                pe = 0.0f;
                for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16){
                    for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                        pch->band[ch][w+g].thr = modify_thr(pch->band[ch][w+g].thr, r);
                        calc_pe(&pch->band[ch][w+g], cpe->ch[ch].ics.swb_sizes[g]);
                        if(pch->band[ch][w+g].thr < pch->band[ch][w+g].energy){
                            pch->a[ch]   += pch->band[ch][w+g].a;
                            pch->b[ch]   += pch->band[ch][w+g].b;
                            pch->pe[ch]  += pch->band[ch][w+g].pe;
                            pch->thr[ch] += pch->band[ch][w+g].thr;
                        }
                    }
                }
            }
        }

        //determine scalefactors - 5.6.2 "Scalefactor determination"
        for(ch = 0; ch < chans; ch++){
            prev_scale = -1;
            for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16){
                for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                    cpe->ch[ch].zeroes[w+g] = pch->band[ch][w+g].thr >= pch->band[ch][w+g].energy;
                    if(cpe->ch[ch].zeroes[w+g]) continue;
                    //spec gives constant for lg() but we scaled it for log2()
                    cpe->ch[ch].sf_idx[w+g] = (int)(2.66667 * (log2(6.75*pch->band[ch][w+g].thr) - log2(pch->band[ch][w+g].ffac)));
                    if(prev_scale != -1)
                        cpe->ch[ch].sf_idx[w+g] = av_clip(cpe->ch[ch].sf_idx[w+g], prev_scale - SCALE_MAX_DIFF, prev_scale + SCALE_MAX_DIFF);
                    prev_scale = cpe->ch[ch].sf_idx[w+g];
                }
            }
        }
        break;
    case PSY_MODE_QUALITY:
        for(ch = 0; ch < chans; ch++){
            start = 0;
            for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16){
                for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                    if(pch->band[ch][w+g].thr >= pch->band[ch][w+g].energy){
                        cpe->ch[ch].sf_idx[w+g] = 0;
                        cpe->ch[ch].zeroes[w+g] = 1;
                    }else{
                        cpe->ch[ch].zeroes[w+g] = 0;
                        cpe->ch[ch].sf_idx[w+g] = (int)(2.66667 * (log2(6.75*pch->band[ch][w+g].thr) - log2(pch->band[ch][w+g].ffac)));
                        while(cpe->ch[ch].sf_idx[w+g] > 3){
                            float dist = calc_distortion(cpe->ch[ch].coeffs + start, cpe->ch[ch].ics.swb_sizes[g], SCALE_ONE_POS + cpe->ch[ch].sf_idx[w+g]);
                            if(dist < pch->band[ch][w+g].thr) break;
                            cpe->ch[ch].sf_idx[w+g] -= 3;
                        }
                    }
                    start += cpe->ch[ch].ics.swb_sizes[g];
                }
            }
        }
        break;
    }

    //limit scalefactors
    for(ch = 0; ch < chans; ch++){
        int min_scale = 256;
        for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16)
            for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                if(cpe->ch[ch].zeroes[w + g]) continue;
                min_scale = FFMIN(min_scale, cpe->ch[ch].sf_idx[w + g]);
            }
        for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16)
            for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                if(cpe->ch[ch].zeroes[w + g]) continue;
                cpe->ch[ch].sf_idx[w + g] = FFMIN(cpe->ch[ch].sf_idx[w + g], min_scale + SCALE_MAX_DIFF);
            }
        for(w = 0; w < cpe->ch[ch].ics.num_windows*16; w += 16)
            for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                if(cpe->ch[ch].zeroes[w + g])
                    cpe->ch[ch].sf_idx[w + g] = 256;
                else
                    cpe->ch[ch].sf_idx[w + g] = av_clip(SCALE_ONE_POS + cpe->ch[ch].sf_idx[w + g], 0, SCALE_MAX_POS);
            }

        //adjust scalefactors for window groups
        w = 0;
        for(wg = 0; wg < cpe->ch[ch].ics.num_window_groups; wg++){
            int min_scale = 256;

            for(g = 0; g < cpe->ch[ch].ics.num_swb; g++){
                for(i = w; i < w + cpe->ch[ch].ics.group_len[wg]*16; i += 16){
                    if(cpe->ch[ch].zeroes[i + g]) continue;
                    min_scale = FFMIN(min_scale, cpe->ch[ch].sf_idx[i + g]);
                }
                for(i = w; i < w + cpe->ch[ch].ics.group_len[wg]*16; i += 16)
                    cpe->ch[ch].sf_idx[i + g] = min_scale;
            }
            w += cpe->ch[ch].ics.group_len[wg] * 16;
        }
    }

    memcpy(pch->prev_band, pch->band, sizeof(pch->band));
    psy_create_output(apc, cpe, chans);
}

static av_cold void psy_3gpp_end(AACPsyContext *apc)
{
    Psy3gppContext *pctx = (Psy3gppContext*) apc->model_priv_data;
    av_freep(&pctx->ch);
    av_freep(&apc->model_priv_data);
}

static const AACPsyModel psy_models[AAC_NB_PSY_MODELS] =
{
    {
       "Test model",
        NULL,
        psy_test_window,
        psy_test_process,
        NULL,
    },
    {
       "3GPP TS 26.403-inspired model",
        psy_3gpp_init,
        psy_3gpp_window,
        psy_3gpp_process,
        psy_3gpp_end,
    },
};

int av_cold ff_aac_psy_init(AACPsyContext *ctx, AVCodecContext *avctx,
                            enum AACPsyModelType model, int elements, int flags,
                            const uint8_t *bands1024, int num_bands1024,
                            const uint8_t *bands128,  int num_bands128)
{
    int i;

    if(model < 0 || model >= AAC_NB_PSY_MODELS){
         av_log(avctx, AV_LOG_ERROR, "Invalid psy model\n");
         return -1;
    }

#ifndef CONFIG_HARDCODED_TABLES
   for (i = 0; i < 316; i++)
        ff_aac_pow2sf_tab[i] = pow(2, (i - 200)/4.);
#endif /* CONFIG_HARDCODED_TABLES */

    ctx->avctx = avctx;
    ctx->flags = flags;
    ctx->bands1024 = bands1024;
    ctx->num_bands1024 = num_bands1024;
    ctx->bands128 = bands128;
    ctx->num_bands128 = num_bands128;
    ctx->model = &psy_models[model];

    if(ctx->flags & PSY_MODEL_NO_ST_ATT || PSY_MODEL_MODE(ctx->flags) == PSY_MODE_QUALITY){
        ctx->flags |= PSY_MODEL_NO_ST_ATT;
        ctx->stereo_att = 0.5f;
    }else{
        ctx->stereo_att = av_clipf(avctx->bit_rate / elements / 192000.0, 0.0f, 0.5f);
    }
    if(ctx->flags & PSY_MODEL_NO_LOWPASS || PSY_MODEL_MODE(ctx->flags) == PSY_MODE_QUALITY){
        ctx->flags |= PSY_MODEL_NO_LOWPASS;
    }else{
        int cutoff;
        cutoff = avctx->bit_rate / elements / 8;
        if(ff_lowpass_filter_init_coeffs(&ctx->lp_coeffs, avctx->sample_rate/2, cutoff) < 0){
            ctx->flags |= PSY_MODEL_NO_LOWPASS;
        }else{
            ctx->lp_state = av_mallocz(sizeof(LPFilterState) * elements * 2);
        }
    }
    if(ctx->model->init)
        return ctx->model->init(ctx, elements);
    return 0;
}

void ff_aac_psy_suggest_window(AACPsyContext *ctx, int16_t *audio, int16_t *la, int tag, int type, ChannelElement *cpe)
{
    ctx->model->window(ctx, audio, la, tag, type, cpe);
}

void ff_aac_psy_analyze(AACPsyContext *ctx, int tag, int type, ChannelElement *cpe)
{
    ctx->model->process(ctx, tag, type, cpe);
}

void av_cold ff_aac_psy_end(AACPsyContext *ctx)
{
    av_freep(&ctx->lp_state);
    if(ctx->model->end)
        return ctx->model->end(ctx);
}

void ff_aac_psy_preprocess(AACPsyContext *ctx, int16_t *audio, int16_t *dest, int tag, int type)
{
    int chans = type == TYPE_CPE ? 2 : 1;
    const int chstride = ctx->avctx->channels;
    int i, ch;
    float t[2];

    if(chans == 1){
        for(ch = 0; ch < chans; ch++){
            for(i = 0; i < 1024; i++){
                dest[i * chstride + ch] = audio[i * chstride + ch];
            }
        }
    }else{
        for(i = 0; i < 1024; i++){
            if(ctx->flags & PSY_MODEL_NO_ST_ATT){
                for(ch = 0; ch < 2; ch++)
                    t[ch] = audio[i * chstride + ch];
            }else{
                t[0] = audio[i * chstride + 0] * (0.5 + ctx->stereo_att) + audio[i * chstride + 1] * (0.5 - ctx->stereo_att);
                t[1] = audio[i * chstride + 0] * (0.5 - ctx->stereo_att) + audio[i * chstride + 1] * (0.5 + ctx->stereo_att);
            }
            if(!(ctx->flags & PSY_MODEL_NO_LOWPASS)){
                LPFilterState *is = (LPFilterState*)ctx->lp_state + tag*2;
                for(ch = 0; ch < 2; ch++)
                    t[ch] = ff_lowpass_filter(&ctx->lp_coeffs, is + ch, t[ch]);
            }
            for(ch = 0; ch < 2; ch++)
                dest[i * chstride + ch] = av_clip_int16(t[ch]);
        }
    }
}




More information about the ffmpeg-devel mailing list