[FFmpeg-devel] [RFC] fast audio interleaving

Andreas Öman andreas
Thu Sep 27 19:20:49 CEST 2007

Rich Felker wrote:
> I agree strongly with this proposed approach. "Planar" format is
> preferred anyway for most filtering operations, and makes everything
> audio-channel-order-agnostic just like planar video makes things
> color-channel-order-agnostic.

Yes, av_resample() is a good example of this.

Hoping that we end up with something more that just a discussion
I'd like to raise various topics that i think need to be addressed
in a new audio API. I can certainly help working on some of these
things but i doubt that I'll manage to pull all of this ashore all by

* Flexible sample format for codecs to choose from.

'enum SampleFormat' is already around for this purpose.
Should it be extended with P-postfixed formats to indicate
planar mode?

I also think it's useful to be able to get float output from
codecs that produce it, since floats are used in various
sound frameworks such as jack, fmod, etc.

Thus, the float_to_int16-stuff would have to be revamped in some way.

Today, each codec sets 'add_bias' and 'mod_bias' depending on whether
or not dsp->float_to_int16 is the c-version or a SIMD-version.
IMHO, this isn't exactly nice. Also, i just discovered (by a very
unpleasant experience in my head-phones :-) ) that the DTS-codec
does not worry about this at all. It expects the float_to_int16
code to behave like the sse-version. This should of course be
fixed either way. I can have a look at it later (or tomorrow)

However, the long term question is whether or not to remove the
pre-float_to_int-conversion add&multiply entirely from the codecs
responsibility and rather keep it all in a future
audio-output-sample-format conversion code.

Either way, new encode/decode APIs that delivers/accepts audio in the
codec's "native" format is needed.

avcodec_decode_audio2 and avcodec_encode_audio should be reworked
to use the new api and have built-in conversion functions to
make int16 interleaved data.

It might be too much work to do a fork-lift upgrade of all
audio codecs. Perhaps it's a good idea to add a 
CODEC_CAP_USE_SAMPLE_FORMAT that tells the API how to behave?

* Channel ordering.

This was discussed a while ago, and a solution proposed by Justin,
but unfortunately that thread faded out.
Apart from the extra bells and whistles this would
give there are two small issues that I'm aware of that is a bit
annoying today:

1) avctx->channels is abused. This was addressed recently
by Reimar D?ffinger. However, never finished. Perhaps
for the better good, i dunno. Justin's solution is IMHO
better. But then again, if something dosent hit svn it doesn't
matter how good or bad it is.

2) AC3 and DTS does not output channels in the same order when in
5.1 mode.

AC3 does Left, Center, Right, SL, SR, LFE
DTS (and libfaad) does Center, Left, Right, SL, SR, LFE

* Resampling, Downmixing/Rematrixing

Perhaps this rather should reside in libavfilter.
I haven't put much thought into this, neither to I know
how well prepared libavfilter is to handle audio.
The two bullets above is probably enough for now :-)

... This ended up being a little more than I initially thought.
It's like opening Pandora's box.

The comments, ideas, flames -window is open...

More information about the ffmpeg-devel mailing list