[FFmpeg-devel] [WIP PATCH 0/1] avcodec/aacenc: improve bit_rate_tolerance=0

[Pauli Virtanen]

In the FFMPEG native AAC encoder, bit_rate_tolerance=0 (-bt:a 0)
enforces a strict bitrate mode, where maximum length of the AAC frame
(frame_bits) is capped below nominal bitrate (rate_bits).

It currently seems to have some issues:

- encoder can get stuck in an infinite loop
- sound quality issues, as it's never scaling lambda above the initial value
- bitrates can be far below bit_rate target

Some samples for reproducing:
https://gitlab.freedesktop.org/pvir/repros/-/tree/main/ffmpeg-aacenc-bt

The option was added in the native AAC encoder in commit b92af7b64e7acd,
and it is needed in Bluetooth A2DP AAC encoding, where each frame must
fit into MTU or it cannot be sent to receiver. (AFAIK, receivers don't
support fragmentation, and everyone including Android seems to be just
using FDK-AAC with its equivalent AACENC_PEAK_BITRATE option.)

I would like to allow using FFMPEG native AAC encoder for Bluetooth AAC
audio in PipeWire instead of FDK-AAC that we require right now, but the
above quality problems of the encoder are a problem for this right now.
(If you want to test PW+FFMPEG-AAC yourself, you can build PipeWire from
https://gitlab.freedesktop.org/pvir/pipewire/-/commits/libav
try it out with some BT headphones that support AAC.)

It would be great to have the situation improved. Here's a patch trying
to address some of the issues with bit_rate_tolerance=0, by doing the
following:

- target a bitrate slightly below frame_bits cap with the usual code path

- if frame_bits exceeds cap, find good lambda with a root finding method
  (usually converges in a few iterations).

The sound quality with this patch seems better than before, as now it
maintains sufficient bitrate.

Problems remain, though:

- it appears the resulting frame_bits depends also on some other state
  than s->lambda. iteration with lambda1, lambda2>lambda1, and then again
  with lambda1 can produce different frame_bits on the two lambda1
  iterations. Is there some state that is modified across iterations?

  The problem here is that the root finding sometimes cannot go back to
  OK value of lambda it found earlier, so the frame ends up being too
  large.

- for some inputs, frame_bits appears to not be reduced much by any
  value of lambda.  This is the case where it previously went to
  nonterminating loop.  Now I just made it return the too large frame in
  this case, and let the caller deal with it...

It doesn't look like that can be fixed by just playing with lambda,
maybe some other changes in the encoder are needed.  There's a sample of
such "adversarial" input in the above link.

I'm not at the moment familiar enough with AAC encoding to know how to
fix the above, that'd need some more work / additional hints.

Pauli Virtanen (1):
  avcodec/aacenc: improve bit_rate_tolerance=0

 libavcodec/aacenc.c | 153 +++++++++++++++++++++++++++++++++++---------
 libavcodec/aacenc.h |   1 +
 2 files changed, 124 insertions(+), 30 deletions(-)

-- 
2.46.0