[FFmpeg-devel] [RFC PATCH] avcodec/nvenc: add alpha layer encoding support

[Timo Rothenpieler]

On 30.12.2022 18:50, Zhao Zhili wrote:
> From: Zhao Zhili <zhilizhao at tencent.com>
> 
> Signed-off-by: Zhao Zhili <zhilizhao at tencent.com>
> ---
> I need help. A lot of things doesn't work as Nvidia's doc:
> 
> 1. Global header is broken. NV_ENC_SEQUENCE_PARAM_PAYLOAD doesn't work with
>     multi SPS/PPS. It output VPS, SPS_0, SPS_1, PPS_0, with PPS_1 missing.
> 2. NV_ENC_LOCK_BITSTREAM.alphaLayerSizeInBytes is always 0, although the data is there.
> 3. The alpha layer bitstream is corrupted. It can be decoded with a lot of errors.
I had played around with this before as well:> 
https://github.com/BtbN/FFmpeg/commit/589f96f2a3fb0cc719d76555f09f1bc1e8cb0403

But my conclusion was similar. The feature seems to be half broken and 
on top of that poorly specified.
I eventually gave up investigating and just concluded it was broken.