[FFmpeg-devel] [PATCH v2 0/4] Add hevc alpha layer decoding basic support

[Zhao Zhili]

From: Zhao Zhili <zhilizhao at tencent.com>

v2:
1. Check vps_max_layers and vps_max_layer_id inside hevc decoder
2. Add hevc_extract_layer bsf to do the job rather than modifing hevc_metadata,
3. Check vps_max_layers_minus1 and vps_max_layer_id inside bsf
4. Update vps_max_layers_minus1 and vps_max_layer_id when rewrite as base layer
5. vps_extension should be updated rather than drop entirely, add a TODO

---------------------------------------------
1. Don't drop nal->nuh_layer_id > 0 until decoder.
2. Make hevcdec able to decode a selected layer.
3. Add hevc_extract_layer bsf to extract a selected layer.

Partially fix #7965. There is a sample in ticket #7965 which can be used to test
the patchset. Or use videotoolbox encoder to create a sample (FFmpeg needs to be
patched first, "avutil/hwcontext_videotoolbox: BGRA should be full range"
http://ffmpeg.org/pipermail/ffmpeg-devel/2023-January/305252.html), e.g.,

./ffmpeg -i cctv.mp4 \
    -i pig.mp4 \
    -c:v hevc_videotoolbox \
    -alpha_quality 0.75 \
    -filter_complex '[1:v]scale=640x480[v1];[0:v][v1]alphamerge[v2]' \
    -map [v2] -an -t 60 \
    -pix_fmt bgra \
    -b:v 4M \
    -tag:v hvc1 \
    alpha.mp4 -y


Now it's able to use two decoders to decode the base layer and alpha
layer, and then combine the results, e.g.,

./ffmpeg -i alpha.mp4 \
	-strict -2  -nuh_layer_id 1 -i alpha.mp4 \
	-frames 60 \
	-filter_complex '[0:v][1:v]alphamerge[vout]' \
	-map '[vout]' -an test-%d.png

It only works with software decoder.

hevc_extract_layer can be used together with hardware decoder. The following
example is how to extract the alpha layer and rewrite as base layer:

./ffmpeg -i alpha.mp4 \
   -an -c:v copy \
   -bsf:v hevc_extract_layer=nuh_layer_id=1 \
   test.mp4

Zhao Zhili (4):
  avcodec/hevc_parse: keep nal->nuh_layer_id > 0
  avcodec/cbs_h2645: keep nal->nuh_layer_id > 0
  avcodec/hevcdec: add nuh_layer_id option
  avcodec: add hevc_extract_layer bsf

 configure                           |   1 +
 doc/bitstream_filters.texi          |  12 +++
 libavcodec/Makefile                 |   1 +
 libavcodec/bitstream_filters.c      |   1 +
 libavcodec/cbs_h2645.c              |   3 -
 libavcodec/hevc_extract_layer_bsf.c | 126 ++++++++++++++++++++++++++++
 libavcodec/hevc_parse.c             |   2 -
 libavcodec/hevcdec.c                |  56 ++++++++++++-
 libavcodec/version.h                |   2 +-
 9 files changed, 197 insertions(+), 7 deletions(-)
 create mode 100644 libavcodec/hevc_extract_layer_bsf.c

-- 
2.25.1