[FFmpeg-devel] [PATCH] fate: add VVC decoder tests

[Nuo Mi]

On Sat, Jan 6, 2024 at 9:13 AM James Almer <jamrial at gmail.com> wrote:

> On 1/5/2024 10:09 PM, Nuo Mi wrote:
> > On Sat, Jan 6, 2024 at 5:09 AM James Almer <jamrial at gmail.com> wrote:
> >
> >> On 1/3/2024 9:19 PM, James Almer wrote:
> >>> Signed-off-by: James Almer <jamrial at gmail.com>
> >>> ---
> >>>    tests/Makefile                                |   1 +
> >>>    tests/fate/vvc.mak                            |  50 +++++++
> >>>    tests/ref/fate/vvc-conformance-APSALF_A_2     |  13 ++
> >>>    tests/ref/fate/vvc-conformance-APSLMCS_D_1    |  37 +++++
> >>>    tests/ref/fate/vvc-conformance-APSMULT_A_4    |  53 +++++++
> >>>    tests/ref/fate/vvc-conformance-AUD_A_3        |  35 +++++
> >>>    tests/ref/fate/vvc-conformance-BUMP_A_2       |  45 ++++++
> >>>    tests/ref/fate/vvc-conformance-CROP_B_4       | 105 ++++++++++++++
> >>>    .../fate/vvc-conformance-CodingToolsSets_A_2  |   7 +
> >>>    tests/ref/fate/vvc-conformance-DCI_A_3        |   7 +
> >>>    tests/ref/fate/vvc-conformance-HRD_A_3        |  65 +++++++++
> >>>    tests/ref/fate/vvc-conformance-OPI_B_3        |   6 +
> >>>    tests/ref/fate/vvc-conformance-PHSH_B_1       |  11 ++
> >>>    tests/ref/fate/vvc-conformance-POC_A_1        |  25 ++++
> >>>    tests/ref/fate/vvc-conformance-PPS_B_1        |  69 +++++++++
> >>>    tests/ref/fate/vvc-conformance-RAP_A_1        |   6 +
> >>>    tests/ref/fate/vvc-conformance-SAO_A_3        |  65 +++++++++
> >>>    tests/ref/fate/vvc-conformance-SCALING_A_1    |  69 +++++++++
> >>>    tests/ref/fate/vvc-conformance-SLICES_A_3     |  30 ++++
> >>>    tests/ref/fate/vvc-conformance-SPS_B_1        | 133
> ++++++++++++++++++
> >>>    tests/ref/fate/vvc-conformance-STILL_B_1      |  10 ++
> >>>    tests/ref/fate/vvc-conformance-SUBPIC_A_3     |   9 ++
> >>>    tests/ref/fate/vvc-conformance-TILE_A_2       |  14 ++
> >>>    tests/ref/fate/vvc-conformance-VPS_A_3        |   6 +
> >>>    tests/ref/fate/vvc-conformance-WPP_A_3        |  54 +++++++
> >>>    tests/ref/fate/vvc-conformance-WP_A_3         |  22 +++
> >>>    tests/ref/fate/vvc-conformance-WRAP_A_4       |  14 ++
> >>>    27 files changed, 961 insertions(+)
> >>>    create mode 100644 tests/fate/vvc.mak
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-APSALF_A_2
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-APSLMCS_D_1
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-APSMULT_A_4
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-AUD_A_3
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-BUMP_A_2
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-CROP_B_4
> >>>    create mode 100644
> tests/ref/fate/vvc-conformance-CodingToolsSets_A_2
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-DCI_A_3
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-HRD_A_3
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-OPI_B_3
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-PHSH_B_1
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-POC_A_1
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-PPS_B_1
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-RAP_A_1
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-SAO_A_3
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-SCALING_A_1
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-SLICES_A_3
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-SPS_B_1
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-STILL_B_1
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-SUBPIC_A_3
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-TILE_A_2
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-VPS_A_3
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-WPP_A_3
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-WP_A_3
> >>>    create mode 100644 tests/ref/fate/vvc-conformance-WRAP_A_4
> >>
> >> This is using the samples already in the fate suite. I don't know if
> >> they give good coverage (I think i chose them to cover as much CBS
> >> methods as possible, not caring about actual decode paths, and not even
> >> that is complete), so I'd like someone more familiar with the
> >> conformance suite to suggest better samples to upload and use, too.
> >>
> >
> > Here are the clips and their sources:
> > https://github.com/ffvvc/tests/tree/main
> > passed v1 is about 194M , and each clip may have different features. It's
> > better to add them all to vvc-conformance, without name change
> > passed v2 is about 871M , we can pick some smaller clips that cover all
> bit
> > depths and color formats.
>
> Yeah, it's not so much about the size of the samples than it is about
> runtime. That many tests would make a fate run take way too much, even
> after we add assembly.
> We need to choose a subset that cover as many decoding cases as possible.
>
Good point.
If I disable the asm code on my machine, "make fate-hevc" takes about 1
minute and 26 seconds. We can use this as the baseline.
If we remove the largest 5 files in v1, we can decode it in 5 minutes and
13 seconds. We can remove more if we find it's still slow.
>From a feature perspective, TREE_A_HHI_3.bit may cover major features in
TREE_B_HHI_3.bit, TREE_C_HHI_3.bit. It's reasonable to remove them

Largest 5 files:
TRANS_B_Chipsnmedia_2.bit
TRANS_A_Chipsnmedia_2.bit
TREE_B_HHI_3.bit
LFNST_D_HHI_3.bit
TREE_C_HHI_3.bit

> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>