[FFmpeg-devel] [PATCH 1/3] avcodec/x86/vvc/vvc_alf: fix integer overflow

[Andreas Rheinhardt]

toqsxw at outlook.com:
> From: Wu Jianhua <toqsxw at outlook.com>
> 
> Some tests fails with certain seeds
> 
> tests/checkasm/checkasm 2325607578 --test=vvc_alf
> checkasm: using random seed 2325607578
> AVX2:
>     vvc_alf_filter_luma_120x20_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x24_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x28_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x32_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x36_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x40_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x44_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x48_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x52_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x56_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x60_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x64_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x68_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x72_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x76_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x80_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x84_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x88_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x92_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x96_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x100_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x104_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x108_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x112_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x116_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x120_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x124_12_avx2 (vvc_alf.c:104)
>     vvc_alf_filter_luma_120x128_12_avx2 (vvc_alf.c:104)
>   - vvc_alf.alf_filter   [FAILED]
>   - vvc_alf.alf_classify [OK]
> checkasm: 28 of 9216 tests have failed
> 
> Reported-by: James Almer <jamrial at gmail.com>
> Signed-off-by: Wu Jianhua <toqsxw at outlook.com>
> ---
>  libavcodec/x86/vvc/vvc_alf.asm | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/libavcodec/x86/vvc/vvc_alf.asm b/libavcodec/x86/vvc/vvc_alf.asm
> index 71e821c27b..91f158bac9 100644
> --- a/libavcodec/x86/vvc/vvc_alf.asm
> +++ b/libavcodec/x86/vvc/vvc_alf.asm
> @@ -278,7 +278,9 @@ SECTION .text
>      psrad             m0, SHIFT + 3
>      psrad             m1, SHIFT + 3
>  %%shift_end:
> +%if ps == 1
>      packssdw          m0, m0, m1
> +%endif
>  %endmacro
>  
>  ; FILTER_VB(line)
> @@ -356,7 +358,18 @@ SECTION .text
>  
>      FILTER_VB         xq
>  
> +    ; sum += curr
> +%if ps == 1
>      paddw             m0, m2
> +%else
> +    vpunpcklqdq      m11, m2, m2
> +    vpunpckhqdq      m12, m2, m2
> +    vpunpcklwd       m11, m11, m14
> +    vpunpcklwd       m12, m12, m14
> +    paddd             m0, m11
> +    paddd             m1, m12
> +    packssdw          m0, m0, m1
> +%endif
>  
>      ; clip to pixel
>      CLIPW             m0, m14, m15

Can this happen with real inputs (like when called from the decoder)? If
not, then the test needs to be made more realistic.
Anyway, what is the performance impact of this?

- Andreas