[FFmpeg-devel] [PATCH 10/14] swscale/range_convert: fix mpeg ranges in yuv range conversion for non-8-bit pixel formats

[Michael Niedermayer]

On Mon, Sep 23, 2024 at 02:40:13PM +0200, Ramiro Polla wrote:
> There is an issue with the constants used in YUV to YUV range conversion,
> where the upper bound is not respected when converting to mpeg range.
> 
> With this commit, the constants are calculated at runtime, depending on
> the bit depth. This approach also allows us to more easily understand how
> the constants are derived.
> 
> For bit depths <= 14, the number of fixed point bits has been set to 14
> for all conversions, to simplify the code.
> For bit depths > 14, the number of fixed points bits has been raised and
> set to 18, to allow for the conversion to be accurate enough for the mpeg
> range to be respected.
> 
> The convert functions now take the conversion constants (amax, coeff,
> and offset) as function arguments.
> For bit depths <= 14, amax is 16-bit and offset is 32-bit.
> For bit depths > 14, amax is 32-bit and offset is 64-bit.
> 
> NOTE: all simd optimizations for range_convert have been disabled.
>       they will be re-enabled when they are fixed for each sub-arch.
> 
> NOTE2: the same issue still exists in rgb2yuv conversions, which is not
>        addressed in this commit.
> ---

This appears to break:

make fate-filter-owdenoise-sample
TEST    filter-owdenoise-sample
stddev:12247.77 PSNR: 14.57 MAXDIFF:65280 bytes:   576000/   576000
MAXDIFF: |65280 - 1| >= 3539
Test filter-owdenoise-sample failed. Look at tests/data/fate/filter-owdenoise-sample.err for details.
make: *** [tests/Makefile:312: fate-filter-owdenoise-sample] Error 1

What peformance impact do non constants have ?

[...]

thx
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Asymptotically faster algorithms should always be preferred if you have
asymptotical amounts of data
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20240924/d086a0d6/attachment.sig>