[FFmpeg-devel] [RFC PATCH 0/2] Make fate tests succeed with zlib-ng

[Derek Buitenhuis]

On 10/18/2024 1:02 PM, Alexander Strasser via ffmpeg-devel wrote:
> Alexander Strasser (2):
>   Reapply "tests/fate: disable compression for zlib-based codecs"
>   fate: Make it possible to have alternative reference files

I want to add this here, since I brought it up on IRC.

I think this entire saga has been pretty silly, and a workaround for
what is fundementally a bad test. Testing the exact contents and size
of encoded output here is fundementally the wrong thing to test and
provides no value as a test. I'd argue it provides negative value.

For a lossles encoder, especially one that uses external deps, but this
applies to internal ones too, almost no value is provided by ensuring
that its output is bit-exact - I'd actually consider it negative value.
It's testing the wrong thing (that output remains correct, decodable,
metadata, etc.) and only ensures no changes to anything can ever be
made without a reference update.

- Derek