[FFmpeg-devel] [RFC PATCH 0/2] Make fate tests succeed with zlib-ng

[Alexander Strasser]

On 2024-10-18 15:55 +0100, Derek Buitenhuis wrote:
> On 10/18/2024 1:02 PM, Alexander Strasser via ffmpeg-devel wrote:
> > Alexander Strasser (2):
> >   Reapply "tests/fate: disable compression for zlib-based codecs"
> >   fate: Make it possible to have alternative reference files
>
> I want to add this here, since I brought it up on IRC.
>
> I think this entire saga has been pretty silly, and a workaround for
> what is fundementally a bad test. Testing the exact contents and size
> of encoded output here is fundementally the wrong thing to test and
> provides no value as a test. I'd argue it provides negative value.
>
> For a lossles encoder, especially one that uses external deps, but this
> applies to internal ones too, almost no value is provided by ensuring
> that its output is bit-exact - I'd actually consider it negative value.
[...]

Regarding especially the general assertion, there is still the
question: How should the tests be done? With which software?


  Alexander