[FFmpeg-devel] [PATCH] fate: Add a target for listing failed tests

[Martin Storsjö]

On Sun, 1 Dec 2024, Alexander Strasser via ffmpeg-devel wrote:

> Looks good and useful to me.

Thanks!

> Just a few ideas follow:
>
>> diff --git a/doc/build_system.txt b/doc/build_system.txt
>> index 0b1b0c2054..91c7a5e9af 100644
>> --- a/doc/build_system.txt
>> +++ b/doc/build_system.txt
>> @@ -30,6 +30,12 @@ fate
>>  fate-list
>>      List all fate/regression test targets.
>>
>> +fate-list-failing
>> +    List the fate tests that failed the last time they were executed.
>> +
>> +fate-clear-results
>> +    Remove the test results from previous test executions.
>> +
>
> Would it be better to use the same description as int fate.texi ?

Sure, I can add that extra parenthesis.

>> diff --git a/tests/Makefile b/tests/Makefile
>> index 9b70145015..adb0799328 100644
>> --- a/tests/Makefile
>> +++ b/tests/Makefile
>> @@ -313,6 +313,12 @@ $(FATE): $(FATE_UTILS:%=tests/%$(HOSTEXESUF)) | $(FATE_OUTDIRS)
>>  fate-list:
>>  	@printf '%s\n' $(sort $(FATE))
>>
>> +fate-list-failing:
>> +	@cat tests/data/fate/*.rep | cut -f 1-2 -d : | grep -v :0 | sed 's/:.*//;s/^/fate-/'
>> +
>> +fate-clear-results:
>> +	@rm -f tests/data/fate/*.rep
>> +
>>  coverage.info: TAG = LCOV
>>  coverage.info:
>>  	$(M)lcov -q -d $(CURDIR) -b $(patsubst src%,./,$(SRC_LINK)) --capture | \
>> --
>
> Maybe the attached patch would make sense on top of your changes?

Nice, thanks. I'm not very familiar with awk so I seldom write such things 
myself, but this does indeed seem neater.

As a second question, do you have any opinion on the fate target names? 
Should the second be "fate-clear-failing" to match the list-failing one? 
But technically it's not only about clearing the failing ones but clearing 
all results...

// Martin