[FFmpeg-devel] [PATCH 2/2] fate: add decoder test for ATRAC9

[Michael Niedermayer]

On Tue, Jul 03, 2018 at 09:35:53PM +0200, Carl Eugen Hoyos wrote:
> 2018-07-01 14:51 GMT+02:00, James Almer <jamrial at gmail.com>:
> > On 7/1/2018 8:46 AM, Rostislav Pehlivanov wrote:
> >> Signed-off-by: Rostislav Pehlivanov <atomnuker at gmail.com>
> >> ---
> >>  tests/fate/atrac.mak | 13 ++++++++++++-
> >>  1 file changed, 12 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/tests/fate/atrac.mak b/tests/fate/atrac.mak
> >> index acf79a539c..38a5f33150 100644
> >> --- a/tests/fate/atrac.mak
> >> +++ b/tests/fate/atrac.mak
> >> @@ -33,7 +33,17 @@ fate-atrac3p-2: REF =
> >> $(SAMPLES)/atrac3p/sonateno14op27-2-cut.pcm
> >>
> >>  FATE_ATRAC3P-$(call DEMDEC, OMA, ATRAC3P) += $(FATE_ATRAC3P)
> >>
> >> -FATE_ATRAC_ALL = $(FATE_ATRAC1-yes) $(FATE_ATRAC3-yes)
> >> $(FATE_ATRAC3P-yes)
> >> +FATE_ATRAC9 += fate-atrac9-1
> >> +fate-atrac9-1: CMD = pcm -i $(TARGET_SAMPLES)/atrac9/sy_title01.at9
> >> +fate-atrac9-1: REF = $(SAMPLES)/atrac9/sy_title01.at9.pcm
> >> +
> >> +FATE_ATRAC9 += fate-atrac9-2
> >> +fate-atrac9-2: CMD = pcm -i
> >> $(TARGET_SAMPLES)/atrac9/3_br144_nb10_ib10_g04_bex1_sf1_d1.at9
> >> +fate-atrac9-2: REF =
> >> $(SAMPLES)/atrac9/3_br144_nb10_ib10_g04_bex1_sf1_d1.at9.pcm
> >
> > Are these pcm files from a test suit (if any), or at least created
> > by an official binary decoder?
> 
> I believe we added support for codecs when we were not able to
> produce "official" binary output.
> 
> > I'd rather avoid adding tests using reference files created by
> > this same decoder
> 
> Why not?
> I agree that in an ideal world, we would only have known correct
> reference files but this is not the case now (we have always been
> knowingly testing "wrong" output") and, more important, I don't
> think fate is a test for correctness but much more a regression
> test to find unintended changes in FFmpeg's behaviour, don't you
> agree? Finding such unintended changes even makes sense for
> decoders that are known not to produce correct output.

The problem with raw (pcm,rgb.yuv) reference files coming from
our decoder and not teh official decoder is that they can change
(that is for lossy codecs mainly, its harder to miss a relevant bug in a
 lossless decoder)
Each time a bug is fixed in the decoder the reference can change
and the pcm/yuv/rgb file in fate samples may then need to be updated.
And we cannot change files in fate just add more files. So this can
lead to "dead" files in the samples ...


[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Awnsering whenever a program halts or runs forever is
On a turing machine, in general impossible (turings halting problem).
On any real computer, always possible as a real computer has a finite number
of states N, and will either halt in less than N cycles or never halt.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180704/303da86f/attachment.sig>