[FFmpeg-user] Disparity in VMAF scores

Mon May 9 22:16:43 EEST 2022

I'm not familiar with that tool, but one thing I have noticed with some
3rd-party VMAF tools is the scores they generate aren't always accurate if
the material in question is not 1080p for the standard model, or 4k in the
case of the 4k model.  So if your source is smaller than 1080p I wouldn't
trust the score.  The official vmaf.exe has you tell it what the resolution
of the source is and the calculations get adjusted accordingly.  You can
end up with a wildly different score if the resolution isn't taken into
account during the score calculation, and some 3rd-party tools seem to get
this part wrong somehow and you end up with a score such as you're seeing,
somewhere in the 70s, when with the official tool and the proper resolution
set it would probably give you something in the 90s.  I noticed this when I
was testing one day with some 4k stuff, some 1080p stuff, and some 480p
stuff with a DVD source.  The DVD stuff was returning scores in the 70s
even when encoded with x264 and CRF 16 or 17, which seemed very strange to
me.  So instead I used ffmpeg to dump to yuv and then used vmaf.exe on
those files to double check and the scores came back in the upper mid 90s,
which made much more sense.  Since VMAF determines your viewing distance
with a multiple of the file's vertical resolution as one of the factors it
is necessary that the calculations be fed the proper information with
regard to the file's resolution or the score will be incorrect.  I'm not
saying your 3rd-party tool in particular is making this mistake for sure,
but it is possible and is worth double checking with the official tool just
to be sure you're getting sane scores.  I've seen some online VMAF guides
written by random people that state something like "You should expect lower
scores for lower resolution files such as DVD."  The actual case is
whatever tool they're using to get their scores isn't handling resolution
properly and that's why their lower resolution files are resulting in lower
scores.

On Mon, May 9, 2022 at 10:35 AM Kamaldeep Tumkur <kamaldeep.tumkur at gmail.com>
wrote:

> On Mon, 9 May 2022 at 21:49, Clayton Macleod <cherrytwist at gmail.com>
> wrote:
>
> > Using the official VMAF.exe is a bit more of a pain since you have to
> > convert to yuv first, but being able to examine the results file is
> > sometimes helpful. 75 sounds like something may still be off if the files
> > don’t appear drastically different to your eye.
> >
> > --
> > Clayton Macleod
> > If no one comes from the future to stop you from doing it, then how bad
> of
> > a decision can it really be?
> >
> > > On May 9, 2022, at 1:14 AM, Kamaldeep Tumkur <
> kamaldeep.tumkur at gmail.com>
> > wrote:
> > >
> > > So my question is, does libvmaf implementation do any kind of
> detection
> > of
> > > audio in the distorted asset before computing a perceptual video
> quality
> > > score.
> > >
> > >
> > >> On Mon, 9 May 2022 at 13:37, Kamaldeep Tumkur <
> > kamaldeep.tumkur at gmail.com>
> > >> wrote:
> > >>
> > >> @Clayton, thanks for your response.
> > >>
> > >> It turned out that the external encoder was adding an audio track to
> the
> > >> encode. This was throwing the libvmaf scores off. On removing the
> audio
> > >> track and checking the encode, the score jumped from 25 to 77.64.
> > >>
> > >> Now there is basis to compare the ffmpeg and the external encode.
> > >>
> > >> Regards
> > >> Kamaldeep
> > >>
> > >>
> > >>
> > >> On Sun, 8 May 2022 at 15:37, Kamaldeep Tumkur <
> > kamaldeep.tumkur at gmail.com>
> > >> wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> I'm new to the computation of VMAF scores using libvmaf in ffmpeg.
> > While
> > >>> trying out generating scores through libvmaf, I have an issue that I
> > cannot
> > >>> seem to find an explanation for.
> > >>>
> > >>> I encoded the 'crowd_run' HQ MP4 with two different encoders. The
> > >>> resulting files were compared with the same source (original).
> > >>>
> > >>> VMAF score of ffmpeg encoded clip:
> > >>>
> > >>> [libvmaf @ 0x7fc5d8f2cb00] VMAF score: 90.644028
> > >>>
> > >>>
> > >>> Command used:
> > >>>
> > >>> ffmpeg -i crowd_run_1080p50.mp4 -c:v libx264 -g 100 -keyint_min 100
> > >>> -sc_threshold 0 -b:v 12000k -maxrate 15000k -bufsize 15000k -c:a copy
> > -f
> > >>> mpegts pass2.ts
> > >>>
> > >>>
> > >>> VMAF score for externally encoded clip:
> > >>>
> > >>> [libvmaf @ 0x7fc53d729500] VMAF score: 25.168431
> > >>>
> > >>>
> > >>>
> > >>> I just don't know how to explain this disparity in generated scores
> > when
> > >>> framerates and resolution of the source were retained. The second
> clip
> > >>> doesn't seem to show such a degradation expected through the score.
> > >>> Attaching both the encoded clips here for analysis.
> > >>>
> > >>>
> > >>> Could anyone point me to why the external encoder generates an output
> > >>> with low VMAF score. Anything in the frame structure?
> > >>>
> > >>>
> > >>> ffmpeg encode:
> > >>>
> > >>>
> > >>>
> >
> https://drive.google.com/file/d/15-9YyNhYWJoTMxdmESkc7b0mPKKUMqG7/view?usp=sharing
> > >>>
> > >>>
> > >>> external encode:
> > >>>
> > >>>
> > >>>
> >
> https://drive.google.com/file/d/1Mt2jP51KZ4vTG7SYPnJkCG1uMWqOadid/view?usp=sharing
> > >>>
> > >>>
> > >>>
> > >>> Thank you.
> > >>>
> >
> >
>
> I've been using the crowd_run y4m source at:
> https://media.xiph.org/video/derf/y4m/crowd_run_1080p50.y4m
>  Yes, I'd prefer a better score too. However, I've only been able to
> generate an encode with 77.936 at most today, with the external encoder and
> a qvbr setting. ffmpeg-based encodes are at 90.12, which is quite in the
> zone. Yes, also need to check the standalone vmaf exec, but what I've
> learned is using ffmpeg filters to scale up the HLS renditions to the
> source resolution and frame-rate along with the libvmaf filter is easier.
> Although, please note that the distorted asset with score of 77 had the
> same resolution and frame rate as the source, so no additional filtering
> was performed.
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
>

-- 
Clayton Macleod
If no one comes from the future to stop you from doing it, then how bad of
a decision can it really be?