[FFmpeg-trac] #9689(undetermined:new): Mismatched frames when using WebM containers and MP4 references

[FFmpeg]

#9689: Mismatched frames when using WebM containers and MP4 references
-------------------------------------+-------------------------------------
             Reporter:  Cosmin       |                     Type:
  Stejerean                          |  enhancement
               Status:  new          |                 Priority:  normal
            Component:               |                  Version:
  undetermined                       |  unspecified
             Keywords:               |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 WebM container forces timestamps to be rounded to the nearest ms to the
 forced timebase. This then results in mismatched frames if attempting to
 compare such a video stored in WebM against a reference in say MP4 for
 quality metric comparisons like SSIM or VMAF.

 To illustrate the video I'm attaching the exact same video in both WebM
 and MP4, both encoded from a 30fps source, with no frame rate conversions.
 Attempting to compare the two videos should result in identical SSIM of
 1.0, but due to frame mismatches the end result ends up being ~0.9

 When looking at frame by frame data we can see that every 3rd frame is
 mismatched (in this example of 30fps video, the results would be slightly
 different at other frame rates, but anything that can't be expressed
 precisely in ms would show this issues).

 {{{
 n:1 Y:1.000000 U:1.000000 V:1.000000 All:1.000000 (inf)
 n:2 Y:1.000000 U:1.000000 V:1.000000 All:1.000000 (inf)
 n:3 Y:0.788922 U:0.951481 V:0.981816 All:0.848164 (8.186263)
 n:4 Y:1.000000 U:1.000000 V:1.000000 All:1.000000 (inf)
 n:5 Y:1.000000 U:1.000000 V:1.000000 All:1.000000 (inf)
 n:6 Y:0.724944 U:0.878058 V:0.951394 All:0.788205 (6.740836)
 }}}

 essentially what happens with the timestamps (shown in seconds to
 illustrate the problem) is that in every group of 3 the first frame
 matches exactly, the second frame rounds down so is slightly earlier, and
 the 3rd frame rounds up.

 {{{
 0.0000 => 0.0000
 0.0333 => 0.0330
 0.0666 => 0.0670
 }}}


 Then when the filter is trying to frame sync the video the 3rd frame (from
 the group on the left with proper 30 fps timestamps) ends up being
 compared to the second one from the right because the 3rd frame is just
 slightly in the future. What would be desired is for framesync to instead
 find the nearest frame, so that 0.0666 is compared against 0.0670 instead
 of to 0.0330

 That's a departure from how framesync works today. Fortunately in recent
 versions of ffmpeg a warning is shown in such cases, which at least
 highlights that this will be a problem.

 {{{
 [Parsed_ssim_0 @ 0x152209740] not matching timebases found between first
 input: 1/15360 and second input 1/1000, results may be incorrect!
 }}}

 Workarounds include decoding to YUV first (at which point frames become
 compared pairwise without regards to the odd timestamp rounding but this
 requires more disk space and IO), not using WebM in the first place to
 preserve accurate frame timestamps, or re-applying an fps filter to get
 both files onto the same timebase,

 {{{
 "[0] fps=fps=30 [dst]; [1] fps=fps=30 [ref]; [dst][ref] ssim
 }}}

 But in practice I've found this to be a very common mistake when looking
 at quality metrics of videos in WebM containers, so it would be great if
 there was an easier way to get the expected behavior directly from
 framesync.
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/9689>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker