[FFmpeg-trac] #6006(avfilter:reopened): FPS filter stutters when converting framerate

[FFmpeg]

#6006: FPS filter stutters when converting framerate
------------------------------------+------------------------------------
             Reporter:  Misaki      |                    Owner:
                 Type:  defect      |                   Status:  reopened
             Priority:  normal      |                Component:  avfilter
              Version:  git-master  |               Resolution:
             Keywords:  fps         |               Blocked By:
             Blocking:              |  Reproduced by developer:  1
Analyzed by developer:  0           |
------------------------------------+------------------------------------

Comment (by Misaki):

 Maybe people don't discuss design choices in comments here, and instead
 use other venues like IRC. But I feel out of place there, particularly
 since I don't know any programming languages, and have not found it to be
 useful.

 Current details of fps filter, as far as I can gather from limited testing
 without understanding the code:
 Filter gets start point from user or first frame. I don't understand this
 comment or the code that follows it:
 {{{
 +     * The dance with offsets is required to match the rounding behaviour
 of the
 +     * previous version of the fps filter when using the start_time
 option. */"
 }}}
 But anyway, having timestamps 0.04 sec later seems to cause the same
 output as using start_time=-0.04.

 The input timestamps are rounded up or down to an output timestamp, as
 opposed to rounding the output timestamps to an input one.

 The frame with the last input timestamp corresponding to an output
 timestamp is used for that output frame.

 So with output r=10 and round=near, two input frames at 0.04 and 1.04 (1
 fps with offset added), the output frame at 1.0 uses the second input
 frame. All frames before it duplicate the first frame.

 With 100 fps input and 10 fps output, the out_frame at 1.0 uses in_frame
 at 1.04 for round=near; in_frame at 1.09 for round=down; in_frame at 1.0
 for round=up.

 One concern for design and use is whether the video remains synchronized.
 With the default round=near, the frame that's displayed for [1,1.1> is an
 average of what's happening during that time. At least one media player,
 totem, shows the upcoming frame if you pause it, which may or may not be a
 bug. If a 30-fps video is converted to 60 fps by duplicating frames and
 the user is processing the 60-fps version, then the second of each frame
 could be slightly higher quality, though I suppose it could be the
 opposite if the first displayed frame is a B-frame. I would tend to say
 that it's better for content to be displayed late (or rather, "on time")
 than early, but this might be my biases like the way low-fps lag works in
 computer games. Low fps triggers an expectation of what the content should
 be, even if it isn't being updated.

 But more important is whether there are unwanted duplicates or skipping of
 frames. Suppose a user wants the input frame at 1.0 to be displayed at 1.0
 in the output. Currently they could use round=up, which (maybe
 counterintuitively) does this. But if it causes every third frame to
 jitter by being one input frame early, this filter option is not useful.

 Out of the three options (for positive timestamps), round=near is in the
 middle. Output frames should be displayed around the input timestamps used
 for this method. The other two rounding methods can return frames that are
 either later or earlier than this time.

 With the 100 fps to 10 fps example, either start_time=0:round=near would
 continue to select input frames at 0.04, 0.14, 0.24 etc. but change to
 display them at those same times by using a timebase lower than 1/fps
 (timestamp interval higher than 1), or use the frames at 0, 0.1, 0.2 etc.
 and display them at those times.

 Currently, round=near appears to work like the other methods, by
 processing all input frames up to the next output frame's time interval
 then outputting the last one. Interval for 10 fps, 1.0 timestamp appears
 to be [0.95,1.05>, so if input fps is 30.1 with frames at 1.03 and 1.063,
 the frame at 1.03 will be used even though 1.063 is closer to 1.05. The
 input frames are associated with the nearest output frame; the output
 frame does not select the input frame that's closest.

 I think this should be changed so the output frame at time X is, in fact,
 the input frame closest to X, but the other rounding methods also have
 this bias. If only 'near' is adjusted, then on average it might be
 slightly above the average of 'round=up' and 'round=down'.

 But anyway, as an adjustment to current code, there would be a check to
 see if each input frame is closer to the output frame, instead of just
 using the last one. Then adjust the intervals for round={up,down} down by
 half a frame.

 This only fixes jittering or stuttering for the simple case, as discussed
 above.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/6006#comment:5>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker