Jumping frames (wrong presentation order) when copying video (muxing) of an h264 file (to m4v/mp4) in Quicktime

[Alex__]

Description: Jumping frames when copying video of an h264 file
Version: latest trunk
URL: see below

Steps that will reproduce the problem?
ffmpeg -i sample.avi -vcodec copy sample-copy.m4v

What is the expected result?
Converted file w/o any issues.

What happens instead?
The frames "jump" in the converted file when playing it with QuickTimePlayer? X 10.1 (more) or VLC (less).

Possible workaround:
ffmpeg -i sample.avi sample-copy.m4v

Any additional information:

• Example: see below
• Command:

  $ ffmpeg -y -i sample.avi -vcodec copy sample-copy.m4v
  ffmpeg version N-30168-g6d789f5, Copyright (c) 2000-2011 the FFmpeg developers
    built on Sep 23 2011 09:20:16 with clang 2.1 (tags/Apple/clang-163.7.1)
    configuration: --enable-zlib --disable-debug --enable-libfaac --enable-libx264 --enable-gpl --enable-nonfree --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libxvid --cc=clang
    libavutil    51. 17. 0 / 51. 17. 0
    libavcodec   53. 17. 0 / 53. 17. 0
    libavformat  53. 13. 0 / 53. 13. 0
    libavdevice  53.  4. 0 / 53.  4. 0
    libavfilter   2. 43. 3 /  2. 43. 3
    libswscale    2.  1. 0 /  2.  1. 0
    libpostproc  51.  2. 0 / 51.  2. 0
  Input #0, avi, from 'sample.avi':
    Metadata:
      encoder         : transcode-1.0.7rc1
    Duration: 02:02:15.00, start: 0.000000, bitrate: 0 kb/s
      Stream #0.0: Video: h264 (Main) (H264 / 0x34363248), yuv420p, 720x576 [SAR 16:15 DAR 4:3], 25 fps, 25 tbr, 25 tbn, 50 tbc
      Stream #0.1: Audio: mp3 (U[0][0][0] / 0x0055), 48000 Hz, stereo, s16, 192 kb/s
  Output #0, ipod, to 'sample-copy.m4v':
    Metadata:
      encoder         : Lavf53.13.0
      Stream #0.0: Video: h264 (avc1 / 0x31637661), yuv420p, 720x576 [SAR 16:15 DAR 4:3], q=2-31, 25 tbn, 25 tbc
      Stream #0.1: Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, s16, 128 kb/s
  Stream mapping:
    Stream #0.0 -> #0.0 (copy)
    Stream #0.1 -> #0.1 (mp3 -> libfaac)
  Press [q] to stop, [?] for help
  [ipod @ 0x7ff8f32d1c00] pts has no value
      Last message repeated 64 times
  frame=   65 fps=  0 q=-1.0 Lsize=     479kB time=00:00:02.60 bitrate=1509.5kbits/s    
  video:436kB audio:40kB global headers:0kB muxing overhead 0.564932%
  

[Alex__]

•  http://dl.dropbox.com/u/1358485/bugs/ffmpeg/502/sample-issue-502.avi
•  http://dl.dropbox.com/u/1358485/bugs/ffmpeg/502/sample-copy-issue-502.m4v
•  http://dl.dropbox.com/u/1358485/bugs/ffmpeg/502/sample-recode-issue-502.m4v

[cehoyos]

Not reproducible with MPlayer and ffplay.

[Alex__]

Too bad, so it's Apple specific again. Also I've to admin that VLC plays the file w/o issues. Here is a screencast example:

•  http://dl.dropbox.com/u/1358485/bugs/ffmpeg/502/screencast-issue-502.m4v

[Alex__]

Why is it invalid? I described in detail that this issue is valid, does really exist, and also added example files.

[reimar]

I am not entirely sure, and the "[ipod @ 0x7ff8f32d1c00] pts has no value" is suspicious, however if only QuickTime? has issues it might just be a QuickTime? bug.
It certainly does mean that most developers will not be able to reproduce it. So you'll have to explain what "jumping" exactly means.
If it means that it plays at irregular speed, that might be something wrong in FFmpeg, if however it means the frames play out of order or decode completely wrong that is most likely a bug in the QuickTime? H.264 decoder and you'll just have to re-encode. In that case you should be able to also see the issue when playing the AVI file (I think there are plugins that should allow playing it directly in QuickTime?).
I'd like to ask developers to avoid changing bug state without writing a few words though.

[Alex__]

Thank you for the feedback.

It certainly does mean that most developers will not be able to reproduce it.

I can start an old VM to test it using Windows with QuickTime?.

So you'll have to explain what "jumping" exactly means.

Therefore I added the above linked screencast.

if however it means the frames play out of order

I guess this is the case.

that is most likely a bug in the QuickTime? H.264 decoder

Hard to say which fault it is: wrong encoding with a resilient decoder (VLC) or correct encoding with a bug in the decoder (QuickTime?).

In that case you should be able to also see the issue when playing the AVI file (I think there are plugins that should allow playing it directly in QuickTime?).

The AVI file plays fine.

I'd like to ask developers to avoid changing bug state without writing a few words though.

If you need any other feedback, please let me know.

[reimar]

Looks like a reorder issue. I guess it's possible that FFmpeg mixes up pts and dts and then QuickTime? uses those time stamps as they are even when they contradict the decoder reordering, though it seems like a strange design.

[RyanS]

I am seeing this when using ffmpeg to mux to mp4 with: ffmpeg -i x.264 -vcodec copy x.mp4
I am using the windows (zeranoe) build of ffmpeg:

ffmpeg version N-45279-g1a104bf Copyright (c) 2000-2012 the FFmpeg developers

built on Oct 10 2012 19:21:13 with gcc 4.7.2 (GCC)

After this ffmpeg mux, Quicktime plays the frames back out of order. It is presenting frames in decode order rather than presentation order. It does this for two different h.264 encoders I am working with, so does not appear to be an h.264 encoder issue. This only happens around B frames, since those cause the difference between decode and presentation order.

It does seem like pts's are wrong. Perhaps pts is simply ascending despite frames being in decode order.

[cehoyos]

This is at least missing complete, uncut console output (together with your command line) and probably a sample, but also some indication that what you claim (wrong pts) is true, ffprobe allows to print pts for each packet.

[RyanS]

My command line and ffmpeg version output was in my prior post. My complete console output does not appear to be substantively different from the original bug report, but I can include it if you think it is worth the extra post size.

Note that I do not see the symptom in VLC (Windows version 2.0.1) just Quicktime.

Using ffprobe, it looks like my speculation was backwards. The ffmpeg-muxed mp4 has pts's that are not in order, they match the coded_picture_number order. However, an mp4box-muxed mp4 (that quicktime plays correctly) has pts's that are simply ascending. Here is ffprobe -show_frames output for the mp4box mp4 that plays correctly (notice coded picture numbers 7,9,8):

[FRAME]
media_type=video
key_frame=0
pkt_pts=32032
pkt_pts_time=0.444889
pkt_dts=36036
pkt_dts_time=0.500500
pkt_duration=4004
pkt_duration_time=0.055611
pkt_pos=107727
width=720
height=480
pix_fmt=yuv420p
sample_aspect_ratio=N/A
pict_type=P
coded_picture_number=7
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
reference=3
[FRAME]
[FRAME]
media_type=video
key_frame=0
pkt_pts=36036
pkt_pts_time=0.500500
pkt_dts=40040
pkt_dts_time=0.556111
pkt_duration=4004
pkt_duration_time=0.055611
pkt_pos=128869
width=720
height=480
pix_fmt=yuv420p
sample_aspect_ratio=N/A
pict_type=B
coded_picture_number=9
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
reference=0
[FRAME]
[FRAME]
media_type=video
key_frame=0
pkt_pts=40040
pkt_pts_time=0.556111
pkt_dts=44044
pkt_dts_time=0.611722
pkt_duration=4004
pkt_duration_time=0.055611
pkt_pos=117491
width=720
height=480
pix_fmt=yuv420p
sample_aspect_ratio=N/A
pict_type=P
coded_picture_number=8
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
reference=3
[FRAME]

I notice that the pts is always < dts, and that pts is always equal to prior frame's dts (in stream order).

[cehoyos]

Replying to RyanS:

My command line and ffmpeg version output was in my prior post. My complete console output does not appear to be substantively different from the original bug report, but I can include it if you think it is worth the extra post size.

I would say it is always required when reopening a ticket, but see below.

[...]

I notice that the pts is always < dts, and that pts is always equal to prior frame's dts (in stream order).

Do you agree that the ticket can be closed as invalid or do I miss something?
(Consider testing vanilla WMP, if you are unable, please point me to a sample.)

[RyanS]

My last comment was not clear. I think ffmpeg mux is getting the pts's wrong. Here is ffprobe -show_frames on the ffmpeg-muxed mp4 from the same x.264 file:

[FRAME]
key_frame=0
pkt_pts=448399
pkt_dts=581865
pkt_duration=66733
pkt_pos=106296
pict_type=P
coded_picture_number=7
reference=3
[FRAME]
[FRAME]
key_frame=0
pkt_pts=581865
pkt_dts=648599
pkt_duration=66734
pkt_pos=127438
pict_type=B
coded_picture_number=9
reference=0
[FRAME]
[FRAME]
key_frame=0
pkt_pts=515132
pkt_dts=715332
pkt_duration=66733
pkt_pos=116060
pict_type=P
coded_picture_number=8
reference=3
[FRAME]
(I cut a bunch of lines that are irrelevant for space.)

You can see here from pkt_pos that the frames are in the same order here as they were above in the mp4box mux. (Note that the pkt_pos values are offset by 1431 bytes probably because the different muxers put in a different header or something, but still the correspondence is clear.) So the frames here appear in the correct presentation order. So the pts values should simply be ascending, just like in the mp4box mux. But they aren't. So it looks like ffmpeg correctly re-orders the frames to presentation order but doesn't set the pts's correctly.

If that is true, then the playback behavior would make sense if Quicktime pays attention to the pts's and ffplay/vlc doesn't. ffplay/vlc may just get the frames from libavformat/libavcodec and play them in the order received, because the decoder is supposed to (and does) put them in presentation order.

Note that because I'm using ffprobe to inspect the mp4's, and ffprobe uses the same libs as ffmpeg, there's some question about what is being done by the mux step vs by the ffprobe decode step. But given the other evidence I think it's the mux step that has the problem.

Here is the x.264 file I'm using:
 https://dl.dropbox.com/u/62496791/x.264

[martinr92]

I've got the same problem with my video files.
If you need more video-samples, let me know.