-vf mp=pullup leads to massive A/V-desync

[dave rice]

When I call -vf mp=pullup or (pullup,softskip) I get an "Error opening filters!" warning.

ffmpeg -y -i example.mov -vf mp=pullup -r 24000/1001 -c:v v210 -c:a copy example_ivtc.mov
ffmpeg version N-33392-gf3bacce, Copyright (c) 2000-2011 the FFmpeg developers
  built on Oct  8 2011 18:33:19 with gcc 4.2.1 (Apple Inc. build 5666) (dot 3)
  configuration: --enable-gpl
  libavutil    51. 20. 0 / 51. 20. 0
  libavcodec   53. 19. 1 / 53. 19. 1
  libavformat  53. 14. 0 / 53. 14. 0
  libavdevice  53.  4. 0 / 53.  4. 0
  libavfilter   2. 43. 6 /  2. 43. 6
  libswscale    2.  1. 0 /  2.  1. 0
  libpostproc  51.  2. 0 / 51.  2. 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101005800] Unimplemented container channel layout.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101005800] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/MPlayer/incoming/ and contact the ffmpeg-devel mailing list.

Seems stream 0 codec frame rate differs from container frame rate: 30000.00 (30000/1) -> 29.97 (30000/1001)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'example.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    creation_time   : 2011-11-05 17:32:17
  Duration: 00:19:52.02, start: -39.572900, bitrate: 1122 kb/s
    Stream #0:0(eng): Video: v210 (v210 / 0x30313276), yuv422p10le, 720x486, 223725 kb/s, SAR 9:10 DAR 4:3, 29.97 fps, 29.97 tbr, 30k tbn, 30k tbc
    Metadata:
      creation_time   : 2011-11-05 17:32:17
    Stream #0:1(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, 2 channels, s16, 1536 kb/s
    Metadata:
      creation_time   : 2011-11-05 17:32:17
    Stream #0:2(eng): Data: none (tmcd / 0x64636D74)
    Metadata:
      creation_time   : 2011-11-05 17:32:17
[buffer @ 0x100e01900] w:720 h:486 pixfmt:yuv422p10le tb:1/1000000 sar:9/10 sws_param:
[mp @ 0x100e040c0] This is a unholy filter, it will be purified by the ffmpeg exorcist team
which will change its syntax from dark -vf mp to light -vf.
Thou shalst not make spells or scripts that depend on it
[buffersink @ 0x100e03d80] auto-inserting filter 'auto-inserted scale 0' between the filter 'Parsed_mp_0' and the filter 'out'
[mp @ 0x100e040c0] auto-inserting filter 'auto-inserted scale 1' between the filter 'src' and the filter 'Parsed_mp_0'
[scale @ 0x100e04a80] w:720 h:486 fmt:yuv422p10le -> w:720 h:486 fmt:yuv420p flags:0x4
Error opening filters!

[Carl Eugen Hoyos]

Sample:
​http://samples.mplayerhq.hu/MPEG-VOB/inverse_telecine_problem/vts_01_1.1st_50M.vob

MEncoder command line:

mencoder vts_01_1.1st_50M.vob -ovc lavc -oac faac -vf pullup=4:4:20:20:-1:0 -ofps 24000/1001 -ss 75 -endpos 10 -nosub -o out.avi

(Intentionally missing softskip which is required for a cleaner result)

FFmpeg / mp=pullup skips nearly all frames:

$ ffmpeg -ss 65 -i vts_01_1.1st_50M.vob -qscale 2 -vf mp=pullup=4:4:20:20:-1:0 -t 20 -r 24001/1001 out.avi
ffmpeg version N-35159-g65f2485, Copyright (c) 2000-2011 the FFmpeg developers
  built on Nov 25 2011 14:39:13 with gcc 4.5.3
  configuration: --cc=/usr/local/gcc-4.5.3/bin/gcc --enable-gpl --disable-everything --enable-protocol=file --enable-demuxer='mpegps,mpegvideo' --enable-decoder='ac3,mpeg2video,rawvideo' --enable-encoder='mpeg4,ac3' --enable-muxer=avi --enable-filter='mp,buffersink' --enable-parser='mpegaudio,mpegvideo'
  libavutil    51. 29. 0 / 51. 29. 0
  libavcodec   53. 37. 0 / 53. 37. 0
  libavformat  53. 21. 0 / 53. 21. 0
  libavdevice  53.  4. 0 / 53.  4. 0
  libavfilter   2. 49. 0 /  2. 49. 0
  libswscale    2.  1. 0 /  2.  1. 0
  libpostproc  51.  2. 0 / 51.  2. 0
[mpeg @ 0x9007a0] max_analyze_duration 5000000 reached at 5004667

Seems stream 0 codec frame rate differs from container frame rate: 59.94 (60000/1001) -> 59.94 (60000/1001)
Input #0, mpeg, from 'vts_01_1.1st_50M.vob':
  Duration: 00:01:32.65, start: 0.196500, bitrate: 4526 kb/s
    Stream #0:0[0x1e0]: Video: mpeg2video (Main), yuv420p, 720x480 [SAR 32:27 DAR 16:9], 9800 kb/s, 23.98 fps, 59.94 tbr, 90k tbn, 59.94 tbc
    Stream #0:1[0x80]: Audio: ac3, 48000 Hz, stereo, s16, 192 kb/s
    Stream #0:2[0x81]: Audio: ac3, 48000 Hz, 5.1(side), s16, 448 kb/s
    Stream #0:3[0x82]: Audio: ac3, 48000 Hz, stereo, s16, 192 kb/s
    Stream #0:4[0x83]: Audio: ac3, 48000 Hz, stereo, s16, 192 kb/s
[buffer @ 0x907340] w:720 h:480 pixfmt:yuv420p tb:1/1000000 sar:32/27 sws_param:
[mp @ 0xb04a40] 'pullup' is a wrapped MPlayer filter (libmpcodecs). This filter may be removed
once it has been ported to a native libavfilter.
Incompatible sample format 's16' for codec 'ac3', auto-selecting format 'flt'
Output #0, avi, to 'out.avi':
  Metadata:
    ISFT            : Lavf53.21.0
    Stream #0:0: Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 720x480 [SAR 32:27 DAR 16:9], q=2-31, 200 kb/s, 23.98 tbn, 23.98 tbc
    Stream #0:1: Audio: ac3 ([0] [0][0] / 0x2000), 48000 Hz, 5.1(side), flt, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg2video -> mpeg4)
  Stream #0:2 -> #0:1 (ac3 -> ac3)
Press [q] to stop, [?] for help
[ac3 @ 0x9940a0] frame sync error
Error while decoding stream #0:2
[buffer @ 0x907340] request_frame() called with no available frame!
    Last message repeated 1 times
[mpeg @ 0x9007a0] parser not found for codec dvd_subtitle, packets or times may be invalid.

affinity: +0.+1..2+.3.
breaks:   .0..1..2..3.
duration: 1

affinity: +0..1++2..3+
breaks:   .0..1..2..3.
duration: 1

affinity: .0++1..2+.3.
breaks:   .0..1..2..3.
duration: 2

affinity: .0++1..2+.3.
breaks:   .0..1..2..3.
duration: 2

affinity: .0++1..2+.3.
breaks:   .0..1..2..3.
duration: 2


...


affinity: .0++1..2+.3.
breaks:   .0..1..2..3.
duration: 2

affinity: .0++1..2+.3.
breaks:   .0..1..2..3.
duration: 2

affinity: .0++1..2+.3.
breaks:   .0..1..2..3.
duration: 2

affinity: .0++1..2+.3.
breaks:   .0..1..2..3.
duration: 2

frame=    2 fps=  1 q=2.0 Lsize=     389kB time=00:00:00.08 bitrate=38242.5kbits/s dup=0 drop=472
video:44kB audio:320kB global headers:0kB muxing overhead 6.814036%

[Carl Eugen Hoyos]

The filter works fine with the following command line for the given sample:

$ ffmpeg -i vts_01_1.1st_50M.vob -qscale 2 -vf 'mp=pullup=4:4:20:20:-1:0,mp=softskip,setpts=N/(24000/1001*TB)' -r 24001/1001 out.avi

It fails for another sample here.

[Carl Eugen Hoyos]

Replying to cehoyos:

It fails for another sample here.

That was because height was not a multiple of four (error message committed to MPlayer, will be merged).

The filter works in the sense that it can remove interlaced frames from telecined content, unfortunately it breaks AV-sync for longer samples.

Note that the filter only supports yuv420p and I believe this is unlikely to change soon (it simply does not work for v210 samples if you don't change the colour space).

[Carl Eugen Hoyos]

The following works for clean (uncut) telecined material like a sample I produced with -vf mp=telecine:

$ ffmpeg -i telecined -vf mp=detc=am=0:dr=2:fr=0,setpts='N/(24000/1001*TB)' -qscale 2 -r 24000/1001 out.avi

If this is of any use, it should be possible to extend it for yuv422p10.

[Jeff]

Thank you, cehoyos, for your work on this. I found this ticket #681 while searching for a filterchain to do effective ivtc in ffmpeg.

There is an excellent discussion in the Mplayer documentation of the various ivtc options in section ​7.2.3.4. Mixed progressive and telecine. (The filmdint filter mentioned there, which sounds very promising, is unfortunately not supported in ffmpeg.)

When I first found this ticket, ​this solution was your last update. When I tried it, I too experienced massive A/V desync. In fact, there was desync with every other Mplayer filter I tried except the one you quote ​just above, and with the same options—

mp=detc=am=1:dr=2

for example, also causes desync.

Section 7.2.3.4 of the Mplayer docs suggests placing the softpulldown filter before detc or ivtc in order to insure those filters have a uniformly telecine'd stream to process. Additionally, I noticed that the output of detc is interlaced. I was able to get good results with this filterchain:

-vf mp=softpulldown,mp=detc=am=0:dr=2:fr=0,yadif,setpts='N/(24000/1001*TB)' -r 24000/1001

---

Ultimately, the reason for the A/V desync with the other Mplayer filters is the equation for setpts (='N/(24000/1001*TB)') which, according to ​Section 31.2.1 of the ffmpeg docs, sets a fixed frame rate. What is needed is an equation for setpts that produces an output stream with the same running time as the input stream, regardless of how many frames the preceding filters drop. Perhaps someone with some math skills and an understanding of the setpts internal variables could write one for us???? :)

(It seems likely that MEncoder includes code to make this calculation internally. In every discussion I have read concerning how to ivtc with MEncoder, there is no mention of any A/V desync problems, and no inclusion of any sort of a setpts filter in any of the sample command lines—these things are conspicuous by their absence. Instead, MEncoder has separate command line options for the input stream framerate, and the desired output framerate.)

[Carl Eugen Hoyos]

Replying to darkmoon:

When I first found this ticket, ​this solution was your last update. When I tried it, I too experienced massive A/V desync.

In fact, there was desync with every other Mplayer filter I tried

Could you elaborate?
I don't think this is a known problem. (Or do you mean: With every other MPlayer inverse telecine filter except detc which unfortunately does not work for the real-world samples you have?)

except the one you quote ​just above, and with the same options—

mp=detc=am=1:dr=2

for example, also causes desync.

Section 7.2.3.4 of the Mplayer docs suggests placing the softpulldown filter before detc or ivtc in order to insure those filters have a uniformly telecine'd stream to process.

Additionally, I noticed that the output of detc is interlaced.

I don't think this is correct as-such:
Of course I believe you that you tried the filter with a sample and the output looked interlaced, but the point is that the filter can only work on clean (unedited) telecined material on which it will perform a perfect inverse telecine process if invoked with the correct start-pattern.

I was able to get good results with this filterchain:

-vf mp=softpulldown,mp=detc=am=0:dr=2:fr=0,yadif,setpts='N/(24000/1001*TB)' -r 24000/1001

I live in PAL-country and I am therefore certainly no expert for telecined material, but I believe you first have to decide if your input video is interlaced or telecined and then you should either use a de-interlacer (yadif) or try an inverse telecine process (which is currently not working perfectly within FFmpeg), combining them seems like a very bad idea to me.
(Or in other words: inverse telecine is a process that recovers the original material without any quality loss whatsoever, while a de-interlacer gets broken input and tries to hide the brokenness by changing the video severely. Doing this on telecined input ruins every possibility to ever get back the original video.)

Ultimately, the reason for the A/V desync with the other Mplayer filters is the equation for setpts (='N/(24000/1001*TB)') which, according to ​Section 31.2.1 of the ffmpeg docs, sets a fixed frame rate.

I absolutely may miss something, but imo the output of inverse telecine has a fixed frame rate of 24000/1001.

What is needed is an equation for setpts that produces an output stream with the same running time as the input stream, regardless of how many frames the preceding filters drop. Perhaps someone with some math skills and an understanding of the setpts internal variables could write one for us???? :)

(It seems likely that MEncoder includes code to make this calculation internally. In every discussion I have read concerning how to ivtc with MEncoder, there is no mention of any A/V desync problems, and no inclusion of any sort of a setpts filter in any of the sample command lines—these things are conspicuous by their absence. Instead, MEncoder has separate command line options for the input stream framerate, and the desired output framerate.)

Note that in my experience, current MEncoder fails very badly for telecined material so whoever is interested in solving this problem and has some programming skills, should really work on porting pullup (or filmdint) to a native FFmpeg filter (it is imo not unlikely that timestamps are the only thing missing.)

[doconnor]

I think it might work if the NoSkip filter was implemented. That should solve the problem of mencoder skipping frames without telling ffmpeg about it.

[Elon Musk]

There is native pullup filter now.