avformat_seek_file seeking to wrong frame for AVCHD sample

[rmk]

I have been observing that for files from a certain camera avformat_seek_file did not seek to the closest keyframe as requested (it seeked past the requested interval, although suitable frames do exist in the specified interval).

To reproduce this using ffmpeg, please try the file from ticket 2336 and attempt to seek to position 0.96 using -ss 0.96 as an input option. Looking at the code in ffmpeg_opt.c ffmpeg issues a key_frame seek for the interval (INT64_MIN, 0.96 in timebase units, 0.96 in timebase units), so the first frame decoded by ffmpeg should be the keyframe closest to 0.96 _before_ 0.96 but the result is different.

From debug output added to h264_parser.c I get the following frame numbers (calculated by pts) for key frames (the fact that the decoder doesn't see those as keyframes but the parser does, is a separate issue I described in http://ffmpeg.org/pipermail/libav-user/2013-March/003915.html but I am not sure it should be filed as a ticket, probably yes).

PTS Sec. Frame number
67515 0 0
89115 0.24 6
99915 0.36 9
110715 0.48 12
121515 0.6 15
132315 0.72 18
143115 0.84 21
153915 0.96 24
164715 1.08 27
175515 1.2 30
186315 1.32 33

So 0.96 should work and be exact but from looking at the file, it appears it is really returning the next keyframe (27 at 1.08 Seconds).

Command lines to reproduce:

ffmpeg -ss 0.96 -i ~/tmp/gh2-50i-psf.mts -an -sn -vframes 1 ~/tmp/out096.jpg

Produces frame #27 at 1.08 seconds

ffmpeg -ss 0.92 -i ~/tmp/gh2-50i-psf.mts -an -sn -vframes 1 ~/tmp/out092.jpg

Produces frame #24 at 0.96 seconds.

Looking at the code an API doc I do not see any other explanation than avformat_seek_file not seeking to the right keyframe (it appears it is one too late in this case and others I have observed).

I attached the patch with the log output for the h264 parser. I am not sure I am interpreting this correctly but from what I know about the camera's GOP size, it does make sense.

Here's the output of the runs.

ffmpeg -ss 0.96 -i ~/tmp/gh2-50i-psf.mts -an -sn -vframes 1 -y ~/tmp/out096.jpg
ffmpeg version N-50562-g199db97 Copyright (c) 2000-2013 the FFmpeg developers

built on Mar 8 2013 10:08:49 with llvm-gcc 4.2.1 (LLVM build 2336.11.00)
configuration:
libavutil 52. 18.100 / 52. 18.100
libavcodec 54. 92.100 / 54. 92.100
libavformat 54. 63.104 / 54. 63.104
libavdevice 54. 3.103 / 54. 3.103
libavfilter 3. 42.103 / 3. 42.103
libswscale 2. 2.100 / 2. 2.100
libswresample 0. 17.102 / 0. 17.102

Input #0, mpegts, from '/Users/krueger/tmp/gh2-50i-psf.mts':

Duration: 00:00:01.38, start: 0.710178, bitrate: 46737 kb/s
Program 1

Stream #0:0[0x1011]: Video: h264 (High) (HDMV / 0x564D4448), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 50 tbr, 90k tbn, 50 tbc
Stream #0:1[0x1100]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s
Stream #0:2[0x1200]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090)

Output #0, image2, to '/Users/krueger/tmp/out096.jpg':

Metadata:

encoder : Lavf54.63.104
Stream #0:0: Video: mjpeg, yuvj420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 90k tbn, 25 tbc

Stream mapping:

Stream #0:0 -> #0:0 (h264 -> mjpeg)

Press [q] to stop, ? for help
[h264 @ 0x7f9c42bc6000] Missing reference picture, default is 0
[h264 @ 0x7f9c42bc6000] decode_slice_header error
[h264 @ 0x7f9c42bc6c00] Missing reference picture, default is 0
[h264 @ 0x7f9c42bc6c00] decode_slice_header error
frame= 1 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A
video:140kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.015333%

ffmpeg -ss 0.90 -i ~/tmp/gh2-50i-psf.mts -an -sn -vframes 1 -y ~/tmp/out090.jpg
ffmpeg version N-50562-g199db97 Copyright (c) 2000-2013 the FFmpeg developers

built on Mar 8 2013 10:08:49 with llvm-gcc 4.2.1 (LLVM build 2336.11.00)
configuration:
libavutil 52. 18.100 / 52. 18.100
libavcodec 54. 92.100 / 54. 92.100
libavformat 54. 63.104 / 54. 63.104
libavdevice 54. 3.103 / 54. 3.103
libavfilter 3. 42.103 / 3. 42.103
libswscale 2. 2.100 / 2. 2.100
libswresample 0. 17.102 / 0. 17.102

Input #0, mpegts, from '/Users/krueger/tmp/gh2-50i-psf.mts':

Duration: 00:00:01.38, start: 0.710178, bitrate: 46737 kb/s
Program 1

Stream #0:0[0x1011]: Video: h264 (High) (HDMV / 0x564D4448), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 50 tbr, 90k tbn, 50 tbc
Stream #0:1[0x1100]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s
Stream #0:2[0x1200]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090)

Output #0, image2, to '/Users/krueger/tmp/out090.jpg':

Metadata:

encoder : Lavf54.63.104
Stream #0:0: Video: mjpeg, yuvj420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 90k tbn, 25 tbc

Stream mapping:

Stream #0:0 -> #0:0 (h264 -> mjpeg)

Press [q] to stop, ? for help
frame= 1 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A
video:141kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.015187%

[rmk]

One more thing I just noticed is that the stream start time of the video (and thus the start time of the container) seems to be wrong. It is detected as 63916 (0,7101777778 as reported by ffmpeg on the command line). However, the pts of the first decoded frame is 67515. Do you want me to make a new report for this? The wrong start time would certainly explain seeking inaccuracy to a certain extend as it is used in open_input_file to compute the seek target.

[rmk]

The place in the code, where the packet PTS is "corrected" to the strange value 63916, is the following in compute_pkt_fields

if (pc && pc->dts_sync_point >= 0) {

we have synchronization info from the parser
int64_t den = st->codec->time_base.den * (int64_t) st->time_base.num;
if (den > 0) {

int64_t num = st->codec->time_base.num * (int64_t) st->time_base.den;
if (pkt->dts != AV_NOPTS_VALUE) {

got DTS from the stream, update reference timestamp
st->reference_dts = pkt->dts - pc->dts_ref_dts_delta * num / den;
pkt->pts = pkt->dts + pc->pts_dts_delta * num / den;

here the correct pts 67515 is replaced by DTS (63915) + 1, i.e. 63916

} else if (st->reference_dts != AV_NOPTS_VALUE) {

compute DTS based on reference timestamp
pkt->dts = st->reference_dts + pc->dts_ref_dts_delta * num / den;
pkt->pts = pkt->dts + pc->pts_dts_delta * num / den;

}
if (pc->dts_sync_point > 0)

st->reference_dts = pkt->dts; new reference

}

}