[FFmpeg-devel] Fixing av_seek

[Don Moir]

----- Original Message ----- 
From: "Don Moir" <donmoir at comcast.net>
To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
Sent: Saturday, January 21, 2012 1:18 AM
Subject: Re: [FFmpeg-devel] Fixing av_seek


>> Finnally getting around to looking more closely at seek problems in 
>> ffmpeg.
>>
>> This is a follow up to ticket #504. 
>> https://ffmpeg.org/trac/ffmpeg/ticket/504
>>
>> My goal is to write a higher level seek function that depends on the 
>> lower level ffmpeg seek functions. In some cases, patches to ffmpeg may 
>> need to be applied.
>>
>> The higher level function will only deal with a timestamp that might 
>> correspond to a slider or similiar, and will seek very close to that 
>> timestamp without image distortion.
>>
>> There are several categories of success or failure when seeking. This 
>> first group (Group A), deals with things that work or are easy to deal 
>> with.
>>
>> o - everything works and works well. There are plently of index_entries 
>> and the format in question does the right thing. CODEC_ID_FLV1 is an 
>> example that works well but some other formats work just as well.
>>
>> o - there are no index_entries or none of the index_entries are marked as 
>> AVINDEX_KEYFRAME. This will cause avformat_seek_file or av_seek_frame to 
>> fail. If there are index_entries and the first entry represents the first 
>> frame, then I have found that just marking this first index_entry with 
>> AVINDEX_KEYFRAME fixes the ffmpeg seek failure problem. If there are no 
>> index_entries then you can add one using av_add_index_entry using the 
>> correct parameters for the first frame and marking it as 
>> AVINDEX_KEYFRAME. This assumes the first frame must be a keyframe and 
>> easier to deal with this at startup rather then trying to deal with it 
>> when doing a seek.
>>
>> o - lower level seek functions mostly deal with keyframes and thats all 
>> good, but thats not good enough for an end user trying to use a slider to 
>> seek. So get as close as possible using one of the ffmpeg seek functions 
>> and then proceed forward to the timestamp of interest. This works pretty 
>> good and is fast enough. Mostly, good files will have enough 
>> AVINDEX_KEYFRAME index_entries to make this behave quite well. Even when 
>> there are not enough entries, it works good. You probably want to do this 
>> in a separate thread. Make sure you first seek to a timestamp that is 
>> less than or equal to the timestamp of interest so you can proceed 
>> forward. There are probably ways to improve on this.
>>
>> The second group (Group B) falls into the class of failures or bad 
>> behavior that is not easy to deal with. e.g. an inability to make it work 
>> as expected for various reasons.
>>
>> I am still dealing with trying to find out Group B problems. I have had 
>> issues with MPEG1, THEORA, and a few others, and I will know more about 
>> these problems as I get into it further.
>>
>> I use VLC and SMPlayer to cross check and both of these have their share 
>> of seek problems. I use my own player to test with.
>
> Using this h264 file, the index_entries are built on the fly and appear to 
> be incorrect.
>
> http://sms.pangolin.com/temp/bad_seek_index_h264.zip  (2.5mb)
>
> codebase: git-67f5650 - after fixes with h264 seeking but I don't see any 
> difference with older codebase versions than this one.
>
> Just after av_open_input_file, there are 2 index_entries. The first one 
> has timestamp of 0 and 2nd has timestamp of 0x48f. The odd thing is they 
> both have the same pos which 0x122F. So seems strange from the start. As 
> playback progresses, the number of index_entries increases and this is 
> normal.
>
> When you try to seek on this file to a supposed keyframe, you don't get a 
> finished frame until much later (~4 seconds). What should happen is if you 
> have seeked to a keyframe, you should be getting a finished frame 
> immediately. This is the way I see it at any rate. So trying to seek into 
> this file breaks the playback, meaning you will get a long pause before 
> normal playback resumes because you are not getting a finsihed frame when 
> you should be. If I just seek to zero on this file, then all is good and I 
> get the finished frames as expected but of course it's slower to reach the 
> requested timestamp.
>
> I bring this up because I think this represents a key problem with some of 
> the codec types. That is index_entries are added that don't click with 
> actual keyframes. I asked a question about this relationship on the 
> original ticket but there was no response.
>
> I think getting this fixed will go a long way in fixing several seek 
> problems. But I need information from you if you have any on this. If you 
> don't have a answer, maybe "I or we don't know" would be better than no 
> answer. Also not to just focus on h264 here because I believe there are 
> other formats with this very same problem.

This file identifies some problems with the ogg decoder when seeking:

http://sms.pangolin.com/temp/bad_seek_ogg_BuckBunny.zip (46 mb)

I am not making any attempt to cut some of these samples down to make sure 
all is relevant.

This file plays fine in my own player without any problems. SMPlayer chokes 
on it abit, then a reload and it plays ok. ffplay plays it fine with maybe a 
few artifacts.

The problems come in when you try to seek on it. The ogg decoder has just a 
basic seek that calls ff_seek_frame_binary. This can be found in 
libavformat\oggdec.c in the function ogg_read_seek. Seems seek was kind of 
an after thought and patched in to the orignial code.

o - unlike many of the other decoders, the ogg decoder makes no attempt to 
build it's index_entries when you do a seek. This is the first cause of 
failure. Since it does not have these entries, it will fail in 
ff_seek_frame_binary if the file has not been read sufficiently to create 
it's index. Some ogg files work ok in this regard but BuckBunny will fail. I 
currently go ahead and build these entries if needbe before I call any seek 
for an ogg decoder file and this is a start to get this working. BuckBunny 
is THEORA but ogg decoder handles several CODEC id's.

o - The ogg decoder keeps a private structure in AVFormatContext.priv_data 
(struct ogg) and within this it keeps some private stream information 
(struct ogg_stream). In the ogg_stream it keeps track of lastdts and 
lastpts. When you do a seek everything seems to be ok like position 
information etc. But these 2 variables are not reset to AV_NOTPTS_VALUE and 
they contain the values that were in effect before you did the seek. So on 
the first read after a seek, you will get the right packet but the pts and 
dts values will be wrong for the packet. No amount of flushing etc seems to 
fix it and this is just a bug. I believe the ogg_read_seek functions just 
needs to set these 2 vairaibles to the proper values. With the second read 
after the seek the dts and pts values seem ok.

For my own code I fixed the above issues and now seeking for this file works 
fine. I still have some painting artifacts but this is more of a general 
problem and I will deal with that later.

I develop under windows because thats what most of our users use. Some mac 
too. My experinece with linux is somewhat limited but I used to run that 
using vmware. The application I use to test ffmpeg is designed for windows 
and so it's a bit of a pain to debug and trace thru ffmpeg under windows but 
this is just busy work.

I can idenify and pinpoint these various problems but I am just not setup to 
provide proper patches. I have people yelling at me already about spending 
too much time on this and I have to earn a living. So while I will continue 
to do this and idenify the problems, I will have to leave it up to you to do 
the patches. Finding the problems is the most work anyway.

So the 2 things for the ogg decoder is to make sure the index_entries are 
built before you do a seek. Most of the other decoders do this correctly. 
Make sure the lastpts and lastdts variables (struct ogg_stream) are set to 
proper values in the ogg_read_seek function. ogg_read_seek should also make 
sure the index_entries are built up to the timestamp of interest at least 
before calling ff_seek_frame_binary. Note: just reading the packets with no 
decoding is quick and sufficient to build the index_entries. I don't do it 
if sufficient entries are built alreadly and stop when I have built up to 
the timestamp of interest. These entries are normally built on the fly as 
you read packets when processing the file. I think some decoders mght have 
more elegant ways to do it but this is a good fallback.

ffplay complains about first frame not being key frame when you try to seek 
with ffplay but that doesn't seem to be a problem and may be related to the 
above bugs. Seeking fails badly in SMPlayer for this file and generally 
seeking does not work well at all in ffplay for this file. For my own app 
with the above fixes, it now works perfect outside of some artifacts but 
thats a different issue.