[FFmpeg-devel] Seeking in Apple HTTP Live Streaming
Takis Issaris
takis.issaris at uhasselt.be
Mon Jan 16 01:20:48 CET 2012
2012/1/14 Michael Niedermayer <michaelni at gmx.at>:
> On Tue, Jan 10, 2012 at 03:23:53PM +0100, Takis Issaris wrote:
>> 2012/1/10 Takis Issaris <takis.issaris at uhasselt.be>:
>> > 2012/1/10 Takis Issaris <takis.issaris at uhasselt.be>:
>> >> Hi,
>> >>
>> >> 2011/12/14 Michael Niedermayer <michaelni at gmx.at>:
>> >>> On Wed, Dec 14, 2011 at 02:59:18PM +0100, Takis Issaris wrote:
>> >>>> Hi,
>> >>>>
>> >>>> 2011/12/14 Takis Issaris <t4k1s at yahoo.com>:
>> >>>> > Hi,
>> >>>> >
>> >>>> > ----- Original Message -----
>> >>>> >
>> >>>> >> From: Michael Niedermayer <michaelni at gmx.at>
>> >>>> >> To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
>> >>>> >> Cc:
>> >>>> >> Sent: Monday, November 28, 2011 10:20 PM
>> >>>> >> Subject: Re: [FFmpeg-devel] Seeking in Apple HTTP Live Streaming
>> >>>> >>
>> >>>> >> On Mon, Nov 28, 2011 at 04:45:18AM -0800, Takis Issaris wrote:
>> >>>> >>> Hi,
>> >>>> >>>
>> >>>> >>> [...]
>> >>>> >>>
>> >>>> >>> I had a look at this last Tuesday, and only did some experiments, but,
>> >>>> >>> nevertheless, I'd rather post the code and get your opinion on the
>> >>>> >>> approach, then let it rot for another week :-/
>> >>>> >>>
>> >>>> >>> It is in no way clean nor did I take into account possible variants, MPEG-TS
>> >>>> >>> files with many streams etc. But, if the general approach is okay (that is, storing
>> >>>> >>> seeking state in applehttp_read_seek() and afterwards skipping packets in
>> >>>> >>> applehttp_read_packet(), I'll start cleaning and correcting the code for all cases.
>> >>>> >>>
>> >>>> >>> So, I'm obviously not suggesting committing this :-)
>> >>>> >>>
>> >>>> >>> In the patch ending in v0p2.diff, I tried to allow seeking to keyframes,
>> >>>> >>> but didn't know how to do this cleanly, thus the horrible hack. Should the
>> >>>> >>> MPEG TS demuxer be modified to set AV_PKT_FLAG_KEY in pkt.flags?
>> >>>> >>
>> >>>> >> the AVParser that comes after the demuxer should set that flag
>> >>>> >> already. judging from your hack it seems it does for some reason not
>> >>>> >> do that, i dont know why
>> >>>> >>
>> >>>> >
>> >>>> > That was my mistake, sorry about that. I didn't work because I had built a minimal FFmpeg version and had left out the AVParser.
>> >>>> >
>> >>>> >
>> >>>> > As all audio packets have the keyframe flag set, seeks would stop on the first audiopacket when seeking for a keyframe. Which is not what one would want. So, in the current patch I keep a variable indicating if the current variant contains video, if so, seeks without AVSEEK_FLAG_ANY will seek to the first video keyframe.
>> >>>> >
>> >>>> >
>> >>>> >>> applehttp.c | 34 ++++++++++++++++++++++++++--------
>> >>>> >>> 1 file changed, 26 insertions(+), 8 deletions(-)
>> >>>> >>> 2e7bec013da203499f9cdf6392571cbcef63fc5f
>> >>>> >> phmi-20111128T1324-ffmpeg-hls_seek_timestamp_v0.diff
>> >>>> >>> commit 432ceba2329492669fa19fc87d2114f4b42e67bd
>> >>>> >>> Author: Panagiotis H.M. Issaris <takis.issaris at uhasselt.be>
>> >>>> >>> Date: Tue Nov 22 11:33:17 2011 +0100
>> >>>> >>>
>> >>>> >>> Seek until right after the requested timestamp
>> >>>> >>>
>> >>>> >>> diff --git a/libavformat/applehttp.c b/libavformat/applehttp.c
>> >>>> >>> index 1694096..0e7074b 100644
>> >>>> >>> --- a/libavformat/applehttp.c
>> >>>> >>> +++ b/libavformat/applehttp.c
>> >>>> >>> @@ -100,6 +100,7 @@ typedef struct AppleHTTPContext {
>> >>>> >>> int end_of_segment;
>> >>>> >>> int first_packet;
>> >>>> >>> int64_t first_timestamp;
>> >>>> >>> + int64_t seek_timestamp;
>> >>>> >>> AVIOInterruptCB *interrupt_callback;
>> >>>> >>> } AppleHTTPContext;
>> >>>> >>>
>> >>>> >>> @@ -529,6 +530,7 @@ static int applehttp_read_header(AVFormatContext *s,
>> >>>> >> AVFormatParameters *ap)
>> >>>> >>>
>> >>>> >>> c->first_packet = 1;
>> >>>> >>> c->first_timestamp = AV_NOPTS_VALUE;
>> >>>> >>> + c->seek_timestamp = AV_NOPTS_VALUE;
>> >>>> >>>
>> >>>> >>> return 0;
>> >>>> >>> fail:
>> >>>> >>> @@ -588,14 +590,29 @@ start:
>> >>>> >>> /* Make sure we've got one buffered packet from each open
>> >>>> >> variant
>> >>>> >>> * stream */
>> >>>> >>> if (var->needed && !var->pkt.data) {
>> >>>> >>> - ret = av_read_frame(var->ctx, &var->pkt);
>> >>>> >>> - if (ret < 0) {
>> >>>> >>> - if (!url_feof(&var->pb))
>> >>>> >>> - return ret;
>> >>>> >>> - reset_packet(&var->pkt);
>> >>>> >>> - } else {
>> >>>> >>> - if (c->first_timestamp == AV_NOPTS_VALUE)
>> >>>> >>> - c->first_timestamp = var->pkt.dts;
>> >>>> >>> + int skipped_packets = 0;
>> >>>> >>> + while (1) {
>> >>>> >>> + ret = av_read_frame(var->ctx, &var->pkt);
>> >>>> >>> + if (ret < 0) {
>> >>>> >>> + if (!url_feof(&var->pb))
>> >>>> >>> + return ret;
>> >>>> >>> + reset_packet(&var->pkt);
>> >>>> >>> + } else {
>> >>>> >>> + if (c->first_timestamp == AV_NOPTS_VALUE)
>> >>>> >>> + c->first_timestamp = var->pkt.dts;
>> >>>> >>> + }
>> >>>> >>> +
>> >>>> >>> + if (c->seek_timestamp==AV_NOPTS_VALUE)
>> >>>> >>> + break;
>> >>>> >>> +
>> >>>> >>> + int64_t ts_diff = var->pkt.dts - c->seek_timestamp;
>> >>>> >>
>> >>>> >> i dont think dts is guranteed to be != AV_NOPTS_VALUE here
>> >>>> >
>> >>>> > I'm not sure how to handle this. Would it be better to use pts instead of dts? After looking at the MPEG TS spec it appears pts is more likely to be available then dts.
>> >>>> >
>> >>>> > Currently, I abort seeking when encountering a packet without dts.
>> >>>>
>> >>>>
>> >>>> Attached, the current code.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>>
>> >>>> >
>> >>>> >>
>> >>>> >> otherwise the principle looks good
>> >>>> >>
>> >>>> >>
>> >>>> >> [...]
>> >>>> >>
>> >>>> >> --
>> >>>> >> Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>> >>>> >>
>> >>>> >> The educated differ from the uneducated as much as the living from the
>> >>>> >> dead. -- Aristotle
>> >>>>
>> >>>>
>> >>>> With friendly regards,
>> >>>> Takis
>> >>>
>> >>>> applehttp.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------
>> >>>> 1 file changed, 52 insertions(+), 8 deletions(-)
>> >>>> 158ba23f61df7b2d76677d20220c8fa816cada7e 0001-Enhance-Apple-HTTP-Livestream-seeking.patch
>> >>>> From d539b11d0d180170a3f2197828e15fd7d1fb4379 Mon Sep 17 00:00:00 2001
>> >>>> From: "Panagiotis H.M. Issaris" <takis.issaris at uhasselt.be>
>> >>>> Date: Wed, 14 Dec 2011 14:29:51 +0100
>> >>>> Subject: [PATCH] Enhance Apple HTTP Livestream seeking
>> >>>>
>> >>>> Enhances seeking by demuxing until the requested timestamp is reached within
>> >>>> the segment selected by the seek code using the playlist info.
>> >>>> ---
>> >>>> libavformat/applehttp.c | 60 ++++++++++++++++++++++++++++++++++++++++------
>> >>>> 1 files changed, 52 insertions(+), 8 deletions(-)
>> >>>>
>> >>>> diff --git a/libavformat/applehttp.c b/libavformat/applehttp.c
>> >>>> index be8ea32..13b41eb 100644
>> >>>> --- a/libavformat/applehttp.c
>> >>>> +++ b/libavformat/applehttp.c
>> >>>> @@ -91,6 +91,7 @@ struct variant {
>> >>>>
>> >>>> char key_url[MAX_URL_SIZE];
>> >>>> uint8_t key[16];
>> >>>> + char contains_video;
>> >>>> };
>> >>>>
>> >>>> typedef struct AppleHTTPContext {
>> >>>> @@ -100,6 +101,8 @@ typedef struct AppleHTTPContext {
>> >>>> int end_of_segment;
>> >>>> int first_packet;
>> >>>> int64_t first_timestamp;
>> >>>> + int64_t seek_timestamp;
>> >>>> + int seek_flags;
>> >>>> AVIOInterruptCB *interrupt_callback;
>> >>>> } AppleHTTPContext;
>> >>>>
>> >>>> @@ -523,12 +526,16 @@ static int applehttp_read_header(AVFormatContext *s, AVFormatParameters *ap)
>> >>>> if (v->bandwidth)
>> >>>> av_dict_set(&st->metadata, "variant_bitrate", bitrate_str,
>> >>>> 0);
>> >>>> +
>> >>>> + if (st->codec->codec_type == AVMEDIA_TYPE_VIDEO)
>> >>>> + v->contains_video = 1;
>> >>>> }
>> >>>> stream_offset += v->ctx->nb_streams;
>> >>>> }
>> >>>>
>> >>>> c->first_packet = 1;
>> >>>> c->first_timestamp = AV_NOPTS_VALUE;
>> >>>> + c->seek_timestamp = AV_NOPTS_VALUE;
>> >>>>
>> >>>> return 0;
>> >>>> fail:
>> >>>> @@ -588,14 +595,47 @@ start:
>> >>>> /* Make sure we've got one buffered packet from each open variant
>> >>>> * stream */
>> >>>> if (var->needed && !var->pkt.data) {
>> >>>> - ret = av_read_frame(var->ctx, &var->pkt);
>> >>>> - if (ret < 0) {
>> >>>> - if (!url_feof(&var->pb))
>> >>>> - return ret;
>> >>>> - reset_packet(&var->pkt);
>> >>>> - } else {
>> >>>> - if (c->first_timestamp == AV_NOPTS_VALUE)
>> >>>> - c->first_timestamp = var->pkt.dts;
>> >>>> + while (1) {
>> >>>> + ret = av_read_frame(var->ctx, &var->pkt);
>> >>>> + if (ret < 0) {
>> >>>> + if (!url_feof(&var->pb))
>> >>>> + return ret;
>> >>>> + else {
>> >>>> + if ((var->cur_seq_no - var->start_seq_no) == (var->n_segments))
>> >>>> + return AVERROR_EOF;
>> >>>> + }
>> >>>> + reset_packet(&var->pkt);
>> >>>> + } else {
>> >>>> + if (c->first_timestamp == AV_NOPTS_VALUE)
>> >>>> + c->first_timestamp = var->pkt.dts;
>> >>>> + }
>> >>>> +
>> >>>> + /* Stop if not seeking */
>> >>>> + if (c->seek_timestamp == AV_NOPTS_VALUE)
>> >>>> + break;
>> >>>> +
>> >>>> + /* Stop if we do not have timestamps */
>> >>>> + if (var->pkt.dts == AV_NOPTS_VALUE) {
>> >>>> + c->seek_timestamp = AV_NOPTS_VALUE;
>> >>>> + break;
>> >>>> + }
>> >>>> +
>> >>>> + int64_t ts_diff = var->pkt.dts - c->seek_timestamp;
>> >>>> + if (ts_diff > 0) {
>> >
>> > Hmm. This should have been ">= 0".
>>
>> Updated patch attached, also preventing an ISO C90 warning.
>>
>>
>> >>>> + if (c->seek_flags & AVSEEK_FLAG_ANY) {
>> >>>> + c->seek_timestamp = AV_NOPTS_VALUE;
>> >>>> + break;
>> >>>> + }
>> >>>> +
>> >>>> + if (var->pkt.flags & AV_PKT_FLAG_KEY) {
>> >>>> +
>> >>>> + /* When there's a video stream, we'll want to seek to the video keyframe */
>> >>>
>> >>> is using the stream_index from applehttp_read_seek() not achiving this ?
>> >>
>> >> Yes, you are right, if -1 is used when seeking the video stream is
>> >> used anyway. And if not, well, the user of the library should select
>> >> the stream he wants to seek in.
>> >>
>> >> Updated patch attached.
>> >>
>> >>
>> >> Met vriendelijke groeten,
>> >> Takis
>> >
>> >
>> >
>> > --
>> > Met vriendelijke groeten,
>> > Takis
>>
>>
>>
>> --
>> Met vriendelijke groeten,
>> Takis
>
>> applehttp.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++--------
>> 1 file changed, 50 insertions(+), 8 deletions(-)
>> bb5629ab00c00fdbc47f759d6d10057a2dda1e5c 0001-Enhance-HLS-seeking.patch
>> From 8d101b88a860a8b748e952d6a22b6e59782e340c Mon Sep 17 00:00:00 2001
>> From: "Panagiotis H.M. Issaris" <takis.issaris at uhasselt.be>
>> Date: Tue, 10 Jan 2012 14:48:57 +0100
>> Subject: [PATCH] Enhance HLS seeking.
>
> LGTM
> should i push it?
Yes. Thanks!
Next, I want to have a go at implementing prefetching of segments
which is mentioned in the draft in section 6.3.3:
"The client SHOULD attempt to load media segments in advance of when
they will be required for uninterrupted playback to compensate for
temporary variations in latency and throughput."
>
> Thanks
>
> [...]
>
> --
> Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>
> The misfortune of the wise is better than the prosperity of the fool.
> -- Epicurus
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
With friendly regards,
Takis
More information about the ffmpeg-devel
mailing list