[FFmpeg-devel] [RFC] Filtering of data packets

[Michael Niedermayer]

On Thu, Jul 18, 2013 at 03:54:19PM +0200, Nicolas George wrote:
[...]
> Data structure
> 
>   I already checked, AVFrame has almost all the fields present in AVPacket.
>   The only exception is convergence_duration, that nobody uses, and it would
>   be trivial to add.

>   The packet data and size can go in AVFrame.data[0] and
>   AVFrame.linesize[0].

smells like a hack


> 
> Format negotiation and automatic conversion
> 
>   Data packets are too abstract, I believe format negotiation and automatic
>   conversion are not possible/relevant. The pixel_/sample_format field can
>   be used to store a CodecID, to avoid feeding MP3 to an AC3 decoder, but
>   that is all.
> 
> Generic pads
> 
>   We already have setpts and asetpts, adding dsetpts, and later ssetpts for
>   subtitles will start getting old pretty fast. Therefore, we need generic
>   pads, so that setpts can work with any type of packet.
> 
>   Set AVFilterPad.type to AVMEDIA_TYPE_UNKNOWN to allow lavfi to connect
>   this pad to anything without checking the type, and let
>   AVFilter.query_formats check that the formats are consistent. If several
>   generic filters are connected together, use the complex format negotiation
>   to stabilize the type.
> 
> Attachments
> 
>   The demuxer source should be able to expose any requested stream as an
>   output pad, but I believe it is more practical if all attachments are
>   optionally combined into a single output, with one packet per attachment.
>   Typical use: connect the attachments output of the demuxer to the ASS
>   subtitles renderer for embedded fonts; no need to know how many fonts
>   there are.
> 
> Sparse streams
> 
>   This is the second difficulty for subtitles filtering, and I believe it
>   may also apply to data filtering although I do not have a specific
>   scenario in mind for that case. But I have a solution.
> 
>   Consider the following scenario:
> 
>     [0:v] [1:v] overlay [out]
> 
>   A frame arrives on 0:v, so overlay will request one on 1:v. That works
>   well because frames can be read from 0:v and 1:v independently.
> 
>     [0:a:0] [0:a:1] amerge [out]
> 
>   The same will happen, but it is not possible to read frames independently
>   from any stream: frames will come however they appear in the input file.
>   Requesting a frame on 0:a:1 may cause a frame to arrive on 0:a:0 instead,
>   and it will need to be queued.
> 

>   It will still work because audio streams are continuous and properly
>   interleaved: an audio frame covering the 42.1 - 42.3 time interval in
>   stream #0 will be almost immediately followed or preceded by a frame

Only if you are lucky
Real files are not guranteed to e strictly correctly interleaved
also streams can stop and start and have gaps if you are unlucky
enough.


>   covering a similar interval in stream #1. The filter needs some buffering,
>   but not a lot.
> 
>     [0:v] [0:s] hardsub [out]
> 
>   The same issue happens, but in this case the subtitles stream is not
>   continuous: you can have an action scene with only sfx for five minutes,
>   hardsub will request a frame on 0:s, that will trigger reading from the
>   file, it will find and decode a video frame. Video frames will accumulate
>   in the 0:v input until either the buffer queue overflows or OOM happens.
> 

>   The solution I suggest is to have regular dummy frames on sparse streams.
>   A dummy frame is marked by data[0] being NULL, and counts only for its
>   timestamp.

In a sane world subtitle streams would have sane keyframe intervals
and there would be subtitle packets every 1-2sec

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Dictatorship naturally arises out of democracy, and the most aggravated
form of tyranny and slavery out of the most extreme liberty. -- Plato
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20130718/74c1e19d/attachment.asc>