[FFmpeg-devel] [PATCH] filter_design: document ownership and permissions.

[Stefano Sabatini]

On date Sunday 2012-08-12 14:00:03 +0200, Nicolas George encoded:
> 
> Signed-off-by: Nicolas George <nicolas.george at normalesup.org>
> ---
>  doc/filter_design.txt |  155 ++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 154 insertions(+), 1 deletion(-)
> 
> 
> If what is stated below is considered correct, then the next task will be to
> ensure that all filters obey the rules (and I am fairly sure a lot of them
> do not).
> 
> 
> diff --git a/doc/filter_design.txt b/doc/filter_design.txt
> index 9e3eb3d..2754930 100644
> --- a/doc/filter_design.txt
> +++ b/doc/filter_design.txt
> @@ -33,7 +33,160 @@ Format negotiation
>  Buffer references ownership and permissions
>  ===========================================
>  
> -  TODO
> +  Principle
> +  ---------
> +
> +    Audio and video data are voluminous; the buffer and buffer reference
> +    mechanism is intended to avoid, as much as possible, expensive copies of
> +    that data while still allowing the filters to produce correct results.
> +
> +    The data is stored in buffers represented by AVFilterBuffer structures.
> +    They must not be accessed directly, but through references stored in
> +    AVFilterBufferRef structures. Several references can point to the
> +    same buffer; the buffer is automatically deallocated once all
> +    corresponding references have been destroyed.
> +
> +    The characteristics of the data (resolution, sample rate, etc.) are
> +    stored in the reference; different references for the same buffer can
> +    show different characteristics. In particular, a video reference can
> +    point to only a part of a video buffer.
> +
> +    A reference is usually obtained as input to the start_frame or
> +    filter_samples method or requested using the ff_get_video_buffer or
> +    ff_get_audio_buffer functions. A new reference on an existing buffer can
> +    be created with the avfilter_ref_buffer. A reference is destroyed using
> +    the avfilter_unref_bufferp function.
> +

> +  Reference ownership
> +  -------------------
> +
> +    At any time, a reference “belongs” to a particular piece of code,
> +    usually a filter. With a few caveats that will be explained below, only
> +    that piece of code is allowed to access it. It is also responsible for
> +    destroying it, although this is sometimes done automatically (see the
> +    section on link reference fields).

Here it is not very clear what "at any time" really references. Also
"is allowed to access it" is not really very clear (indeed assuming a
mono-thread application only one "piece of code" can "access a buffer" at any
given time), so the interpretation can be ambiguous here.

Not that I can't conceive a better explanation at the moment, so don't
consider this remark as blocking.

> +
> +    Here are the (fairly obvious) rules for reference ownership:
> +

> +    * A reference received by the start_frame or filter_samples method
> +      belong to the corresponding filter.
> +      

> +      Special exception: for video references: it may be used internally for

Style nit: s/it/the reference/

(not referring the subject of the previous sentence will be easier on
the reader)

> +      automatic copying and must not be destroyed before end_frame; it can
> +      be given away to ff_start_frame.
> +
> +    * A reference passed to ff_start_frame or ff_filter_samples is given
> +      away and must no longer be used.
> +
> +    * A reference created with avfilter_ref_buffer belongs to the code that
> +      created it.
> +
> +    * A reference obtained with ff_get_video_buffer of ff_get_audio_buffer
> +      belongs to the code that requested it.
> +
> +    * A reference given as return value by the get_video_buffer or
> +      get_audio_buffer method is given away and must no longer be used.
> +
> +  Link reference fields
> +  ---------------------
> +
> +    The AVFilterLink structure has a few AVFilterBufferRef fields. Here are
> +    the rules to handle them:
> +
> +    * cur_buf is set before the start_frame and filter_samples methods to
> +      the same reference given as argument to the methods and belongs to the
> +      destination filter of the link. If it has not been cleared after
> +      end_frame or filter_samples, libavfilter will automatically destroy
> +      the reference; therefore, any filter that needs to keep the reference
> +      for longer must set cur_buf to NULL.
> +
> +    * out_buf belongs to the source filter of the link and can be used to
> +      store a reference to the buffer that has been sent to the destination.
> +      If it is not NULL after end_frame or filter_samples, libavfilter will
> +      automatically destroy the reference.
> +

> +      If a video input pad does not have a start_frame method, the default
> +      method will allocate a buffer on the first output of the filter, store

Nit: will request a buffer on the first output of the filter,...
 
> +      the reference in out_buf and push a second reference to the output.
> +
> +    * src_buf, cur_buf_copy and partial_buf are used by libavfilter
> +      internally and must not be accessed by filters.
> +
> +  Reference permissions
> +  ---------------------
> +
> +    The AVFilterBufferRef structure has a perms field that describes what
> +    the code that owns the reference is allowed to do to the buffer data.
> +    Different references for the same buffer can have different permissions.
> +
> +    For video filters, the permissions only apply to the parts of the buffer
> +    that have already been covered by the draw_slice method.
> +
> +    The value is a binary OR of the following constants:
> +
> +    * AV_PERM_READ: the owner can read the buffer data; this is essentially
> +      always true and is there for self-documentation.
> +
> +    * AV_PERM_WRITE: the owner can modify the buffer data.
> +

> +    * AV_PERM_PRESERVE: the owner can rely on the fact that the buffer data
> +      will not be modified by previous filters.

I'm not sure about this. When a filter receives a filter with
PRESERVE, I interpret this like "this buffer should be preserved",
that is the current filter is not allowed to write on it, other
filters (e.g. the filter which marked it with "PRESERVE"), may be
modifying it.

> +    * AV_PERM_REUSE: the owner can output the buffer several times, without
> +      modifying the data in between.
> +

> +    * AV_PERM_REUSE2: the owner can output the buffer several times an

an -> and

> +      modify the data in between (useless without the WRITE permissions).

My understanding is that REUSE/REUSE2 are mostly useful when
*requesting* the buffer, and possibly used by the allocator, but are
ignored by filtering operations.

> +
> +    * AV_PERM_ALIGN: the owner can access the data using fast operations
> +      that require data alignment.
> +
> +    The READ, WRITE and PRESERVE permissions are about sharing the same
> +    buffer between several filters to avoid expensive copies without them
> +    doing conflicting changes on the data.
> +
> +    The REUSE and REUSE2 permissions are about special memory for direct
> +    rendering. For example a buffer directly allocated in video memory must
> +    not modified once it is displayed on screen, or it will cause tearing;
> +    it will therefore not have the REUSE2 permission.
> +
> +    The ALIGN permission is about extracting part of the buffer, for
> +    copy-less padding or cropping for example.
> +
> +
> +    References received on input pads are guaranteed to have all the
> +    permissions stated in the min_perms field and none of the permissions
> +    stated in the rej_perms.
> +
> +    References obtained by ff_get_video_buffer and ff_get_video_buffer are
> +    guaranteed to have at least all the permissions requested as argument.
> +
> +    References created by avfilter_ref_buffer have the same permissions as
> +    the original reference minus the ones explicitly masked; the mask is
> +    usually ~0 to keep the same permissions.
> +
> +    Filters should remove permissions on reference they give to output
> +    whenever necessary. It can be automatically done by setting the
> +    rej_perms field on the output pad.
> +
> +    Here are a few guidelines corresponding to common situations:
> +
> +    * Filters that modify and forward their frame (like drawtext) need the
> +      WRITE permission.
> +

> +    * Filters that read their input to produce a new frame on output (like
> +      scale) need the WRITE permission on input and WRITE permission on the
                         ^^^^^^^^^^^^^^^^
> +      new buffer.

I suppose it is "READ permission on input". 

About: "and WRITE permission on the new buffer"
maybe a more explicit equivalent:
and must request a buffer with WRITE permission

[...]

Should be good otherwise.
-- 
FFmpeg = Freak and Frenzy Maxi Philosophical Educated Goblin