[FFmpeg-devel] [PATCH] lavc/mediacodec: add hwaccel support

Michael Niedermayer michael at niedermayer.cc
Fri Jun 24 18:18:02 CEST 2016


On Fri, Jun 24, 2016 at 11:17:41AM +0200, Matthieu Bouron wrote:
> On Thu, Apr 07, 2016 at 02:51:44PM +0200, Matthieu Bouron wrote:
> > On Wed, Mar 23, 2016 at 6:16 PM, Matthieu Bouron <matthieu.bouron at gmail.com>
> > wrote:
> > 
> > >
> > >
> > > On Tue, Mar 22, 2016 at 10:04 AM, Matthieu Bouron <
> > > matthieu.bouron at gmail.com> wrote:
> > >
> > >>
> > >>
> > >> On Fri, Mar 18, 2016 at 5:50 PM, Matthieu Bouron <
> > >> matthieu.bouron at gmail.com> wrote:
> > >>
> > >>> From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
> > >>>
> > >>> ---
> > >>>
> > >>> Hello,
> > >>>
> > >>> The following patch add hwaccel support to the mediacodec (h264) decoder
> > >>> by allowing
> > >>> the user to render the output frames directly on a surface.
> > >>>
> > >>> In order to do so the user needs to initialize the hwaccel through the
> > >>> use of
> > >>> av_mediacodec_alloc_context and av_mediacodec_default_init functions.
> > >>> The later
> > >>> takes a reference to an android/view/Surface as parameter.
> > >>>
> > >>> If the hwaccel successfully initialize, the decoder output frames pix
> > >>> fmt will be
> > >>> AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to
> > >>> render
> > >>> the frames on the surface:
> > >>>
> > >>>     AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3];
> > >>>     av_mediacodec_release_buffer(buffer, 1);
> > >>>
> > >>> The last argument of av_mediacodec_release_buffer enable rendering of the
> > >>> buffer on the surface (or not if set to 0).
> > >>>
> > >>> Regarding the internal changes in the mediacodec decoder:
> > >>>
> > >>> MediaCodec.flush() discards both input and output buffers meaning that if
> > >>> MediaCodec.flush() is called all output buffers the user has a reference
> > >>> on are
> > >>> now invalid (and cannot be used).
> > >>> This behaviour does not fit well in the avcodec API.
> > >>>
> > >>> When the decoder is configured to output software buffers, there is no
> > >>> issue as
> > >>> the buffers are copied.
> > >>>
> > >>> Now when the decoder is configured to output to a surface, the user
> > >>> might not
> > >>> want to render all the frames as fast as the decoder can go and might
> > >>> want to
> > >>> control *when* the frame are rendered, so we need to make sure that the
> > >>> MediaCodec.flush() call is delayed until all the frames the user retains
> > >>> has
> > >>> been released or rendered.
> > >>>
> > >>> Delaying the call to MediaCodec.flush() means buffering any inputs that
> > >>> come
> > >>> the decoder until the user has released/renderer the frame he retains.
> > >>>
> > >>> This is a limitation of this hwaccel implementation, if the user retains
> > >>> a
> > >>> frame (a), then issue a flush command to the decoder, the packets he
> > >>> feeds to
> > >>> the decoder at that point will be queued in the internal decoder packet
> > >>> queue
> > >>> (until he releases the frame (a)). This scenario leads to a memory usage
> > >>> increase to say the least.
> > >>>
> > >>> Currently there is no limitation on the size of the internal decoder
> > >>> packet
> > >>> queue but this is something that can be added easily. Then, if the queue
> > >>> is
> > >>> full, what would be the behaviour of the decoder ? Can it block ? Or
> > >>> should it
> > >>> returns something like AVERROR(EAGAIN) ?
> > >>>
> > >>> About the other internal decoder changes I introduced:
> > >>>
> > >>> The MediaCodecDecContext is now refcounted (using the lavu/atomic api)
> > >>> since
> > >>> the (hwaccel) frames can be retained by the user, we need to delay the
> > >>> destruction of the codec until the user has released all the frames he
> > >>> has a
> > >>> reference on.
> > >>> The reference counter of the MediaCodecDecContext is incremented each
> > >>> time an
> > >>> (hwaccel) frame is outputted by the decoder and decremented each time a
> > >>> (hwaccel) frame is released.
> > >>>
> > >>> Also, when the decoder is configured to output to a surface the pts that
> > >>> are
> > >>> given to the MediaCodec API are now rescaled based on the codec_timebase
> > >>> as
> > >>> those timestamps values are propagated to the frames rendered on the
> > >>> surface
> > >>> since Android M. Not sure if it's really useful though.
> > >>>
> > >>> On the performance side:
> > >>>
> > >>> On a nexus 5, decoding an h264 stream (main profile) 1080p at 60fps:
> > >>>   - software output + rgba conversion goes at 59~60fps
> > >>>   - surface output + render on a surface goes at 100~110fps
> > >>>
> > >>>
> > >> [...]
> > >>
> > >> Patch updated with the following differences:
> > >>   * the public mediacodec api is now always built (not only when
> > >> mediacodec is available) (and the build when mediacodec is not available
> > >> has been fixed)
> > >>   * the documentation of av_mediacodec_release_buffer has been improved a
> > >> bit
> > >>
> > >
> > > Patch updated with the following differences:
> > >   MediaCodecBuffer->released type is now a volatile int (instead of a int*)
> > >   MediaCodecContext->refcount type is now a volatile int (instead of a
> > > int*)
> > >
> > 
> > Ping.
> 
> Rebased patch attached.
> 
> Matthieu

>  configure                       |    1 
>  libavcodec/Makefile             |    6 
>  libavcodec/allcodecs.c          |    1 
>  libavcodec/mediacodec.c         |  133 ++++++++++++++++++++
>  libavcodec/mediacodec.h         |   88 +++++++++++++
>  libavcodec/mediacodec_surface.c |   66 +++++++++
>  libavcodec/mediacodec_surface.h |   31 ++++
>  libavcodec/mediacodec_wrapper.c |    5 
>  libavcodec/mediacodecdec.c      |  265 +++++++++++++++++++++++++++++++++-------
>  libavcodec/mediacodecdec.h      |   17 ++
>  libavcodec/mediacodecdec_h264.c |   44 +++++-
>  libavcodec/version.h            |    2 

>  libavutil/pixdesc.c             |    4 
>  libavutil/pixfmt.h              |    2 
>  libavutil/version.h             |    2 

libavutil changes look good, i dont know mediacodec so cannot easily
comment about the rest

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Those who are best at talking, realize last or never when they are wrong.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20160624/c4332a39/attachment.sig>


More information about the ffmpeg-devel mailing list