[FFmpeg-devel] [PATCH] lavc/mediacodec: add hwaccel support
Matthieu Bouron
matthieu.bouron at gmail.com
Thu Apr 14 14:19:42 CEST 2016
On Thu, Apr 7, 2016 at 4:18 PM, wm4 <nfxjfg at googlemail.com> wrote:
> On Fri, 18 Mar 2016 17:50:39 +0100
> Matthieu Bouron <matthieu.bouron at gmail.com> wrote:
>
> > From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
> >
> > ---
> >
> > Hello,
>
> Can't say much about this, so just some minor confused comments.
>
Thanks for your comments and sorry for the late reply.
>
> >
> > The following patch add hwaccel support to the mediacodec (h264) decoder
> by allowing
> > the user to render the output frames directly on a surface.
> >
> > In order to do so the user needs to initialize the hwaccel through the
> use of
> > av_mediacodec_alloc_context and av_mediacodec_default_init functions.
> The later
> > takes a reference to an android/view/Surface as parameter.
> >
> > If the hwaccel successfully initialize, the decoder output frames pix
> fmt will be
> > AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to
> render
> > the frames on the surface:
> >
> > AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3];
> > av_mediacodec_release_buffer(buffer, 1);
> >
> > The last argument of av_mediacodec_release_buffer enable rendering of the
> > buffer on the surface (or not if set to 0).
> >
>
> I don't understand this (at all), but unreferencing the AVFrame should
> unref the underlying surface.
>
In this case, the underlying surface will remain (it is owned by the codec
itself) but the output buffer (that should be renderered to the surface)
will be discarded.
>
> > Regarding the internal changes in the mediacodec decoder:
> >
> > MediaCodec.flush() discards both input and output buffers meaning that if
> > MediaCodec.flush() is called all output buffers the user has a reference
> on are
> > now invalid (and cannot be used).
> > This behaviour does not fit well in the avcodec API.
> >
> > When the decoder is configured to output software buffers, there is no
> issue as
> > the buffers are copied.
> >
> > Now when the decoder is configured to output to a surface, the user
> might not
> > want to render all the frames as fast as the decoder can go and might
> want to
> > control *when* the frame are rendered, so we need to make sure that the
> > MediaCodec.flush() call is delayed until all the frames the user retains
> has
> > been released or rendered.
> >
> > Delaying the call to MediaCodec.flush() means buffering any inputs that
> come
> > the decoder until the user has released/renderer the frame he retains.
> >
> > This is a limitation of this hwaccel implementation, if the user retains
> a
> > frame (a), then issue a flush command to the decoder, the packets he
> feeds to
> > the decoder at that point will be queued in the internal decoder packet
> queue
> > (until he releases the frame (a)). This scenario leads to a memory usage
> > increase to say the least.
> >
> > Currently there is no limitation on the size of the internal decoder
> packet
> > queue but this is something that can be added easily. Then, if the queue
> is
> > full, what would be the behaviour of the decoder ? Can it block ? Or
> should it
> > returns something like AVERROR(EAGAIN) ?
>
> The current API can't do anything like this. It has to output 0 or 1
> frame per input packet. (If it outputs nothing, the frame is either
> discarded or queued internally. The queue can be emptied only when
> draining the decoder at the end of the stream.)
>
> So it looks like all you can do is blocking. (Which could lead to a
> deadlock in the API user, depending of how the user's code works?)
>
Yes if I block at some point, it can lead to a deadlock if the user never
releases all the frames. I'm considering buffering a few input packets
before blocking.
>
> >
> > About the other internal decoder changes I introduced:
> >
> > The MediaCodecDecContext is now refcounted (using the lavu/atomic api)
> since
> > the (hwaccel) frames can be retained by the user, we need to delay the
> > destruction of the codec until the user has released all the frames he
> has a
> > reference on.
> > The reference counter of the MediaCodecDecContext is incremented each
> time an
> > (hwaccel) frame is outputted by the decoder and decremented each time a
> > (hwaccel) frame is released.
> >
> > Also, when the decoder is configured to output to a surface the pts that
> are
> > given to the MediaCodec API are now rescaled based on the codec_timebase
> as
> > those timestamps values are propagated to the frames rendered on the
> surface
> > since Android M. Not sure if it's really useful though.
>
> That's all nice, but where is this stuff documented at all?
>
It is documented here:
http://developer.android.com/reference/android/media/MediaCodec.html#queueInputBuffer%28int,%20int,%20int,%20long,%20int%29
>
> >
> > On the performance side:
> >
> > On a nexus 5, decoding an h264 stream (main profile) 1080p at 60fps:
> > - software output + rgba conversion goes at 59~60fps
> > - surface output + render on a surface goes at 100~110fps
> >
> > Matthieu
> >
> > ---
> > configure | 1 +
> > libavcodec/Makefile | 6 +-
> > libavcodec/allcodecs.c | 1 +
> > libavcodec/mediacodec.c | 125 ++++++++++++++++++
> > libavcodec/mediacodec.h | 85 +++++++++++++
> > libavcodec/mediacodec_surface.c | 66 ++++++++++
> > libavcodec/mediacodec_surface.h | 31 +++++
> > libavcodec/mediacodec_wrapper.c | 5 +-
> > libavcodec/mediacodecdec.c | 272
> +++++++++++++++++++++++++++++++++-------
> > libavcodec/mediacodecdec.h | 17 +++
> > libavcodec/mediacodecdec_h264.c | 23 ++++
> > libavutil/pixdesc.c | 4 +
> > libavutil/pixfmt.h | 2 +
> > 13 files changed, 586 insertions(+), 52 deletions(-)
> > create mode 100644 libavcodec/mediacodec.c
> > create mode 100644 libavcodec/mediacodec.h
> > create mode 100644 libavcodec/mediacodec_surface.c
> > create mode 100644 libavcodec/mediacodec_surface.h
> >
> > diff --git a/configure b/configure
> > index e5de306..4d66673 100755
> > --- a/configure
> > +++ b/configure
> > @@ -2530,6 +2530,7 @@ h264_d3d11va_hwaccel_select="h264_decoder"
> > h264_dxva2_hwaccel_deps="dxva2"
> > h264_dxva2_hwaccel_select="h264_decoder"
> > h264_mediacodec_decoder_deps="mediacodec"
> > +h264_mediacodec_hwaccel_deps="mediacodec"
> > h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser"
> > h264_mmal_decoder_deps="mmal"
> > h264_mmal_decoder_select="mmal"
> > diff --git a/libavcodec/Makefile b/libavcodec/Makefile
> > index 6bb1af1..a3dad7e 100644
> > --- a/libavcodec/Makefile
> > +++ b/libavcodec/Makefile
> > @@ -10,6 +10,7 @@ HEADERS = avcodec.h
> \
> > dirac.h
> \
> > dxva2.h
> \
> > jni.h
> \
> > + mediacodec.h
> \
> > qsv.h
> \
> > vaapi.h
> \
> > vda.h
> \
> > @@ -91,7 +92,7 @@ OBJS-$(CONFIG_LSP) += lsp.o
> > OBJS-$(CONFIG_LZF) += lzf.o
> > OBJS-$(CONFIG_MDCT) += mdct_fixed.o mdct_float.o
> mdct_fixed_32.o
> > OBJS-$(CONFIG_ME_CMP) += me_cmp.o
> > -OBJS-$(CONFIG_MEDIACODEC) += mediacodecdec.o
> mediacodec_wrapper.o mediacodec_sw_buffer.o
> > +OBJS-$(CONFIG_MEDIACODEC) += mediacodecdec.o
> mediacodec_surface.o mediacodec_wrapper.o mediacodec_sw_buffer.o
> > OBJS-$(CONFIG_MPEG_ER) += mpeg_er.o
> > OBJS-$(CONFIG_MPEGAUDIO) += mpegaudio.o mpegaudiodata.o
> \
> > mpegaudiodecheader.o
> > @@ -734,6 +735,7 @@ OBJS-$(CONFIG_H263_VAAPI_HWACCEL) +=
> vaapi_mpeg4.o
> > OBJS-$(CONFIG_H263_VIDEOTOOLBOX_HWACCEL) += videotoolbox.o
> > OBJS-$(CONFIG_H264_D3D11VA_HWACCEL) += dxva2_h264.o
> > OBJS-$(CONFIG_H264_DXVA2_HWACCEL) += dxva2_h264.o
> > +OBJS-$(CONFIG_H264_MEDIACODEC_HWACCEL) += mediacodec.o
> > OBJS-$(CONFIG_H264_VAAPI_HWACCEL) += vaapi_h264.o
> > OBJS-$(CONFIG_H264_VDA_HWACCEL) += vda_h264.o
> > OBJS-$(CONFIG_H264_VDPAU_HWACCEL) += vdpau_h264.o
> > @@ -947,7 +949,7 @@ SKIPHEADERS-$(CONFIG_LIBSCHROEDINGER) +=
> libschroedinger.h
> > SKIPHEADERS-$(CONFIG_LIBUTVIDEO) += libutvideo.h
> > SKIPHEADERS-$(CONFIG_LIBVPX) += libvpx.h
> > SKIPHEADERS-$(CONFIG_LIBWEBP_ENCODER) += libwebpenc_common.h
> > -SKIPHEADERS-$(CONFIG_MEDIACODEC) += mediacodecdec.h
> mediacodec_wrapper.h mediacodec_sw_buffer.h
> > +SKIPHEADERS-$(CONFIG_MEDIACODEC) += mediacodecdec.h
> mediacodec_surface.h mediacodec_wrapper.h mediacodec_sw_buffer.h
> > SKIPHEADERS-$(CONFIG_QSV) += qsv.h qsv_internal.h
> > SKIPHEADERS-$(CONFIG_QSVDEC) += qsvdec.h
> > SKIPHEADERS-$(CONFIG_QSVENC) += qsvenc.h
> > diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
> > index 2a25d66..79ce855 100644
> > --- a/libavcodec/allcodecs.c
> > +++ b/libavcodec/allcodecs.c
> > @@ -78,6 +78,7 @@ void avcodec_register_all(void)
> > REGISTER_HWACCEL(H263_VIDEOTOOLBOX, h263_videotoolbox);
> > REGISTER_HWACCEL(H264_D3D11VA, h264_d3d11va);
> > REGISTER_HWACCEL(H264_DXVA2, h264_dxva2);
> > + REGISTER_HWACCEL(H264_MEDIACODEC, h264_mediacodec);
> > REGISTER_HWACCEL(H264_MMAL, h264_mmal);
> > REGISTER_HWACCEL(H264_QSV, h264_qsv);
> > REGISTER_HWACCEL(H264_VAAPI, h264_vaapi);
> > diff --git a/libavcodec/mediacodec.c b/libavcodec/mediacodec.c
> > new file mode 100644
> > index 0000000..51dd37c
> > --- /dev/null
> > +++ b/libavcodec/mediacodec.c
> > @@ -0,0 +1,125 @@
> > +/*
> > + * Android MediaCodec public API functions
> > + *
> > + * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com>
> > + *
> > + * This file is part of FFmpeg.
> > + *
> > + * FFmpeg is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU Lesser General Public
> > + * License as published by the Free Software Foundation; either
> > + * version 2.1 of the License, or (at your option) any later version.
> > + *
> > + * FFmpeg is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > + * Lesser General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU Lesser General Public
> > + * License along with FFmpeg; if not, write to the Free Software
> > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA
> > + */
> > +
> > +#include "config.h"
> > +
> > +#if CONFIG_H264_MEDIACODEC_HWACCEL
> > +
> > +#include <jni.h>
> > +
> > +#include "libavcodec/avcodec.h"
> > +#include "libavutil/atomic.h"
> > +#include "libavutil/mem.h"
> > +
> > +#include "ffjni.h"
> > +#include "mediacodec.h"
> > +#include "mediacodecdec.h"
> > +
> > +AVMediaCodecContext *av_mediacodec_alloc_context(void)
> > +{
> > + return av_mallocz(sizeof(AVMediaCodecContext));
> > +}
> > +
> > +int av_mediacodec_default_init(AVCodecContext *avctx,
> AVMediaCodecContext *ctx, void *surface)
> > +{
> > + int ret = 0;
> > + JNIEnv *env = NULL;
> > + int attached = 0;
> > +
> > + env = ff_jni_attach_env(&attached, avctx);
> > + if (!env) {
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + ctx->surface = (*env)->NewGlobalRef(env, surface);
> > + if (ctx->surface) {
> > + avctx->hwaccel_context = ctx;
> > + } else {
> > + av_log(avctx, AV_LOG_ERROR, "Could not create new global
> reference\n");
> > + ret = AVERROR_EXTERNAL;
> > + }
> > +
> > + if (attached) {
> > + ff_jni_detach_env(avctx);
> > + }
> > +
> > + return ret;
> > +}
> > +
> > +void av_mediacodec_default_free(AVCodecContext *avctx)
> > +{
> > + JNIEnv *env = NULL;
> > + int attached = 0;
> > +
> > + AVMediaCodecContext *ctx = avctx->hwaccel_context;
> > +
> > + env = ff_jni_attach_env(&attached, avctx);
> > + if (!env) {
> > + return;
> > + }
> > +
> > + if (ctx->surface) {
> > + (*env)->DeleteGlobalRef(env, ctx->surface);
> > + ctx->surface = NULL;
> > + }
> > +
> > + if (attached) {
> > + ff_jni_detach_env(avctx);
> > + }
> > +
> > + av_freep(&avctx->hwaccel_context);
> > +}
> > +
> > +int av_mediacodec_release_buffer(AVMediaCodecBuffer *buffer, int render)
> > +{
> > + MediaCodecDecContext *ctx = buffer->ctx;
> > + int released = avpriv_atomic_int_add_and_fetch(buffer->released, 1);
> > +
> > + if (released == 1) {
> > + return ff_AMediaCodec_releaseOutputBuffer(ctx->codec,
> buffer->index, render);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +#else
> > +
> > +AVMediaCodecContext *av_mediacodec_alloc_context(void)
> > +{
> > + return NULL;
> > +}
> > +
> > +int av_mediacodec_default_init(AVCodecContext *avctx,
> AVMediaCodecContext *ctx, void *surface)
> > +{
> > + return 0;
> > +}
> > +
> > +void av_mediacodec_default_free(AVCodecContext *avctx)
> > +{
> > +}
> > +
> > +int av_mediacodec_release_buffer(AVMediaCodecBuffer *buffer, int render)
> > +{
> > + return 0;
> > +}
> > +
> > +#endif
> > diff --git a/libavcodec/mediacodec.h b/libavcodec/mediacodec.h
> > new file mode 100644
> > index 0000000..f303c63
> > --- /dev/null
> > +++ b/libavcodec/mediacodec.h
> > @@ -0,0 +1,85 @@
> > +/*
> > + * Android MediaCodec public API
> > + *
> > + * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com>
> > + *
> > + * This file is part of FFmpeg.
> > + *
> > + * FFmpeg is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU Lesser General Public
> > + * License as published by the Free Software Foundation; either
> > + * version 2.1 of the License, or (at your option) any later version.
> > + *
> > + * FFmpeg is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > + * Lesser General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU Lesser General Public
> > + * License along with FFmpeg; if not, write to the Free Software
> > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA
> > + */
> > +
> > +#ifndef AVCODEC_MEDIACODEC_H
> > +#define AVCODEC_MEDIACODEC_H
> > +
> > +#include "libavcodec/avcodec.h"
> > +
> > +/**
> > + * This structure holds a reference to a android/view/Surface object
> that will
> > + * be used as output by the decoder.
> > + *
> > + */
> > +typedef struct AVMediaCodecContext {
> > +
> > + /**
> > + * android/view/Surface object reference.
> > + */
> > + void *surface;
> > +
> > +} AVMediaCodecContext;
> > +
> > +/**
> > + * Allocate and initialize a MediaCodec context.
> > + *
> > + * When decoding with MediaCodec is finished, the caller must free the
> > + * MediaCodec context with av_mediacodec_default_free.
> > + *
> > + * @return a pointer to a newly allocated AVMediaCodecContext on
> success, NULL otherwise
> > + */
> > +AVMediaCodecContext *av_mediacodec_alloc_context(void);
> > +
> > +/**
> > + * Convenience function that sets up the MediaCodec context.
> > + *
> > + * @param avctx codec context
> > + * @param ctx MediaCodec context to initialize
> > + * @param surface reference to an android/view/Surface
> > + * @return 0 on success, < 0 otherwise
> > + */
> > +int av_mediacodec_default_init(AVCodecContext *avctx,
> AVMediaCodecContext *ctx, void *surface);
> > +
> > +/**
> > + * This function must be called to free the MediaCodec context
> initialized with
> > + * av_mediacodec_default_init().
> > + *
> > + * @param avctx codec context
> > + */
> > +void av_mediacodec_default_free(AVCodecContext *avctx);
> > +
> > +/**
> > + * Opaque structure representing a MediaCodec buffer to render.
> > + */
> > +typedef struct MediaCodecBuffer AVMediaCodecBuffer;
>
> Why is the typedef differently named than the struct type?
>
A leftover (fixed locally).
>
> > +
> > +/**
> > + * Release a MediaCodec buffer and render it onto the surface the
> decoder is
> > + * associated with.
> > + *
> > + * @param buffer the buffer to render
> > + * @param render 1 to render the buffer onto the surface or 0 to
> discard the buffer
> > + * @return 0 on success, < 0 otherwise
> > + */
> > +int av_mediacodec_release_buffer(AVMediaCodecBuffer *buffer, int
> render);
> > +
> > +#endif /* AVCODEC_MEDIACODEC_H */
> > diff --git a/libavcodec/mediacodec_surface.c
> b/libavcodec/mediacodec_surface.c
> > new file mode 100644
> > index 0000000..903ebe4
> > --- /dev/null
> > +++ b/libavcodec/mediacodec_surface.c
> > @@ -0,0 +1,66 @@
> > +/*
> > + * Android MediaCodec Surface functions
> > + *
> > + * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com>
> > + *
> > + * This file is part of FFmpeg.
> > + *
> > + * FFmpeg is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU Lesser General Public
> > + * License as published by the Free Software Foundation; either
> > + * version 2.1 of the License, or (at your option) any later version.
> > + *
> > + * FFmpeg is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > + * Lesser General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU Lesser General Public
> > + * License along with FFmpeg; if not, write to the Free Software
> > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA
> > + */
> > +
> > +#include <jni.h>
> > +
> > +#include "ffjni.h"
> > +#include "mediacodec_surface.h"
> > +
> > +void *ff_mediacodec_surface_ref(void *surface, void *log_ctx)
> > +{
> > + int attached = 0;
> > + JNIEnv *env = NULL;
> > +
> > + void *reference = NULL;
> > +
> > + env = ff_jni_attach_env(&attached, log_ctx);
> > + if (!env) {
> > + return NULL;
> > + }
> > +
> > + reference = (*env)->NewGlobalRef(env, surface);
> > +
> > + if (attached) {
> > + ff_jni_detach_env(log_ctx);
> > + }
> > +
> > + return reference;
> > +}
> > +
> > +int ff_mediacodec_surface_unref(void *surface, void *log_ctx)
> > +{
> > + int attached = 0;
> > + JNIEnv *env = NULL;
> > +
> > + env = ff_jni_attach_env(&attached, log_ctx);
> > + if (!env) {
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + (*env)->DeleteGlobalRef(env, surface);
> > +
> > + if (attached) {
> > + ff_jni_detach_env(log_ctx);
> > + }
> > +
> > + return 0;
> > +}
> > diff --git a/libavcodec/mediacodec_surface.h
> b/libavcodec/mediacodec_surface.h
> > new file mode 100644
> > index 0000000..0178b8a
> > --- /dev/null
> > +++ b/libavcodec/mediacodec_surface.h
> > @@ -0,0 +1,31 @@
> > +/*
> > + * Android MediaCodec Surface functions
> > + *
> > + * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com>
> > + *
> > + * This file is part of FFmpeg.
> > + *
> > + * FFmpeg is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU Lesser General Public
> > + * License as published by the Free Software Foundation; either
> > + * version 2.1 of the License, or (at your option) any later version.
> > + *
> > + * FFmpeg is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > + * Lesser General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU Lesser General Public
> > + * License along with FFmpeg; if not, write to the Free Software
> > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA
> > + */
> > +
> > +#ifndef AVCODEC_MEDIACODEC_SURFACE_H
> > +#define AVCODEC_MEDIACODEC_SURFACE_H
> > +
> > +#include "libavcodec/avcodec.h"
> > +
> > +void *ff_mediacodec_surface_ref(void *surface, void *log_ctx);
> > +int ff_mediacodec_surface_unref(void *surface, void *log_ctx);
> > +
> > +#endif /* AVCODEC_MEDIACODEC_SURFACE_H */
> > diff --git a/libavcodec/mediacodec_wrapper.c
> b/libavcodec/mediacodec_wrapper.c
> > index 6b3f905..621e40b 100644
> > --- a/libavcodec/mediacodec_wrapper.c
> > +++ b/libavcodec/mediacodec_wrapper.c
> > @@ -1306,12 +1306,9 @@ int ff_AMediaCodec_configure(FFAMediaCodec*
> codec, const FFAMediaFormat* format,
> > int attached = 0;
> > JNIEnv *env = NULL;
> >
> > - /* TODO: implement surface handling */
> > - av_assert0(surface == NULL);
> > -
> > JNI_ATTACH_ENV_OR_RETURN(env, &attached, codec, AVERROR_EXTERNAL);
> >
> > - (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.configure_id, format->object, NULL, NULL, flags);
> > + (*env)->CallVoidMethod(env, codec->object,
> codec->jfields.configure_id, format->object, surface, NULL, flags);
> > if (ff_jni_exception_check(env, 1, codec) < 0) {
> > ret = AVERROR_EXTERNAL;
> > goto fail;
> > diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c
> > index d385651..1e4d9dd 100644
> > --- a/libavcodec/mediacodecdec.c
> > +++ b/libavcodec/mediacodecdec.c
> > @@ -23,6 +23,7 @@
> > #include <string.h>
> > #include <sys/types.h>
> >
> > +#include "libavutil/atomic.h"
> > #include "libavutil/common.h"
> > #include "libavutil/mem.h"
> > #include "libavutil/log.h"
> > @@ -33,6 +34,8 @@
> > #include "avcodec.h"
> > #include "internal.h"
> >
> > +#include "mediacodec.h"
> > +#include "mediacodec_surface.h"
> > #include "mediacodec_sw_buffer.h"
> > #include "mediacodec_wrapper.h"
> > #include "mediacodecdec.h"
> > @@ -118,6 +121,10 @@ static enum AVPixelFormat
> mcdec_map_color_format(AVCodecContext *avctx,
> > int i;
> > enum AVPixelFormat ret = AV_PIX_FMT_NONE;
> >
> > + if (s->surface) {
> > + return AV_PIX_FMT_MEDIACODEC;
> > + }
> > +
> > if (!strcmp(s->codec_name, "OMX.k3.video.decoder.avc") &&
> color_format == COLOR_FormatYCbYCr) {
> > s->color_format = color_format =
> COLOR_TI_FormatYUV420PackedSemiPlanar;
> > }
> > @@ -134,7 +141,113 @@ static enum AVPixelFormat
> mcdec_map_color_format(AVCodecContext *avctx,
> > return ret;
> > }
> >
> > -static int mediacodec_wrap_buffer(AVCodecContext *avctx,
> > +static void ff_mediacodec_dec_ref(MediaCodecDecContext *s)
> > +{
> > + avpriv_atomic_int_add_and_fetch(s->refcount, 1);
> > +}
> > +
> > +static void ff_mediacodec_dec_unref(MediaCodecDecContext *s)
> > +{
> > + if (!avpriv_atomic_int_add_and_fetch(s->refcount, -1)) {
> > + if (s->codec) {
> > + ff_AMediaCodec_delete(s->codec);
> > + s->codec = NULL;
> > + }
> > +
> > + if (s->format) {
> > + ff_AMediaFormat_delete(s->format);
> > + s->format = NULL;
> > + }
> > +
> > + if (s->surface) {
> > + ff_mediacodec_surface_unref(s->surface, NULL);
> > + s->surface = NULL;
> > + }
> > +
> > + av_freep(&s->codec_name);
> > + av_freep(&s->refcount);
> > + }
> > +}
> > +
> > +static void mediacodec_buffer_release(void *opaque, uint8_t *data)
> > +{
> > + AVMediaCodecBuffer *buffer = opaque;
> > + MediaCodecDecContext *ctx = buffer->ctx;
> > + int released = avpriv_atomic_int_get(buffer->released);
> > +
> > + if (!released) {
> > + ff_AMediaCodec_releaseOutputBuffer(ctx->codec, buffer->index,
> 0);
> > + }
> > +
> > + ff_mediacodec_dec_unref(ctx);
> > + av_freep(&buffer->released);
> > + av_freep(&buffer);
> > +}
> > +
> > +static int mediacodec_wrap_hw_buffer(AVCodecContext *avctx,
> > + MediaCodecDecContext *s,
> > + ssize_t index,
> > + FFAMediaCodecBufferInfo *info,
> > + AVFrame *frame)
> > +{
> > + int ret = 0;
> > + AVMediaCodecBuffer *buffer = NULL;
> > +
> > + frame->buf[0] = NULL;
> > + frame->width = avctx->width;
> > + frame->height = avctx->height;
> > + frame->format = avctx->pix_fmt;
> > + frame->pkt_pts = av_rescale_q(info->presentationTimeUs,
> > + av_make_q(1, 1000000),
> > + avctx->pkt_timebase);
> > +
> > + buffer = av_mallocz(sizeof(AVMediaCodecBuffer));
> > + if (!buffer) {
> > + ret = AVERROR(ENOMEM);
> > + goto fail;
> > + }
> > +
> > + buffer->released = av_mallocz(sizeof(*buffer->released));
> > + if (!buffer->released) {
> > + ret = AVERROR(ENOMEM);
> > + goto fail;
> > + }
> > +
> > + frame->buf[0] = av_buffer_create(NULL,
> > + 0,
> > + mediacodec_buffer_release,
> > + buffer,
> > + AV_BUFFER_FLAG_READONLY);
> > +
> > + if (!frame->buf[0]) {
> > + ret = AVERROR(ENOMEM);
> > + goto fail;
> > +
> > + }
> > +
> > + buffer->ctx = s;
> > + ff_mediacodec_dec_ref(s);
> > +
> > + buffer->index = index;
> > + buffer->pts = info->presentationTimeUs;
> > +
> > + frame->data[3] = (uint8_t *)buffer;
> > +
> > + return 0;
> > +fail:
> > + if (buffer) {
> > + av_free(buffer->released);
> > + av_free(buffer);
> > + }
> > +
> > + av_buffer_unref(&frame->buf[0]);
> > +
> > + ff_AMediaCodec_releaseOutputBuffer(s->codec, index, 0);
> > +
> > + return ret;
> > +}
> > +
> > +static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx,
> > MediaCodecDecContext *s,
> > uint8_t *data,
> > size_t size,
> > @@ -307,14 +420,61 @@ static int
> mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecConte
> > return ff_set_dimensions(avctx, width, height);
> > }
> >
> > +
> > +static int mediacodec_dec_flush_codec(AVCodecContext *avctx,
> MediaCodecDecContext *s)
> > +{
> > + FFAMediaCodec *codec = s->codec;
> > + int status;
> > +
> > + s->queued_buffer_nb = 0;
> > + s->dequeued_buffer_nb = 0;
> > +
> > + s->draining = 0;
> > + s->flushing = 0;
> > +
> > + status = ff_AMediaCodec_flush(codec);
> > + if (status < 0) {
> > + av_log(NULL, AV_LOG_ERROR, "Failed to flush MediaCodec %p",
> codec);
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > + s->first_buffer = 0;
> > + s->first_buffer_at = av_gettime();
>
> What is the system time doing here?
>
It is here for debugging/information purpose, to know how many time it took
to the codec to buffer and output its first buffer when it starts or after
a flush (discard).
>
> > +
> > + return 0;
> > +}
> > +
> > int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext
> *s,
> > const char *mime, FFAMediaFormat *format)
> > {
> > int ret = 0;
> > int status;
> >
> > + enum AVPixelFormat pix_fmt;
> > + enum AVPixelFormat pix_fmts[3] = {
> > + AV_PIX_FMT_MEDIACODEC,
> > + AV_PIX_FMT_NONE,
> > + };
> > +
> > s->first_buffer_at = av_gettime();
> >
> > + s->refcount = av_mallocz(sizeof(*s->refcount));
> > + if (!s->refcount) {
> > + av_log(avctx, AV_LOG_ERROR, "Failed to init decoder reference
> counter\n");
> > + goto fail;
> > + }
> > + *s->refcount = 1;
> > +
> > + pix_fmt = ff_get_format(avctx, pix_fmts);
> > + if (pix_fmt == AV_PIX_FMT_MEDIACODEC) {
> > + AVMediaCodecContext *user_ctx = avctx->hwaccel_context;
> > +
> > + if (user_ctx && user_ctx->surface) {
> > + s->surface = ff_mediacodec_surface_ref(user_ctx->surface,
> avctx);
> > + av_log(avctx, AV_LOG_INFO, "Using surface %p\n",
> s->surface);
> > + }
> > + }
> > +
> > s->codec_name = ff_AMediaCodecList_getCodecNameByType(mime,
> avctx->width, avctx->height, avctx);
> > if (!s->codec_name) {
> > ret = AVERROR_EXTERNAL;
> > @@ -329,7 +489,7 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx,
> MediaCodecDecContext *s,
> > goto fail;
> > }
> >
> > - status = ff_AMediaCodec_configure(s->codec, format, NULL, NULL, 0);
> > + status = ff_AMediaCodec_configure(s->codec, format, s->surface,
> NULL, 0);
> > if (status < 0) {
> > char *desc = ff_AMediaFormat_toString(format);
> > av_log(avctx, AV_LOG_ERROR,
> > @@ -377,7 +537,7 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx,
> MediaCodecDecContext *s,
> > {
> > int ret;
> > int offset = 0;
> > - int need_flushing = 0;
> > + int need_draining = 0;
> > uint8_t *data;
> > ssize_t index;
> > size_t size;
> > @@ -389,15 +549,21 @@ int ff_mediacodec_dec_decode(AVCodecContext
> *avctx, MediaCodecDecContext *s,
> > int64_t input_dequeue_timeout_us = INPUT_DEQUEUE_TIMEOUT_US;
> > int64_t output_dequeue_timeout_us = OUTPUT_DEQUEUE_TIMEOUT_US;
> >
> > + if (s->flushing) {
> > + av_log(avctx, AV_LOG_ERROR, "Decoder is flushing and cannot
> accept new buffer "
> > + "until all output buffers have been
> released\n");
> > + return AVERROR_EXTERNAL;
> > + }
> > +
> > if (pkt->size == 0) {
> > - need_flushing = 1;
> > + need_draining = 1;
> > }
> >
> > - if (s->flushing && need_flushing && s->queued_buffer_nb <= 0) {
> > + if (s->draining && need_draining && s->queued_buffer_nb <= 0) {
> > return 0;
> > }
> >
> > - while (offset < pkt->size || (need_flushing && !s->flushing)) {
> > + while (offset < pkt->size || (need_draining && !s->draining)) {
> > int size;
> >
> > index = ff_AMediaCodec_dequeueInputBuffer(codec,
> input_dequeue_timeout_us);
> > @@ -416,26 +582,37 @@ int ff_mediacodec_dec_decode(AVCodecContext
> *avctx, MediaCodecDecContext *s,
> > return AVERROR_EXTERNAL;
> > }
> >
> > - if (need_flushing) {
> > + if (need_draining) {
> > + int64_t pts = pkt->pts;
> > uint32_t flags =
> ff_AMediaCodec_getBufferFlagEndOfStream(codec);
> >
> > + if (s->surface) {
> > + pts = av_rescale_q(pts, avctx->pkt_timebase,
> av_make_q(1, 1000000));
> > + }
> > +
> > av_log(avctx, AV_LOG_DEBUG, "Sending End Of Stream
> signal\n");
> >
> > - status = ff_AMediaCodec_queueInputBuffer(codec, index, 0,
> 0, pkt->pts, flags);
> > + status = ff_AMediaCodec_queueInputBuffer(codec, index, 0,
> 0, pts, flags);
> > if (status < 0) {
> > av_log(avctx, AV_LOG_ERROR, "Failed to queue input
> empty buffer (status = %d)\n", status);
> > return AVERROR_EXTERNAL;
> > }
> >
> > - s->flushing = 1;
> > + s->draining = 1;
> > break;
> > } else {
> > + int64_t pts = pkt->pts;
> > +
> > size = FFMIN(pkt->size - offset, size);
> >
> > memcpy(data, pkt->data + offset, size);
> > offset += size;
> >
> > - status = ff_AMediaCodec_queueInputBuffer(codec, index, 0,
> size, pkt->pts, 0);
> > + if (s->surface) {
> > + pts = av_rescale_q(pts, avctx->pkt_timebase,
> av_make_q(1, 1000000));
> > + }
> > +
> > + status = ff_AMediaCodec_queueInputBuffer(codec, index, 0,
> size, pts, 0);
> > if (status < 0) {
> > av_log(avctx, AV_LOG_ERROR, "Failed to queue input
> buffer (status = %d)\n", status);
> > return AVERROR_EXTERNAL;
> > @@ -447,8 +624,8 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx,
> MediaCodecDecContext *s,
> > }
> > }
> >
> > - if (s->flushing) {
> > - /* If the codec is flushing, block for a fair amount of time to
> > + if (s->draining) {
> > + /* If the codec is draining, block for a fair amount of time to
> > * ensure we got a frame */
> > output_dequeue_timeout_us = OUTPUT_DEQUEUE_BLOCK_TIMEOUT_US;
> > } else if (s->dequeued_buffer_nb == 0) {
> > @@ -471,15 +648,22 @@ int ff_mediacodec_dec_decode(AVCodecContext
> *avctx, MediaCodecDecContext *s,
> > " flags=%" PRIu32 "\n", index, info.offset, info.size,
> > info.presentationTimeUs, info.flags);
> >
> > - data = ff_AMediaCodec_getOutputBuffer(codec, index, &size);
> > - if (!data) {
> > - av_log(avctx, AV_LOG_ERROR, "Failed to get output
> buffer\n");
> > - return AVERROR_EXTERNAL;
> > - }
> > + if (s->surface) {
> > + if ((ret = mediacodec_wrap_hw_buffer(avctx, s, index,
> &info, frame)) < 0) {
> > + av_log(avctx, AV_LOG_ERROR, "Failed to wrap MediaCodec
> buffer\n");
> > + return ret;
> > + }
> > + } else {
> > + data = ff_AMediaCodec_getOutputBuffer(codec, index, &size);
> > + if (!data) {
> > + av_log(avctx, AV_LOG_ERROR, "Failed to get output
> buffer\n");
> > + return AVERROR_EXTERNAL;
> > + }
> >
> > - if ((ret = mediacodec_wrap_buffer(avctx, s, data, size, index,
> &info, frame)) < 0) {
> > - av_log(avctx, AV_LOG_ERROR, "Failed to wrap MediaCodec
> buffer\n");
> > - return ret;
> > + if ((ret = mediacodec_wrap_sw_buffer(avctx, s, data, size,
> index, &info, frame)) < 0) {
> > + av_log(avctx, AV_LOG_ERROR, "Failed to wrap MediaCodec
> buffer\n");
> > + return ret;
> > + }
> > }
> >
> > *got_frame = 1;
> > @@ -516,9 +700,9 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx,
> MediaCodecDecContext *s,
> > } else if (ff_AMediaCodec_infoOutputBuffersChanged(codec, index)) {
> > ff_AMediaCodec_cleanOutputBuffers(codec);
> > } else if (ff_AMediaCodec_infoTryAgainLater(codec, index)) {
> > - if (s->flushing) {
> > + if (s->draining) {
> > av_log(avctx, AV_LOG_ERROR, "Failed to dequeue output
> buffer within %" PRIi64 "ms "
> > - "while flushing remaining
> frames, output will probably lack last %d frames\n",
> > + "while draining remaining
> frames, output will probably lack last %d frames\n",
> > output_dequeue_timeout_us /
> 1000, s->queued_buffer_nb);
> > } else {
> > av_log(avctx, AV_LOG_DEBUG, "No output buffer available,
> try again later\n");
> > @@ -533,39 +717,35 @@ int ff_mediacodec_dec_decode(AVCodecContext
> *avctx, MediaCodecDecContext *s,
> >
> > int ff_mediacodec_dec_flush(AVCodecContext *avctx, MediaCodecDecContext
> *s)
> > {
> > - FFAMediaCodec *codec = s->codec;
> > - int status;
> > -
> > - s->queued_buffer_nb = 0;
> > - s->dequeued_buffer_nb = 0;
> > + if (!s->surface || avpriv_atomic_int_get(s->refcount) == 1) {
> > + int ret;
> >
> > - s->flushing = 0;
> > + if ((ret = mediacodec_dec_flush_codec(avctx, s)) < 0) {
> > + return ret;
> > + }
> >
> > - status = ff_AMediaCodec_flush(codec);
> > - if (status < 0) {
> > - av_log(NULL, AV_LOG_ERROR, "Failed to flush MediaCodec %p",
> codec);
> > - return AVERROR_EXTERNAL;
> > + return 1;
> > }
> >
> > - s->first_buffer = 0;
> > - s->first_buffer_at = av_gettime();
> > -
> > + s->flushing = 1;
> > return 0;
> > }
> >
> > int ff_mediacodec_dec_close(AVCodecContext *avctx, MediaCodecDecContext
> *s)
> > {
> > - if (s->codec) {
> > - ff_AMediaCodec_delete(s->codec);
> > - s->codec = NULL;
> > - }
> > -
> > - if (s->format) {
> > - ff_AMediaFormat_delete(s->format);
> > - s->format = NULL;
> > - }
> > -
> > - av_freep(&s->codec_name);
> > + ff_mediacodec_dec_unref(s);
> >
> > return 0;
> > }
> > +
> > +int ff_mediacodec_dec_is_flushing(AVCodecContext *avctx,
> MediaCodecDecContext *s)
> > +{
> > + return s->flushing;
> > +}
> > +
> > +AVHWAccel ff_h264_mediacodec_hwaccel = {
> > + .name = "mediacodec",
> > + .type = AVMEDIA_TYPE_VIDEO,
> > + .id = AV_CODEC_ID_H264,
> > + .pix_fmt = AV_PIX_FMT_MEDIACODEC,
> > +};
> > diff --git a/libavcodec/mediacodecdec.h b/libavcodec/mediacodecdec.h
> > index 36fdbf5..dae3d67 100644
> > --- a/libavcodec/mediacodecdec.h
> > +++ b/libavcodec/mediacodecdec.h
> > @@ -34,12 +34,17 @@
> >
> > typedef struct MediaCodecDecContext {
> >
> > + int *refcount;
> > +
> > char *codec_name;
> >
> > FFAMediaCodec *codec;
> > FFAMediaFormat *format;
> >
> > + void *surface;
> > +
> > int started;
> > + int draining;
> > int flushing;
> >
> > int width;
> > @@ -79,4 +84,16 @@ int ff_mediacodec_dec_flush(AVCodecContext *avctx,
> > int ff_mediacodec_dec_close(AVCodecContext *avctx,
> > MediaCodecDecContext *s);
> >
> > +int ff_mediacodec_dec_is_flushing(AVCodecContext *avctx,
> > + MediaCodecDecContext *s);
> > +
> > +typedef struct MediaCodecBuffer {
> > +
> > + MediaCodecDecContext *ctx;
> > + ssize_t index;
> > + int64_t pts;
> > + int *released;
> > +
> > +} MediaCodecBuffer;
> > +
> > #endif /* AVCODEC_MEDIACODECDEC_H */
> > diff --git a/libavcodec/mediacodecdec_h264.c
> b/libavcodec/mediacodecdec_h264.c
> > index 2d1d525..4b74fb1 100644
> > --- a/libavcodec/mediacodecdec_h264.c
> > +++ b/libavcodec/mediacodecdec_h264.c
> > @@ -261,6 +261,29 @@ static int mediacodec_decode_frame(AVCodecContext
> *avctx, void *data,
> > av_fifo_generic_write(s->fifo, &input_ref, sizeof(input_ref),
> NULL);
> > }
> >
> > + /*
> > + * MediaCodec.flush() discards both input and output buffers, thus
> we
> > + * need to delay the call to this function until the user has
> released or
> > + * renderered the frames he retains.
> > + *
> > + * After we have buffered an input packet, check if the codec is in
> the
> > + * flushing state. If it is, we need to call
> ff_mediacodec_dec_flush.
> > + *
> > + * ff_mediacodec_dec_flush returns 0 if the flush cannot be
> performed on
> > + * the codec (because the user retains frames). The codec stays in
> the
> > + * flushing state.
> > + *
> > + * ff_mediacodec_dec_flush returns 1 if the flush can actually be
> > + * performed on the codec. The codec leaves the flushing state and
> can
> > + * process again packets.
> > + *
> > + */
> > + if (ff_mediacodec_dec_is_flushing(avctx, &s->ctx)) {
> > + if (!ff_mediacodec_dec_flush(avctx, &s->ctx)) {
> > + return avpkt->size;
> > + }
> > + }
> > +
> > /* process buffered data */
> > while (!*got_frame) {
> > /* prepare the input data -- convert to Annex B if needed */
> > diff --git a/libavutil/pixdesc.c b/libavutil/pixdesc.c
> > index 981fa0e..8864a3d 100644
> > --- a/libavutil/pixdesc.c
> > +++ b/libavutil/pixdesc.c
> > @@ -1974,6 +1974,10 @@ static const AVPixFmtDescriptor
> av_pix_fmt_descriptors[AV_PIX_FMT_NB] = {
> > .name = "qsv",
> > .flags = AV_PIX_FMT_FLAG_HWACCEL,
> > },
> > + [AV_PIX_FMT_MEDIACODEC] = {
> > + .name = "mediacodec",
> > + .flags = AV_PIX_FMT_FLAG_HWACCEL,
> > + },
> > [AV_PIX_FMT_MMAL] = {
> > .name = "mmal",
> > .flags = AV_PIX_FMT_FLAG_HWACCEL,
> > diff --git a/libavutil/pixfmt.h b/libavutil/pixfmt.h
> > index dbd2470..f8533b0 100644
> > --- a/libavutil/pixfmt.h
> > +++ b/libavutil/pixfmt.h
> > @@ -300,6 +300,8 @@ enum AVPixelFormat {
> > AV_PIX_FMT_GBRAP12BE, ///< planar GBR 4:4:4:4 48bpp, big-endian
> > AV_PIX_FMT_GBRAP12LE, ///< planar GBR 4:4:4:4 48bpp, little-endian
> >
> > + AV_PIX_FMT_MEDIACODEC, ///< hardware decoding through MediaCodec
> > +
>
> Don't forget a minor API bump to libavutil (and libavcodec too).
>
Fixed locally.
>
> > AV_PIX_FMT_NB, ///< number of pixel formats, DO NOT USE THIS
> if you want to link with shared libav* because the number of formats might
> differ between versions
> > };
> >
>
I will wait the hwaccel stuff from libav to be merged in order to resubmit
(and push if ok) this patch.
Thanks for the review,
Matthieu
[...]
More information about the ffmpeg-devel
mailing list