[FFmpeg-devel] [PATCH] videotoolbox: add hwcontext support
Aaron Levinson
alevinsn at aracnet.com
Sun May 7 22:01:11 EEST 2017
Are you also planning to change ffmpeg_videotoolbox.c? See below for
more comments.
Aaron Levinson
On 5/2/2017 8:26 PM, wm4 wrote:
> This adds tons of code for no other benefit than making VideoToolbox
> support conform with the new hwaccel API (using hw_device_ctx and
> hw_frames_ctx).
>
> Since VideoToolbox decoding does not actually require the user to
> allocate frames, the new code does mostly nothing.
>
> One benefit is that ffmpeg_videotoolbox.c can be dropped once generic
> hwaccel support for ffmpeg.c is merged from Libav.
>
> Does not consider VDA or VideoToolbox encoding.
>
> Fun fact: the frame transfer functions are copied from vaapi, as the
> mapping makes copying generic boilerplate. Mapping itself is not
> exported by the VT code, because I don't know how to test.
>
> TODO: API bumps
> ---
> doc/APIchanges | 8 ++
> libavcodec/vda_vt_internal.h | 7 ++
> libavcodec/videotoolbox.c | 186 ++++++++++++++++++++++++++--
> libavutil/Makefile | 3 +
> libavutil/hwcontext.c | 3 +
> libavutil/hwcontext.h | 1 +
> libavutil/hwcontext_internal.h | 1 +
> libavutil/hwcontext_videotoolbox.c | 243 +++++++++++++++++++++++++++++++++++++
> libavutil/hwcontext_videotoolbox.h | 54 +++++++++
> 9 files changed, 496 insertions(+), 10 deletions(-)
> create mode 100644 libavutil/hwcontext_videotoolbox.c
> create mode 100644 libavutil/hwcontext_videotoolbox.h
>
> diff --git a/doc/APIchanges b/doc/APIchanges
> index fcd3423d58..71f5563f03 100644
> --- a/doc/APIchanges
> +++ b/doc/APIchanges
> @@ -15,6 +15,14 @@ libavutil: 2015-08-28
Note that the APIchanges part prevents the entire patch from applying,
but that's to be expected.
>
> API changes, most recent first:
>
> +2017-05-03 - xxxxxxxxxx - lavc 57.xx.100 - avcodec.h
> + VideoToolbox hardware accelerated decoding now supports the new hwaccel API,
> + which can create the decoder context and allocate hardware frame automatically.
> + See AVCodecContext.hw_device_ctx and AVCodecContext.hw_frames_ctx.
I'd change the first sentence as follows: "VideoToolbox
hardware-accelerated decoding now supports the new hwaccel API, which
can create the decoder context and allocate hardware frames automatically."
Changes are "hardware accelerated" -> "hardware-accelerated" and
"hardware frame automatically" -> "hardware frames automatically".
> +
> +2017-05-03 - xxxxxxxxxx - lavu 57.xx.100 - hwcontext.h
> + Add AV_HWDEVICE_TYPE_VIDEOTOOLBOX and implementation.
> +
> 2017-04-11 - 8378466507 - lavu 55.61.100 - avstring.h
> Add av_strireplace().
>
> diff --git a/libavcodec/vda_vt_internal.h b/libavcodec/vda_vt_internal.h
> index 9ff63ccc52..e55a813899 100644
> --- a/libavcodec/vda_vt_internal.h
> +++ b/libavcodec/vda_vt_internal.h
> @@ -40,6 +40,13 @@ typedef struct VTContext {
>
> // The core video buffer
> CVImageBufferRef frame;
> +
> + // Current dummy frames context (depends on exact CVImageBufferRef params).
> + struct AVBufferRef *cached_hw_frames_ctx;
> +
> + // Non-NULL if the new hwaccel API is used. This is only a separate struct
> + // to ease compatibility with the old API.
> + struct AVVideotoolboxContext *vt_ctx;
> } VTContext;
>
> int ff_videotoolbox_alloc_frame(AVCodecContext *avctx, AVFrame *frame);
> diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c
> index 67adad53ed..910ac25ea7 100644
> --- a/libavcodec/videotoolbox.c
> +++ b/libavcodec/videotoolbox.c
> @@ -23,11 +23,13 @@
> #include "config.h"
> #if CONFIG_VIDEOTOOLBOX
> # include "videotoolbox.h"
> +# include "libavutil/hwcontext_videotoolbox.h"
> #else
> # include "vda.h"
> #endif
> #include "vda_vt_internal.h"
> #include "libavutil/avutil.h"
> +#include "libavutil/hwcontext.h"
> #include "bytestream.h"
> #include "h264dec.h"
> #include "mpegvideo.h"
> @@ -188,6 +190,79 @@ int ff_videotoolbox_uninit(AVCodecContext *avctx)
> }
>
> #if CONFIG_VIDEOTOOLBOX
> +// Return the AVVideotoolboxContext that matters currently. Where it comes from
> +// depends on the API used.
> +static AVVideotoolboxContext *videotoolbox_get_context(AVCodecContext *avctx)
> +{
> + // Somewhat tricky because the API user can call av_videotoolbox_default_free()
> + // at any time.
Comment will make more sense if "API" is dropped from the sentence.
> + if (avctx->internal && avctx->internal->hwaccel_priv_data) {
> + VTContext *vtctx = avctx->internal->hwaccel_priv_data;
> + if (vtctx->vt_ctx)
> + return vtctx->vt_ctx;
> + }
From your comment, and your answers to my questions on IRC, it is clear
that these various checks are only needed for the case that
av_videotoolbox_default_free() may be called after the codec is closed.
However, this situation isn't relevant for most of the functions in your
patch that call videotoolbox_get_content(). I suggest moving this check
into videotoolbox_default_free() (which would replace the call to
videotoolbox_get_context() there). If that's done, then
videotoolbox_get_context() can be implemented as:
VTContext *vtctx = avctx->internal->hwaccel_priv_data;
if (vtctx->vt_ctx)
return vtctx->vt_ctx;
return avctx->hwaccel_context;
Also, I suggest improving the comment to make it clear why it is
necessary to check for internal in videotoolbox_default_free().
> + return avctx->hwaccel_context;
> +}
> +
> +static int videotoolbox_buffer_create(AVCodecContext *avctx, AVFrame *frame)
> +{
> + VTContext *vtctx = avctx->internal->hwaccel_priv_data;
> + CVPixelBufferRef pixbuf = (CVPixelBufferRef)vtctx->frame;
> + OSType pixel_format = CVPixelBufferGetPixelFormatType(pixbuf);
> + enum AVPixelFormat sw_format = av_map_videotoolbox_format_to_pixfmt(pixel_format);
> + int width = CVPixelBufferGetWidth(pixbuf);
> + int height = CVPixelBufferGetHeight(pixbuf);
> + AVHWFramesContext *cached_frames;
> + int ret;
> +
> + ret = ff_videotoolbox_buffer_create(vtctx, frame);
> + if (ret < 0)
> + return ret;
> +
> + // Old API code path.
> + if (!vtctx->cached_hw_frames_ctx)
> + return 0;
> +
> + // We can still return frames with unknown underlying format, except we need
> + // "some" AVPixelFormat for it. Use AV_PIX_FMT_VIDEOTOOLBOX to signal an
> + // opaque/unknown format, which is very sketchy, but you can't sue me.
Um, I would guess that this sort of comment doesn't really belong in the
ffmpeg code base :-) .
> + if (sw_format == AV_PIX_FMT_NONE)
> + sw_format = AV_PIX_FMT_VIDEOTOOLBOX;
> +
> + cached_frames = (AVHWFramesContext*)vtctx->cached_hw_frames_ctx->data;
> +
> + if (cached_frames->sw_format != sw_format ||
> + cached_frames->width != width ||
> + cached_frames->height != height) {
> + AVBufferRef *hw_frames_ctx = av_hwframe_ctx_alloc(cached_frames->device_ref);
> + AVHWFramesContext *hw_frames;
> + if (!hw_frames_ctx)
> + return AVERROR(ENOMEM);
> +
> + hw_frames = (AVHWFramesContext*)hw_frames_ctx->data;
> + hw_frames->format = cached_frames->format;
> + hw_frames->sw_format = sw_format;
> + hw_frames->width = width;
> + hw_frames->height = height;
> +
> + ret = av_hwframe_ctx_init(hw_frames_ctx);
> + if (ret < 0) {
> + av_buffer_unref(&hw_frames_ctx);
> + return ret;
> + }
> +
> + av_buffer_unref(&vtctx->cached_hw_frames_ctx);
> + vtctx->cached_hw_frames_ctx = hw_frames_ctx;
> + }
> +
> + av_assert0(!frame->hw_frames_ctx);
> + frame->hw_frames_ctx = av_buffer_ref(vtctx->cached_hw_frames_ctx);
> + if (!frame->hw_frames_ctx)
> + return AVERROR(ENOMEM);
> +
> + return 0;
> +}
> +
> static void videotoolbox_write_mp4_descr_length(PutByteContext *pb, int length)
> {
> int i;
> @@ -323,7 +398,7 @@ static OSStatus videotoolbox_session_decode_frame(AVCodecContext *avctx)
> {
> OSStatus status;
> CMSampleBufferRef sample_buf;
> - AVVideotoolboxContext *videotoolbox = avctx->hwaccel_context;
> + AVVideotoolboxContext *videotoolbox = videotoolbox_get_context(avctx);
> VTContext *vtctx = avctx->internal->hwaccel_priv_data;
>
> sample_buf = videotoolbox_sample_buffer_create(videotoolbox->cm_fmt_desc,
> @@ -349,7 +424,7 @@ static OSStatus videotoolbox_session_decode_frame(AVCodecContext *avctx)
> static int videotoolbox_common_end_frame(AVCodecContext *avctx, AVFrame *frame)
> {
> int status;
> - AVVideotoolboxContext *videotoolbox = avctx->hwaccel_context;
> + AVVideotoolboxContext *videotoolbox = videotoolbox_get_context(avctx);
> VTContext *vtctx = avctx->internal->hwaccel_priv_data;
>
> if (!videotoolbox->session || !vtctx->bitstream)
> @@ -365,7 +440,7 @@ static int videotoolbox_common_end_frame(AVCodecContext *avctx, AVFrame *frame)
> if (!vtctx->frame)
> return AVERROR_UNKNOWN;
>
> - return ff_videotoolbox_buffer_create(vtctx, frame);
> + return videotoolbox_buffer_create(avctx, frame);
> }
>
> static int videotoolbox_h264_end_frame(AVCodecContext *avctx)
> @@ -513,7 +588,7 @@ static CMVideoFormatDescriptionRef videotoolbox_format_desc_create(CMVideoCodecT
>
> static int videotoolbox_default_init(AVCodecContext *avctx)
> {
> - AVVideotoolboxContext *videotoolbox = avctx->hwaccel_context;
> + AVVideotoolboxContext *videotoolbox = videotoolbox_get_context(avctx);
> OSStatus status;
> VTDecompressionOutputCallbackRecord decoder_cb;
> CFDictionaryRef decoder_spec;
> @@ -594,7 +669,7 @@ static int videotoolbox_default_init(AVCodecContext *avctx)
>
> static void videotoolbox_default_free(AVCodecContext *avctx)
> {
> - AVVideotoolboxContext *videotoolbox = avctx->hwaccel_context;
> + AVVideotoolboxContext *videotoolbox = videotoolbox_get_context(avctx);
>
> if (videotoolbox) {
> if (videotoolbox->cm_fmt_desc)
> @@ -607,6 +682,92 @@ static void videotoolbox_default_free(AVCodecContext *avctx)
> }
> }
>
> +static int videotoolbox_uninit(AVCodecContext *avctx)
> +{
> + VTContext *vtctx = avctx->internal->hwaccel_priv_data;
> + if (!vtctx)
> + return 0;
> +
> + ff_videotoolbox_uninit(avctx);
> +
> + if (vtctx->vt_ctx)
> + videotoolbox_default_free(avctx);
Unclear why the call to videotoolbox_default_free() is dependent on the
existence of vt_ctx. Why not eliminate this and just call
av_videotoolbox_default_free() at the end of the function? That way, it
will work in the off chance that it gets to this code and
hwaccel_context is valid (in which case vt_ctx will be null).
> +
> + av_buffer_unref(&vtctx->cached_hw_frames_ctx);
> + av_freep(&vtctx->vt_ctx);
vt_ctx is allocated using av_videotoolbox_alloc_context(). While using
av_freep() is correct, since av_videotoolbox_alloc_context() uses
av_mallocz() to allocate the AVVideotoolboxContext object, I think it
would be preferable to have an av_videotoolbox_free_context() function,
which will continue to do the right thing if the implementation of
av_videotoolbox_alloc_context() ever changes (say, to allocate
additional memory in the AVVideotoolboxContext object). This is
technically an issue with the already existing code though, and in
addition, doing this would constitute a change to the public APIs and
documentation, so not really relevant for this patch. There is also
already precedent for this approach--for example,
avcodec_alloc_context3()/avcodec_free_context().
Also, should probably add a call to av_freep(&avctx->hwaccel_context)
here just in case there is a hwaccel_context, since it doesn't call
av_videotoolbox_default_free() in this case (unless you change to call
av_videotoolbox_default_free()).
> +
> + return 0;
> +}
> +
> +static int videotoolbox_common_init(AVCodecContext *avctx)
> +{
> + VTContext *vtctx = avctx->internal->hwaccel_priv_data;
> + AVHWFramesContext *hw_frames;
> + int err;
> +
> + // Old API - do nothing.
> + if (avctx->hwaccel_context)
> + return 0;
> +
> + if (!avctx->hw_frames_ctx && !avctx->hw_device_ctx) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Either hw_frames_ctx or hw_device_ctx must be set.\n");
> + return AVERROR(EINVAL);
> + }
> +
> + vtctx->vt_ctx = av_videotoolbox_alloc_context();
> + if (!vtctx->vt_ctx) {
> + err = AVERROR(ENOMEM);
> + goto fail;
> + }
> +
> + if (avctx->hw_frames_ctx) {
> + hw_frames = (AVHWFramesContext*)avctx->hw_frames_ctx->data;
> + } else {
> + avctx->hw_frames_ctx = av_hwframe_ctx_alloc(avctx->hw_device_ctx);
> + if (!avctx->hw_frames_ctx) {
> + err = AVERROR(ENOMEM);
> + goto fail;
> + }
> +
> + hw_frames = (AVHWFramesContext*)avctx->hw_frames_ctx->data;
> + hw_frames->format = AV_PIX_FMT_VIDEOTOOLBOX;
> + hw_frames->sw_format = AV_PIX_FMT_NV12; // same as av_videotoolbox_alloc_context()
> + hw_frames->width = avctx->width;
> + hw_frames->height = avctx->height;
> +
> + err = av_hwframe_ctx_init(avctx->hw_frames_ctx);
> + if (err < 0) {
> + av_buffer_unref(&avctx->hw_frames_ctx);
> + goto fail;
> + }
> + }
> +
> + vtctx->cached_hw_frames_ctx = av_buffer_ref(avctx->hw_frames_ctx);
> + if (!vtctx->cached_hw_frames_ctx) {
> + err = AVERROR(ENOMEM);
> + goto fail;
> + }
> +
> + vtctx->vt_ctx->cv_pix_fmt_type =
> + av_map_videotoolbox_format_from_pixfmt(hw_frames->sw_format);
> + if (!vtctx->vt_ctx->cv_pix_fmt_type) {
> + av_log(avctx, AV_LOG_ERROR, "Unknown sw_format.\n");
> + err = AVERROR(EINVAL);
> + goto fail;
> + }
> +
> + err = videotoolbox_default_init(avctx);
> + if (err < 0)
> + goto fail;
> +
> + return 0;
> +
> +fail:
> + videotoolbox_uninit(avctx);
> + return err;
> +}
> +
> AVHWAccel ff_h263_videotoolbox_hwaccel = {
> .name = "h263_videotoolbox",
> .type = AVMEDIA_TYPE_VIDEO,
> @@ -616,7 +777,8 @@ AVHWAccel ff_h263_videotoolbox_hwaccel = {
> .start_frame = videotoolbox_mpeg_start_frame,
> .decode_slice = videotoolbox_mpeg_decode_slice,
> .end_frame = videotoolbox_mpeg_end_frame,
> - .uninit = ff_videotoolbox_uninit,
> + .init = videotoolbox_common_init,
> + .uninit = videotoolbox_uninit,
> .priv_data_size = sizeof(VTContext),
> };
>
> @@ -629,7 +791,8 @@ AVHWAccel ff_h264_videotoolbox_hwaccel = {
> .start_frame = ff_videotoolbox_h264_start_frame,
> .decode_slice = ff_videotoolbox_h264_decode_slice,
> .end_frame = videotoolbox_h264_end_frame,
> - .uninit = ff_videotoolbox_uninit,
> + .init = videotoolbox_common_init,
> + .uninit = videotoolbox_uninit,
> .priv_data_size = sizeof(VTContext),
> };
>
> @@ -642,7 +805,8 @@ AVHWAccel ff_mpeg1_videotoolbox_hwaccel = {
> .start_frame = videotoolbox_mpeg_start_frame,
> .decode_slice = videotoolbox_mpeg_decode_slice,
> .end_frame = videotoolbox_mpeg_end_frame,
> - .uninit = ff_videotoolbox_uninit,
> + .init = videotoolbox_common_init,
> + .uninit = videotoolbox_uninit,
> .priv_data_size = sizeof(VTContext),
> };
>
> @@ -655,7 +819,8 @@ AVHWAccel ff_mpeg2_videotoolbox_hwaccel = {
> .start_frame = videotoolbox_mpeg_start_frame,
> .decode_slice = videotoolbox_mpeg_decode_slice,
> .end_frame = videotoolbox_mpeg_end_frame,
> - .uninit = ff_videotoolbox_uninit,
> + .init = videotoolbox_common_init,
> + .uninit = videotoolbox_uninit,
> .priv_data_size = sizeof(VTContext),
> };
>
> @@ -668,7 +833,8 @@ AVHWAccel ff_mpeg4_videotoolbox_hwaccel = {
> .start_frame = videotoolbox_mpeg_start_frame,
> .decode_slice = videotoolbox_mpeg_decode_slice,
> .end_frame = videotoolbox_mpeg_end_frame,
> - .uninit = ff_videotoolbox_uninit,
> + .init = videotoolbox_common_init,
> + .uninit = videotoolbox_uninit,
> .priv_data_size = sizeof(VTContext),
> };
>
> diff --git a/libavutil/Makefile b/libavutil/Makefile
> index d669a924b0..e1fce7732c 100644
> --- a/libavutil/Makefile
> +++ b/libavutil/Makefile
> @@ -37,6 +37,7 @@ HEADERS = adler32.h \
> hwcontext_dxva2.h \
> hwcontext_qsv.h \
> hwcontext_vaapi.h \
> + hwcontext_videotoolbox.h \
> hwcontext_vdpau.h \
> imgutils.h \
> intfloat.h \
> @@ -161,6 +162,7 @@ OBJS-$(CONFIG_QSV) += hwcontext_qsv.o
> OBJS-$(CONFIG_LZO) += lzo.o
> OBJS-$(CONFIG_OPENCL) += opencl.o opencl_internal.o
> OBJS-$(CONFIG_VAAPI) += hwcontext_vaapi.o
> +OBJS-$(CONFIG_VIDEOTOOLBOX) += hwcontext_videotoolbox.o
> OBJS-$(CONFIG_VDPAU) += hwcontext_vdpau.o
>
> OBJS += $(COMPAT_OBJS:%=../compat/%)
> @@ -173,6 +175,7 @@ SKIPHEADERS-$(CONFIG_CUDA) += hwcontext_cuda_internal.h
> SKIPHEADERS-$(CONFIG_DXVA2) += hwcontext_dxva2.h
> SKIPHEADERS-$(CONFIG_QSV) += hwcontext_qsv.h
> SKIPHEADERS-$(CONFIG_VAAPI) += hwcontext_vaapi.h
> +SKIPHEADERS-$(CONFIG_VDPAU) += hwcontext_videotoolbox.h
Hmm, seems like this should use CONFIG_VIDEOTOOLBOX, not CONFIG_VDPAU.
> SKIPHEADERS-$(CONFIG_VDPAU) += hwcontext_vdpau.h
> SKIPHEADERS-$(HAVE_ATOMICS_GCC) += atomic_gcc.h
> SKIPHEADERS-$(HAVE_ATOMICS_SUNCC) += atomic_suncc.h
> diff --git a/libavutil/hwcontext.c b/libavutil/hwcontext.c
> index 4cfe377982..8d50a32b84 100644
> --- a/libavutil/hwcontext.c
> +++ b/libavutil/hwcontext.c
> @@ -44,6 +44,9 @@ static const HWContextType *hw_table[] = {
> #if CONFIG_VDPAU
> &ff_hwcontext_type_vdpau,
> #endif
> +#if CONFIG_VIDEOTOOLBOX
> + &ff_hwcontext_type_videotoolbox,
> +#endif
> NULL,
> };
>
> diff --git a/libavutil/hwcontext.h b/libavutil/hwcontext.h
> index 284b091209..cfc6ad0e28 100644
> --- a/libavutil/hwcontext.h
> +++ b/libavutil/hwcontext.h
> @@ -30,6 +30,7 @@ enum AVHWDeviceType {
> AV_HWDEVICE_TYPE_VAAPI,
> AV_HWDEVICE_TYPE_DXVA2,
> AV_HWDEVICE_TYPE_QSV,
> + AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
> };
>
> typedef struct AVHWDeviceInternal AVHWDeviceInternal;
> diff --git a/libavutil/hwcontext_internal.h b/libavutil/hwcontext_internal.h
> index 30fce2afd9..cf05323e15 100644
> --- a/libavutil/hwcontext_internal.h
> +++ b/libavutil/hwcontext_internal.h
> @@ -144,5 +144,6 @@ extern const HWContextType ff_hwcontext_type_dxva2;
> extern const HWContextType ff_hwcontext_type_qsv;
> extern const HWContextType ff_hwcontext_type_vaapi;
> extern const HWContextType ff_hwcontext_type_vdpau;
> +extern const HWContextType ff_hwcontext_type_videotoolbox;
>
> #endif /* AVUTIL_HWCONTEXT_INTERNAL_H */
> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
> new file mode 100644
> index 0000000000..cc00f1f2f2
> --- /dev/null
> +++ b/libavutil/hwcontext_videotoolbox.c
> @@ -0,0 +1,243 @@
> +/*
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include "config.h"
> +
> +#include <stdint.h>
> +#include <string.h>
> +
> +#include <VideoToolbox/VideoToolbox.h>
> +
> +#include "buffer.h"
> +#include "common.h"
> +#include "hwcontext.h"
> +#include "hwcontext_internal.h"
> +#include "hwcontext_videotoolbox.h"
> +#include "mem.h"
> +#include "pixfmt.h"
> +#include "pixdesc.h"
> +
> +static const struct {
> + uint32_t cv_fmt;
> + enum AVPixelFormat pix_fmt;
> +} cv_pix_fmts[] = {
> + { kCVPixelFormatType_420YpCbCr8Planar, AV_PIX_FMT_YUV420P },
> + { kCVPixelFormatType_422YpCbCr8, AV_PIX_FMT_UYVY422 },
> + { kCVPixelFormatType_32BGRA, AV_PIX_FMT_BGRA },
> +#ifdef kCFCoreFoundationVersionNumber10_7
> + { kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, AV_PIX_FMT_NV12 },
> +#endif
> +};
> +
> +enum AVPixelFormat av_map_videotoolbox_format_to_pixfmt(uint32_t cv_fmt)
> +{
> + int i;
> + for (i = 0; i < FF_ARRAY_ELEMS(cv_pix_fmts); i++) {
> + if (cv_pix_fmts[i].cv_fmt == cv_fmt)
> + return cv_pix_fmts[i].pix_fmt;
> + }
> + return AV_PIX_FMT_NONE;
> +}
> +
> +uint32_t av_map_videotoolbox_format_from_pixfmt(enum AVPixelFormat pix_fmt)
> +{
> + int i;
> + for (i = 0; i < FF_ARRAY_ELEMS(cv_pix_fmts); i++) {
> + if (cv_pix_fmts[i].pix_fmt == pix_fmt)
> + return cv_pix_fmts[i].cv_fmt;
> + }
> + return 0;
> +}
> +
> +static int vt_get_buffer(AVHWFramesContext *ctx, AVFrame *frame)
> +{
> + frame->buf[0] = av_buffer_pool_get(ctx->pool);
> + if (!frame->buf[0])
> + return AVERROR(ENOMEM);
> +
> + frame->data[3] = frame->buf[0]->data;
> + frame->format = AV_PIX_FMT_VIDEOTOOLBOX;
> + frame->width = ctx->width;
> + frame->height = ctx->height;
> +
> + return 0;
> +}
> +
> +static int vt_transfer_get_formats(AVHWFramesContext *ctx,
> + enum AVHWFrameTransferDirection dir,
> + enum AVPixelFormat **formats)
> +{
> + enum AVPixelFormat *fmts = av_malloc_array(2, sizeof(*fmts));
> + if (!fmts)
> + return AVERROR(ENOMEM);
> +
> + fmts[0] = ctx->sw_format;
> + fmts[1] = AV_PIX_FMT_NONE;
> +
> + *formats = fmts;
> + return 0;
> +}
> +
> +static void vt_unmap(AVHWFramesContext *ctx, HWMapDescriptor *hwmap)
> +{
> + CVPixelBufferRef pixbuf = (CVPixelBufferRef)hwmap->source->data[3];
> +
> + CVPixelBufferUnlockBaseAddress(pixbuf, (uintptr_t)hwmap->priv);
> +}
> +
> +static int vt_map_frame(AVHWFramesContext *ctx, AVFrame *dst, const AVFrame *src,
> + int flags)
> +{
> + CVPixelBufferRef pixbuf = (CVPixelBufferRef)src->data[3];
> + OSType pixel_format = CVPixelBufferGetPixelFormatType(pixbuf);
> + CVReturn err;
> + uint32_t map_flags = 0;
> + int ret;
> + int i;
> + enum AVPixelFormat format;
> +
> + format = av_map_videotoolbox_format_to_pixfmt(pixel_format);
> + if (dst->format != format) {
> + av_log(ctx, AV_LOG_ERROR, "Unsupported or mismatching pixel format: %s\n",
> + av_fourcc2str(pixel_format));
> + return AVERROR_UNKNOWN;
> + }
> +
> + if (CVPixelBufferGetWidth(pixbuf) != ctx->width ||
> + CVPixelBufferGetHeight(pixbuf) != ctx->height) {
> + av_log(ctx, AV_LOG_ERROR, "Inconsistent frame dimensions.\n");
> + return AVERROR_UNKNOWN;
> + }
> +
> + if (flags == AV_HWFRAME_MAP_READ)
> + map_flags = kCVPixelBufferLock_ReadOnly;
> +
> + err = CVPixelBufferLockBaseAddress(pixbuf, map_flags);
> + if (err != kCVReturnSuccess) {
> + av_log(ctx, AV_LOG_ERROR, "Error locking the pixel buffer.\n");
> + return AVERROR_UNKNOWN;
> + }
> +
> + if (CVPixelBufferIsPlanar(pixbuf)) {
> + int planes = CVPixelBufferGetPlaneCount(pixbuf);
> + for (i = 0; i < planes; i++) {
> + dst->data[i] = CVPixelBufferGetBaseAddressOfPlane(pixbuf, i);
> + dst->linesize[i] = CVPixelBufferGetBytesPerRowOfPlane(pixbuf, i);
> + }
> + } else {
> + dst->data[0] = CVPixelBufferGetBaseAddress(pixbuf);
> + dst->linesize[0] = CVPixelBufferGetBytesPerRow(pixbuf);
> + }
> +
> + ret = ff_hwframe_map_create(src->hw_frames_ctx, dst, src, vt_unmap,
> + (void *)(uintptr_t)map_flags);
> + if (ret < 0)
> + goto unlock;
> +
> + return 0;
> +
> +unlock:
> + CVPixelBufferUnlockBaseAddress(pixbuf, map_flags);
> + return ret;
> +}
> +
> +static int vt_transfer_data_from(AVHWFramesContext *hwfc,
> + AVFrame *dst, const AVFrame *src)
> +{
> + AVFrame *map;
> + int err;
> +
> + if (dst->width > hwfc->width || dst->height > hwfc->height)
> + return AVERROR(EINVAL);
> +
> + map = av_frame_alloc();
> + if (!map)
> + return AVERROR(ENOMEM);
> + map->format = dst->format;
> +
> + err = vt_map_frame(hwfc, map, src, AV_HWFRAME_MAP_READ);
> + if (err)
> + goto fail;
> +
> + map->width = dst->width;
> + map->height = dst->height;
> +
> + err = av_frame_copy(dst, map);
> + if (err)
> + goto fail;
> +
> + err = 0;
> +fail:
> + av_frame_free(&map);
> + return err;
> +}
> +
> +static int vt_transfer_data_to(AVHWFramesContext *hwfc,
> + AVFrame *dst, const AVFrame *src)
> +{
> + AVFrame *map;
> + int err;
> +
> + if (src->width > hwfc->width || src->height > hwfc->height)
> + return AVERROR(EINVAL);
> +
> + map = av_frame_alloc();
> + if (!map)
> + return AVERROR(ENOMEM);
> + map->format = src->format;
> +
> + err = vt_map_frame(hwfc, map, dst, AV_HWFRAME_MAP_WRITE | AV_HWFRAME_MAP_OVERWRITE);
> + if (err)
> + goto fail;
> +
> + map->width = src->width;
> + map->height = src->height;
> +
> + err = av_frame_copy(map, src);
> + if (err)
> + goto fail;
> +
> + err = 0;
For consistency with the rest of the file and past precedent, would be
preferable to do:
av_frame_free(&map);
return 0;
instead of falling through to fail, which gives the appearance that
something might not have been done properly. Also applies to data_from().
> +fail:
> + av_frame_free(&map);
> + return err;
> +}
> +
> +static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
> + AVDictionary *opts, int flags)
> +{
> + if (device && device[0]) {
> + av_log(ctx, AV_LOG_ERROR, "Device selection unsupported.\n");
> + return AVERROR_UNKNOWN;
> + }
> +
> + return 0;
> +}
> +
> +const HWContextType ff_hwcontext_type_videotoolbox = {
> + .type = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
> + .name = "videotoolbox",
> +
> + .device_create = vt_device_create,
> + .frames_get_buffer = vt_get_buffer,
> + .transfer_get_formats = vt_transfer_get_formats,
> + .transfer_data_to = vt_transfer_data_to,
> + .transfer_data_from = vt_transfer_data_from,
> +
> + .pix_fmts = (const enum AVPixelFormat[]){ AV_PIX_FMT_VIDEOTOOLBOX, AV_PIX_FMT_NONE },
> +};
> diff --git a/libavutil/hwcontext_videotoolbox.h b/libavutil/hwcontext_videotoolbox.h
> new file mode 100644
> index 0000000000..dc7b873204
> --- /dev/null
> +++ b/libavutil/hwcontext_videotoolbox.h
> @@ -0,0 +1,54 @@
> +/*
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#ifndef AVUTIL_HWCONTEXT_VT_H
> +#define AVUTIL_HWCONTEXT_VT_H
> +
> +#include <stdint.h>
> +
> +#include <VideoToolbox/VideoToolbox.h>
> +
> +#include "pixfmt.h"
> +
> +/**
> + * @file
> + * An API-specific header for AV_HWDEVICE_TYPE_VIDEOTOOLBOX.
> + *
> + * This API currently does not support frame allocation, as the raw VideoToolbox
> + * API does allocation, and FFmpeg itself never has the need to allocate frames.
> + *
> + * If the API user sets a custom pool, AVHWFramesContext.pool must return
> + * AVBufferRefs whose data pointer is a CVImageBufferRef or CVPixelBufferRef.
> + *
> + * Currently AVHWDeviceContext.hwctx and AVHWFramesContext.hwctx are always
> + * NULL.
> + */
> +
> +/**
> + * Convert a VideoToolbox (actually CoreVideo) format to AVPixelFormat.
> + * Returns AV_PIX_FMT_NONE if no known equivalent was found.
> + */
> +enum AVPixelFormat av_map_videotoolbox_format_to_pixfmt(uint32_t cv_fmt);
> +
> +/**
> + * Convert a AVPixelFormat to a VideoToolbox (actually CoreVideo) format.
"a AVPixelFormat" -> "an AVPixelFormat"
> + * Returns 0 if no known equivalent was found.
> + */
> +uint32_t av_map_videotoolbox_format_from_pixfmt(enum AVPixelFormat pix_fmt);
> +
> +#endif /* AVUTIL_HWCONTEXT_VT_H */
>
More information about the ffmpeg-devel
mailing list