[FFmpeg-devel] [PATCH v2] hwcontext_vaapi: introduce AV_HWFRAME_MAP_{SEPARATE, COMBINED}_PLANES

[Simon Ser]

This allows callers to ask for separate or combined plane formats.

Combined formats are desirable in many scenarios, for instance when
directly importing the DMA-BUFs into APIs such as GL (as a single
EGLImage), KMS or Wayland.

This patch was tested with the following Mesa MR:
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9015

v2:
- Add SEPARATE_PLANES flag
- Rename to AV_HWFRAME_COMBINED_PLANES

Signed-off-by: Simon Ser <contact at emersion.fr>
Cc: Mark Thompson <sw at jkqxz.net>
Cc: Haihao Xiang <haihao.xiang at intel.com>
Cc: Fei Wang <fei.w.wang at intel.com>
---
 libavutil/hwcontext.h       | 12 ++++++++++++
 libavutil/hwcontext_vaapi.c |  6 +++++-
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/libavutil/hwcontext.h b/libavutil/hwcontext.h
index 04d19d89c2b8..e24cb1470595 100644
--- a/libavutil/hwcontext.h
+++ b/libavutil/hwcontext.h
@@ -538,6 +538,18 @@ enum {
      * be much lower than normal memory.
      */
     AV_HWFRAME_MAP_DIRECT    = 1 << 3,
+    /**
+     * For multi-planar frames, the mapping must break the frame into
+     * single-plane formats.  For instance, an NV12 frame must be split into a
+     * a 8-bit format for the Y plane and an interleaved 16-bit format for the
+     * U/V plane with 2x2 subsampling.
+     */
+    AV_HWFRAME_MAP_SEPARATE_PLANES = 1 << 4,
+    /**
+     * For multi-planar frames, the mapping must result in a multi-planar
+     * format.  This is the contrary of AV_HWFRAME_MAP_SEPARATE_PLANES.
+     */
+    AV_HWFRAME_MAP_COMBINED_PLANES = 1 << 5,
 };
 
 /**
diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
index 2227d6ed6981..62eb1e2908b3 100644
--- a/libavutil/hwcontext_vaapi.c
+++ b/libavutil/hwcontext_vaapi.c
@@ -1161,7 +1161,11 @@ static int vaapi_map_to_drm_esh(AVHWFramesContext *hwfc, AVFrame *dst,
 
     surface_id = (VASurfaceID)(uintptr_t)src->data[3];
 
-    export_flags = VA_EXPORT_SURFACE_SEPARATE_LAYERS;
+    export_flags = 0;
+    if (flags & AV_HWFRAME_MAP_COMBINED_PLANES)
+        export_flags |= VA_EXPORT_SURFACE_COMPOSED_LAYERS;
+    else
+        export_flags |= VA_EXPORT_SURFACE_SEPARATE_LAYERS;
     if (flags & AV_HWFRAME_MAP_READ)
         export_flags |= VA_EXPORT_SURFACE_READ_ONLY;
     if (flags & AV_HWFRAME_MAP_WRITE)
-- 
2.30.1