[FFmpeg-devel] [PATCH 4/6] lavu/hwcontext_vaapi: add vaapi_format_map support for AYUV/Y210/Y410

[Mark Thompson]

On 10/09/2019 17:07, Linjie Fu wrote:
> There is no VA_RT_FORMAT_AYUV in defined in libva, and currently in
> media-driver, VA_FOURCC_AYUV is used to represent VA_RT_FORMAT_AYUV.

That doesn't make sense - VA_RT_FORMAT_* is a bit mask, so using VA_FOURCC_AYUV looks like a random combination of other values.
> Another patch could be sent to refine the code after this issue is
> addressed:
> https://github.com/intel/libva/issues/335
> 
> Signed-off-by: Linjie Fu <linjie.fu at intel.com>
> ---
>  libavutil/hwcontext_vaapi.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
> index cf11764..724bbeb 100644
> --- a/libavutil/hwcontext_vaapi.c
> +++ b/libavutil/hwcontext_vaapi.c
> @@ -116,6 +116,15 @@ static const VAAPIFormatDescriptor vaapi_format_map[] = {
>  #endif
>      MAP(UYVY, YUV422,  UYVY422, 0),
>      MAP(YUY2, YUV422,  YUYV422, 0),
> +#ifdef VA_FOURCC_Y210
> +    MAP(Y210, YUV422_10,Y210, 0),
> +#endif
> +#define VA_RT_FORMAT_AYUV VA_FOURCC_AYUV
> +    MAP(AYUV,   AYUV,     AYUV, 0),
> +#undef VA_RT_FORMAT_AYUV
> +#ifdef VA_FOURCC_Y410
> +    MAP(Y410, YUV444_10,Y410, 0),

That looks suspicious - you've defined Y410 as having an alpha channel, but that render target format doesn't have one.

> +#endif
>      MAP(411P, YUV411,  YUV411P, 0),
>      MAP(422V, YUV422,  YUV440P, 0),
>      MAP(444P, YUV444,  YUV444P, 0),

To try to clarify the intent here, the formats you are intending to use for this new decoder are:

4:2:0:
* 8-bit: NV12.
* 10-bit: P010.
* With alpha: not supported.

4:2:2:
* 8-bit: YUYV or similar?
* 10-bit: Y210.
* With alpha: not supported.

4:4:4:
* 8-bit: the existing 444P, or not supported?
* 10-bit: not supported.
* 8-bit + alpha: AYUV.
* 10-bit + alpha: Y410?  (With the alpha channel truncated?)

Can you fill in the missing parts here?

- Mark