[FFmpeg-devel] [PATCH] libavcodec/amfenc: Add more pixel formats support

[Mark Thompson]

On 20/10/2023 09:13, Evgeny Pavlov wrote:
> On Tue, Jul 18, 2023 at 10:32 AM Evgeny Pavlov <lucenticus at gmail.com> wrote:
> 
>> This commit adds BGRA, RGBA and ARGB pixel formats for AMF encoders
>>
>> Signed-off-by: Evgeny Pavlov <lucenticus at gmail.com>
>> ---
>>   libavcodec/amfenc.c | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/libavcodec/amfenc.c b/libavcodec/amfenc.c
>> index cb48f8c273..234cd012ef 100644
>> --- a/libavcodec/amfenc.c
>> +++ b/libavcodec/amfenc.c
>> @@ -74,6 +74,9 @@ static const FormatMap format_map[] =
>>       { AV_PIX_FMT_NV12,       AMF_SURFACE_NV12 },
>>       { AV_PIX_FMT_BGR0,       AMF_SURFACE_BGRA },
>>       { AV_PIX_FMT_RGB0,       AMF_SURFACE_RGBA },
>> +    { AV_PIX_FMT_BGRA,       AMF_SURFACE_BGRA },
>> +    { AV_PIX_FMT_RGBA,       AMF_SURFACE_RGBA },
>> +    { AV_PIX_FMT_ARGB,       AMF_SURFACE_ARGB },
>>       { AV_PIX_FMT_GRAY8,      AMF_SURFACE_GRAY8 },
>>       { AV_PIX_FMT_YUV420P,    AMF_SURFACE_YUV420P },
>>       { AV_PIX_FMT_YUYV422,    AMF_SURFACE_YUY2 },
>> --
>> 2.37.3.windows.1
>>
>> The purpose of this patch is to fix an issue with feeding ddagrab output
> to AMF directly. The output of ddagrab might be BGRA, AMF supports this
> input, but failed to encode due to missing mapping from AV_PIX_BGRA to
> AMF_SURFACE_BGRA in amfenc.c
> DXGI isn't firm to distinguish between RGBA & RGBX, so it is safer to
> support both to avoid failures like with ddagrab.

See patch just sent to fix the bug in ddagrab that it incorrectly advertises an alpha channel.

Do you have ddagrab->amfenc working with just something to fix the alpha channel?  To make it work I also need to mess with the bind flags because amfenc wants to use the textures as shader resources.

This has been noted as a problem before, where a proper fix would require large changes in the format negotiation setup and so it hasn't been done.  I'm wondering whether adding "-bind_flags shader_resource" option to ddagrab would be a plausible hack for these cases, or whether that's a bit too obscure for an actual user?

Thanks,

- Mark