[FFmpeg-trac] #10723(avutil:new): Issue in vulkan_export_to_cuda when dealing with RGBA frames

[FFmpeg]

#10723: Issue in vulkan_export_to_cuda when dealing with RGBA frames
-----------------------------------+---------------------------------------
             Reporter:  Xaphiosis  |                     Type:  defect
               Status:  new        |                 Priority:  normal
            Component:  avutil     |                  Version:  unspecified
             Keywords:             |               Blocked By:
             Blocking:             |  Reproduced by developer:  0
Analyzed by developer:  0          |
-----------------------------------+---------------------------------------
 Summary of the bug:

 This comes from the ALVR project. We were trying to get a Vulkan-backed
 AVFrame (with vulkan pixel format VK_FORMAT_R8G8B8A8_UNORM and FFMPEG
 pixel format `AV_PIX_FMT_BGRA`) copied to a CUDA-backed AVFrame with same
 `AV_PIX_FMT_BGRA` (for encoding with nvenc).
 It kept dying with CUDA_ERROR_INVALID_VALUE at `cuMemcpy2D`.
 After re-implementing the export and doing the copy manually with CUDA, we
 found the following:

 When requesting the external memory with
 `cuExternalMemoryGetMappedMipmappedArray`, the `arrayDesc` `.NumChannels`
 is
 set in `vulkan_export_to_cuda` as:
   `.NumChannels = 1 + ((planes == 2) && i)`
 but our vulkan frame is VK_FORMAT_R8G8B8A8_UNORM on the vulkan side,
 AV_PIX_FMT_BGRA on the FFMPEG side, which means `.NumChannels` should be
 4, and there's no way it can possibly be 4. It looks like this will work
 with NV12, but cannot work in our case, as number of channels will be set
 to 1.

 After discussing, the following change was proposed, but it's not clear to
 me whether this will work in all cases either:

 {{{
 diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
 index 204b57c011..e3bd6ace9b 100644
 --- a/libavutil/hwcontext_vulkan.c
 +++ b/libavutil/hwcontext_vulkan.c
 @@ -2859,7 +2859,7 @@ static int vulkan_export_to_cuda(AVHWFramesContext
 *hwfc,
                  .arrayDesc = {
                      .Depth = 0,
                      .Format = cufmt,
 -                    .NumChannels = 1 + ((planes == 2) && i),
 +                    .NumChannels = desc->comp[i].step,
                      .Flags = 0,
                  },
                  .numLevels = 1,
 }}}

 You can see the relevant ALVR PR here for more context:
 https://github.com/alvr-org/ALVR/pull/1911
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10723>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker