[Libav-user] av_hwframe_transfer_data does not succeed from NV12 to YUV420P - How to get YUV420P from cuvid?

[Sergio Basurco]

Hi Mark,

I did misread that bit of documentation, thanks!

I'm trying to understand the supported_formats array found in 
hwcontext_cuda.c 
<https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/hwcontext_cuda.c#L35>. 
Can the hw context be initialized so that the decoder returns YUV420P 
instead of NV12?

Saludos / Best regards,

*Sergio Basurco*
Coherent Synchro

On 05/06/2018 22:23, Mark Thompson wrote:
> On 05/06/18 11:15, Sergio Basurco wrote:
>> I'm decoding h264 with ffmpeg. I want to use the hwaccel decoders. I'm using the cuvid decoder via API. In the fftools code there's a function "hwaccel_retrieve_data" that is supposed to convert the decoded frame (NV12) into any other format, I'm trying YUV420P.
>>
>> The conversion does not return any error, however the resulting data is not correct. Here's the original NV12 frame:
>>
>> https://imgur.com/a/jAb8h12
>>
>> And here's the conversion to YUV420P (only avframe->data[0] and avframe->data[1] have any data, data[2] is expected to have the V data but it is missing).
>>
>> https://imgur.com/a/ihQeJ0M
>>
>>
>> I think I'm on the right track, based on the code in hwcontext_cuda.c aparently YUV420P is a supported format, I cannot get my head around how I can tell the decoder to convert from NV12 to YUV420P though.
>>
>> Any tips will be appreciated, I'll update if I find anything.
> See the documentation for av_hwframe_transfer_data() (<http://ffmpeg.org/doxygen/trunk/hwcontext_8h.html#abf1b1664b8239d953ae2cac8b643815a>), in particular:
>
> "If src has an AVHWFramesContext attached, then the format of dst (if set) must use one of the formats returned by av_hwframe_transfer_get_formats(src, AV_HWFRAME_TRANSFER_DIRECTION_FROM). If dst has an AVHWFramesContext attached, then the format of src must use one of the formats returned by av_hwframe_transfer_get_formats(dst, AV_HWFRAME_TRANSFER_DIRECTION_TO)"
>
> For CUDA, no conversion during transfer is supported so the only usable output format returned by av_hwframe_transfer_get_formats() is the same format as the GPU-side frame itself (NV12 in your case): <http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavutil/hwcontext_cuda.c#l176>.
>
> - Mark
> _______________________________________________
> Libav-user mailing list
> Libav-user at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/libav-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20180606/05a135e3/attachment.html>