[FFmpeg-devel] [PATCH] avcodec/cuviddec: Add handling HDR10+ sidedata on cuviddec.

Mon Sep 23 14:37:34 EEST 2024

> Its not possible for a decoder to save memory by scaling references.
> Decoding would simply break.
> Only way of saving memory while decoding is to simply get rid of output
> frames as soon as possible.

Maybe I didn't explain myself well, but if I set a breakpoint and step over
the code while looking at nvidia-smi, I can show you that decoding the
video with cuviddec leaving the "resize" option untouched (so decoding
outputs frames at 1080p) uses over 100MB more GPU Memory than
setting the "resize" flag to 640x360, and the decoding doesn't break at all
(just tested it on my NVIDIA GeForce RTX 4070 Ti with ffmpeg 7.0.2, and
we used to have similar results back when we were using ffmpeg 4.3.6).

> NVDEC does not implement fixed-function downscaling, in fact none of the
> desktop cards have any hardware dedicated to that.
> As far as I know, scaling, deinterlacing, and generally all
post-processing
> is done on the compute engine via cuda. This is still pretty efficient
> since the data can be shared between the decode/compute engines without
> copy.

This is a good point, and something I haven't verified but would be curious
to check. Still, the biggest downside we see is GPU VRAM consumption, as
explained above :(

On Fri, Sep 20, 2024 at 9:34 PM Lynne via ffmpeg-devel <
ffmpeg-devel at ffmpeg.org> wrote:

> On 20/09/2024 20:41, Carlos Ruiz wrote:
> >> It's just that the cuviddec decoder is more of a relic, from before the
> >> native hwaccel existed.
> >> The only reason it's not straight up deprecated is that it's sometimes
> >> nice to have a "second opinion" when issues crop up, and there are a few
> >> specific features like hardware-deinterlacing that can't be exposed via
> >> the native hwaccel.
> >>
> >> But I'd normally not like to expand it even further and add complex and
> >> large features to it. Whenever possible, the native nvdec hwaccel should
> >> be used.
> >
> > As someone who recently switched from *_cuvid to * + hwaccel=cuda I have
> to
> > agree
> > that the performance (timing-wise at least) has been very comparable for
> > decoding.
> >
> > The only comment I'd like to make though, and this might be a bit of
> > unpopular opinion
> > based on some other threads I read, but there are huge advantages of
> hwaccel
> > accelerating not just decoding but also resizing (and I guess optionally
> > cropping).
> > I know that ideally a decoder should only decode, but think about a
> common
> > usecase
> > in the AI world we live in: you get a bunch of simultaneous 4k (or 1080p)
> > incoming
> > rtsp streams and you want to decode the video and pass it through some ML
> > model,
> > e.g. in TensorRT (to stick with the Nvidia example). The native hevc
> codec
> > doesn't
> > support resizing, so you decode video at full 4k on the gpu, which means
> > allocating
> > something like 5-10 surfaces at 3840x2160 which becomes 250MB of GPU
> memory,
> > and then you have immediately take all of those frames, pass them
> through a
> > filterchain,
> > scale them down to e.g. 640x360, and waste CUDA cores instead of
> leveraging
> > the
> > dedicated video downsizing inside the NVDEC chip. Now do that for 50
> camera
> > streams
> > and you'll quickly run out of GPU memory with a GPU utilization under 10%
> > haha.
> >
> > This is exactly why I submitted a patch yesterday that would allow using
> > the hevc
> > codec with nvdec hwaccel, while resizing on the gpu like hevc_cuviddec
> > does, and
> > the memory (and GPU) consuption goes waaay down (e.g. 6MB of GPU VRAM
> > instead of 250MB per camera). I know this is a different discussion but
> > thought
> > it was appropriate to share because deprecating cuviddec or rejecting my
> > patch
> > would leave part of the community out.
> >
> >
> > On Fri, Sep 20, 2024 at 12:48 PM Timo Rothenpieler <
> timo at rothenpieler.org>
> > wrote:
> >
> >> On 20/09/2024 04:08, 김윤주 wrote:
> >>> Does native decoder refer to hevc (hevcdec.c)?
> >>> I tried using hevc and in environments with low CPU performance,
> >> hevc_cuvid
> >>> was much faster.
> >>> So, I used hevc_cuvid for decoding but encountered an issue where
> HDR10+
> >>> sidedata did not exist.
> >>> That's why I wrote this patch.
> >>
> >> You did turn on hwaccel, right?
> >> I don't see why the native decoder would be much slower at parsing then
> >> nvidias parsers.
> >>
> >>> I thought that the original content's sidedata should be preserved even
> >>> after decoding regardless of which decoder is used.
> >>> And I thought hevc_cuvid was the only way to get Nvidia hwaccel
> support.
> >>> If I'm mistaken about anything, please let me know.
> >>
> >> It's just that the cuviddec decoder is more of a relic, from before the
> >> native hwaccel existed.
> >> The only reason it's not straight up deprecated is that it's sometimes
> >> nice to have a "second opinion" when issues crop up, and there are a few
> >> specific features like hardware-deinterlacing that can't be exposed via
> >> the native hwaccel.
> >>
> >> But I'd normally not like to expand it even further and add complex and
> >> large features to it. Whenever possible, the native nvdec hwaccel should
> >> be used.
> >>
> >>> Also, is it correct to respond to your comments like this?
> >>> It seems quite different from the format you sent.
> >>
> >> Top-posting isn't exactly liked here, though I don't really have a
> >> strong opinion on it.
> >>
> >>> Apologies, as I'm still relatively new to the FFmpeg community and
> have a
> >>> lot to learn.
> >>> Any additional guidance would be greatly appreciated.
> >>>
> >>> -----Original Message-----
> >>> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Timo
> >>> Rothenpieler
> >>> Sent: Friday, September 20, 2024 4:48 AM
> >>> To: ffmpeg-devel at ffmpeg.org
> >>> Subject: Re: [FFmpeg-devel] [PATCH] avcodec/cuviddec: Add handling
> HDR10+
> >>> sidedata on cuviddec.
> >>>
> >>> On 19.09.2024 06:43, yoonjoo wrote:
> >>>> Implemented decoding of NAL units and handling HDR10+ sidedata by
> >>>> referring to hevcdec.
> >>>
> >>> Why? Can't you just use the native decoder with nvdec hwaccel?
> >>> _______________________________________________
> >>> ffmpeg-devel mailing list
> >>> ffmpeg-devel at ffmpeg.org
> >>> https://protect2.fireeye.com/v1/url?k=c11e9bb6-9e85a2a9-c11f10f9-
> >>> 000babdfecba-c4d76825032a0b52&q=1&e=342e88a0-295e-4dc1-a57f-
> >>>
> >>
> 8f5aab974009&u=https%3A%2F%2Fffmpeg.org%2Fmailman%2Flistinfo%2Fffmpeg-devel
> >>>
> >>> To unsubscribe, visit link above, or email
> >> ffmpeg-devel-request at ffmpeg.org
> >>> with subject "unsubscribe".
> >>>
> >>> _______________________________________________
> >>> ffmpeg-devel mailing list
> >>> ffmpeg-devel at ffmpeg.org
> >>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >>>
> >>> To unsubscribe, visit link above, or email
> >>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> >>
> >> _______________________________________________
> >> ffmpeg-devel mailing list
> >> ffmpeg-devel at ffmpeg.org
> >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >>
> >> To unsubscribe, visit link above, or email
> >> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> >>
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>
> Its not possible for a decoder to save memory by scaling references.
> Decoding would simply break.
> Only way of saving memory while decoding is to simply get rid of output
> frames as soon as possible.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>