[FFmpeg-devel] [PATCH] NVENC Surface Allocation Reduction

[Timo Rothenpieler]

> This seems to be a regression introduced when filtergraph initialization related changes were passed. The below commit and its follow up seems to have caused the issue. I am looking to fix this. Please let us know if you have any suggestions in mind. Thanks
> https://github.com/FFmpeg/FFmpeg/commit/af1761f7b5b1b72197dc40934953b775c2d951cc

Yes, this initially started happening with that.
What it does for nvenc/cuvid is basically that it swapped around their 
order of initialization. Now cuvid is initialized first, and nvenc is 
not fired up until cuvid returns the first frame.
Before, nvenc was initialized first.

Initially, this completely broke cuvid->nvenc, but that was fixed with:
https://github.com/FFmpeg/FFmpeg/commit/be74ba648cf4063c9805ebe95ee83fd7299f7fd5
Kinda making me wonder why/how it ever worked in the first place.

Now only that weird max_b_frames > 0 behavior is left, but only when 
doing direct hwaccel transcoding from cuvid to nvenc.
Dropping -hwaccel cuvid, so sending frames back and forth through system 
memory, fixes it.

Using -vf hwupload_cuda works just fine with bframes as well, so it 
can't really be an issue with the PIXFMT_CUDA frame handling code in 
nvenc, as that would blow up then as well.