[FFmpeg-devel] [WIP] XComposite window capture demuxer (Linux)

[Lynne]

May 19, 2020, 09:23 by ema at fastwebnet.it:

> Hi Marton,
>
> Thanks very much for the feedback; below answers to your points - let me know further feedback if any.
>
>> And sorry, I cannot say how useful this would be, maybe now is the time
>> for people to speak up if somebody is particularly against adding this
>> for any reason.
>>
>
> I haven't been able to capture video-games/3d apps running in full screen with x11grab, and when running in windowed mode the capture was sub optimal in terms of quality (lost frames/choppy/etc etc).
> Unless we have better solutions with ffmpeg/libav* (which I'm not aware of :) this xcompgrab would target such audience (smooth capture alas more CPU usage).
> But again, if there's already a better capture, then this has been an academic exercise :)
>

There already is a zero-overhead capture on linux - kmsgrab. It works on AMD and Intel.



>> - Is there a way to keep the captured textures as an
>> OpenGL/VAAPI/NVenc/Vulkan object, so the frames can work as some kind of
>> hwaccel frames? Can this improve performance?
>>
>
> I think there would be. I would need to find more in terms of documentation for both hwaccel frames structures/management.
> Please feel free to point me to guides/code/samples etc etc.
>

Uploading to hardware frames is something uses can do themselves already. Unless there is a native
interop, I don't see how doing the upload in the avdevice helps.



> Wouldn't a hwaccel frame imply no choice for encoder (as in now we get a true uncompressed RGBA buffer which can be encoded as we see fit - if we were to get hwacell wouldn't we be forced to use the encoded data as is)?
>

No.



>> - What can be the reason between the quality/smoothness difference of
>> x11grab and the Compositor capturer?
>>
>
> My suspicion is that OpenGL has access to the buffered data on the videocard itself, hence able to get smooth frames.
>
> Related to both these questions, I've asked the same on Stackoverflow:
>
> https://stackoverflow.com/questions/61613441/is-there-a-better-more-efficient-way-to-capture-composite-x-windows-in-linux
>
> Unfortunately no one has been able to give me an answer (I did set a bounty for it) so I posted my own understanding.
>

I wouldn't be surprised if the xlib code has some PTS issues and schedules downloading a frame late.
Maybe its worth fixing that instead of adding a yet another x11 capture?



>> - Maybe some openGL glue can be shared with libavdevice/opengl_enc.c?
>> Take a look, I am not sure, because that was implemented with SDL and
>> cross platform capabilities in mind, but since your capture device is
>> only for linux (or not?), then maybe it makes more sense to keep it
>> separete?
>>
>
> This is indeed a very specific implementation for Linux, I would agree that perhaps not sure this makes sense to use shared OpenGL functions?
>

I'm against any OpenGL code being ran anywhere in our libraries since it will disturb OpenGL's global
state, breaking any OpenGL users of our libraries (there are quite a lot). Yes, there are also people who
want to screencapture and use OpenGL. I'm one.
This is also why we don't have OpenGL filtering in libavfilter.
So that's a big NAK for me.

As-is now, I don't see much value in this patch.