[FFmpeg-devel] [WIP] XComposite window capture demuxer (Linux)

Emanuele Oriani ema at fastwebnet.it
Mon May 18 10:45:49 EEST 2020 

Hi Marton/ffmpeg devs/all,

Haven't seen any response to the proposed XComposite windows capture.

As per below the reasoning for adding this code to libav*/ffmpeg is that 
seems to be better in quality than x11grab, although it uses more CPU in 
its current implementation (uses OpenGL and PBO - optional - to fetch 
XComposite windows).

Please let me know if it's still of interest and/or you have further 
technical feedback.
I've posted some performance tests in the thread below.

Here links of two captures (one with x11grab, the other with xcompgrab) 
and the sources:

https://send.firefox.com/download/df859689f4cc6429/#EN6w9upyIMHqO7n_1hMTaQ
https://github.com/Emanem/replayer/blob/master/src/xcompgrab.c

Thanks in advance for your time and feedback,
Emanuele

Ps. Apologies if I've missed reposes on this list

On 10/05/2020 19:01, Emanuele Oriani wrote:
> Hi Marton/all,
> 
> I've re-uploaded the firefox send link (expired) with some sample 
> capture at 60 FPS:
> 
> https://send.firefox.com/download/51b45decae720c08/#r5o4J2SgCJZndRdMOxxRBg
> 
> Please note I have integrated usage of OpenGL's Pixel Buffer Objects - 
> now the memory management of the captured framed can be managed by the 
> drivers and hopefully should be DMA and less memory hops.
> This new code works on both Nvidia and AMD.
> 
> CPU usage is still higher than x11grab (similar levels as below).
> 
> Let me know your thoughts.
> 
> Thanks,
> Emanuele
> 
> Ps. quick link to demuxer: 
> https://github.com/Emanem/replayer/blob/master/src/xcompgrab.c
> 
> On 08/05/2020 11:43, Emanuele Oriani wrote:
>> Hi Marton,
>>
>> Capturing at 60 FPS 1920x1080, the difference is even more noticeable: 
>> x11grab is choppy, xcompgrab is smooth (this time I used H264 
>> ultrafast to encode).
>>
>> https://send.firefox.com/download/17001ef60837a5ec/#FaQTa-dP4MB28YfPNMXxuw 
>>
>>
>> CPU performance:
>>
>>               real    user   sys
>> - xcompgrab: 10.32s, 18.2s, 0.38s
>> - x11grab:   10.20s, 11.6s, 0.22s
>>
>> Definitely much more CPU usage, but the output quality can't be compared.
>> Please note no frames were lost/skipped, everything was processed on 
>> time for both tests.
>>
>> Thanks,
>> Emanuele
>>
>> On 08/05/2020 11:05, Emanuele Oriani wrote:
>>> Hi Marton,
>>>
>>> TL;DR
>>> xcompgrab uses more CPU but produces much better streams than x11grab.
>>>
>>> I have the following CPU usage performance report (on my i7-8700k, 
>>> Nvidia 2080 Ti RTX, governor set to 'performance', on Ubuntu 18.04 
>>> using Gnome shell).
>>> The capture has been a surface area of  1720x1376 for 10 seconds at 
>>> 30 FPS, only getting the frames, not trying to encode them into a file:
>>>
>>>               real    user   sys
>>> - xcompgrab: 10.13s, 2.06s, 0.085s
>>> - x11grab:   10.05s, 0.20s, 0.035s
>>>
>>> from a first outlook x11grab is much more CPU efficient, almost 90% 
>>> less CPU usage, but upon starting to record the video, it feels like 
>>> it's 'choppier' and loses frames.
>>>
>>> Below videos have been captured at 1920x1080, the encoder is exactly 
>>> the same with same settings and the total CPU time is greater for 
>>> xcompgrab (as expected).
>>>
>>> But if you look at both, the xcompgrab doesn't lose frames/is not 
>>> choppier, instead x11grab seems to be much worse.
>>>
>>> I've uploaded both 10 secs segments at:
>>> https://send.firefox.com/download/0cbf01f9126519c7/#DPKzIfjemAFm263ZSdMOBw 
>>>
>>>
>>> The code used to grab and encode is 
>>> https://github.com/Emanem/replayer again very simple, one thread gets 
>>> the packets/frames, the other picks the frames and encodes them.
>>>
>>> Performance of this run has been:
>>>
>>>               real    user   sys
>>> - xcompgrab: 10.42s, 12.5s, 0.22s
>>> - x11grab:   10.21s, 8.90s, 0.31s
>>>
>>> As expected x11grab yields better CPU usage but the quality of output 
>>> is somehow lacking.
>>>
>>> Let me know your thoughts.
>>>
>>> Thanks,
>>> Emanuele
>>>
>>> On 07/05/2020 19:52, Marton Balint wrote:
>>>>
>>>>
>>>> On Thu, 7 May 2020, Emanuele Oriani wrote:
>>>>
>>>>> Hi FFMPEG devel,
>>>>>
>>>>> I have been writing a simple XComposite window capture demuxer, 
>>>>> heavily inspired from x11grab sources and OBS Window capture 
>>>>> logic/code.
>>>>
>>>> Have you compared performance to x11grab for various resolutions and 
>>>> frame rates? Do you have some numbers?
>>>>
>>>> Thanks,
>>>> Marton
>>>>
>>>>>
>>>>> I would like to give back to the community and share the sources; 
>>>>> before I would formally submit for a review/patch, I would really 
>>>>> appreciate if someone could have a high level pass at the code 
>>>>> (it's relatively simple actually) and point out any major 
>>>>> issues/problems/...
>>>>>
>>>>> The code is available as part of a simple test application on 
>>>>> github, at:
>>>>>
>>>>> https://github.com/Emanem/replayer/blob/master/src/xcompgrab.c
>>>>>
>>>>> Being the author of the code I would be happy to re-license it to 
>>>>> be used in ffmpeg/libav* mainline once I get the OK from you.
>>>>> In order to compile/execute this code, one needs the following dev 
>>>>> packages:
>>>>>
>>>>> libav*-dev, libx11-dev, libxcomposite-dev, OpenGL headers/libs.
>>>>>
>>>>> Libraries to be linked are:
>>>>>
>>>>> -lavcodec -lavformat -lavdevice -lavutil -lswscale -lX11 
>>>>> -lXcomposite -lGL
>>>>>
>>>>> I've tested it and it's working fine on both AMD and Nvidia hardware.
>>>>>
>>>>> Let me know if this is of interest, otherwise no worries and thanks 
>>>>> again for the fantastic library/software.
>>>>>
>>>>> Emanuele
>>>>>
>>>>> _______________________________________________
>>>>> ffmpeg-devel mailing list
>>>>> ffmpeg-devel at ffmpeg.org
>>>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>>>
>>>>> To unsubscribe, visit link above, or email
>>>>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>>>> _______________________________________________
>>>> ffmpeg-devel mailing list
>>>> ffmpeg-devel at ffmpeg.org
>>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>>
>>>> To unsubscribe, visit link above, or email
>>>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel at ffmpeg.org
>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".

More information about the ffmpeg-devel mailing list