[FFmpeg-devel] [PATCH 14/14] videotoolbox: remove opengl compatibility attribute

Wang Bin wbsecg1 at gmail.com
Sat Dec 16 10:19:13 EET 2017

2017-12-16 15:33 GMT+08:00 Xiaolei Yu <dreifachstein at gmail.com>:
> On 12/16/2017 02:07 PM, Wang Bin wrote:
>> 2017-12-16 2:52 GMT+08:00 wm4 <nfxjfg at googlemail.com>:
>>> On Fri, 15 Dec 2017 15:02:44 +0800
>>> wbsecg1 at gmail.com wrote:
>>>> From: wang-bin <wbsecg1 at gmail.com>
>>>> 1. a cvpixelbuffer backed by iosurface can always be converted to an opengl texture, using CGLTexImageIOSurface2D for macOS, and undocumented api texImageIOSurface(which is internally used by public api CVOpenGLESTextureCacheCreateTextureFromImage) for iOS4.0+.
>>>> 2. enabling the attribute can slow down decoding speed a lot. I tested many video clips on my macbook air. for example: ffmpeg -ss 00:00:00 -t 00:03:00 -hwaccel videotoolbox -an -i big_buck_bunny_1080p_h264.mov -f null ->/dev/null, result with the attribute
>>>> enabled: frame= 2082 fps= 85 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=7.34x
>>>> disabled: frame= 2031 fps=104 q=-0.0 Lsize=N/A time=00:03:00.00 bitrate=N/A speed=9.22x
>>>> ---
>>>>  libavcodec/videotoolbox.c | 5 -----
>>>>  1 file changed, 5 deletions(-)
>>>> diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c
>>>> index 9d2f0afa20..24631684d7 100644
>>>> --- a/libavcodec/videotoolbox.c
>>>> +++ b/libavcodec/videotoolbox.c
>>>> @@ -664,11 +664,6 @@ static CFDictionaryRef videotoolbox_buffer_attributes_create(int width,
>>>>      CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfacePropertiesKey, io_surface_properties);
>>>>      CFDictionarySetValue(buffer_attributes, kCVPixelBufferWidthKey, w);
>>>>      CFDictionarySetValue(buffer_attributes, kCVPixelBufferHeightKey, h);
>>>> -    CFDictionarySetValue(buffer_attributes, kCVPixelBufferOpenGLESCompatibilityKey, kCFBooleanTrue);
>>>> -#else
>>>> -    CFDictionarySetValue(buffer_attributes, kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey, kCFBooleanTrue);
>>>> -#endif
>>>>      CFRelease(io_surface_properties);
>>>>      CFRelease(cv_pix_fmt);
>>> Does this have a negative effect on compatibility or performance? (In
>>> both cases I'm asking about the case when actually using GL rendering.)
>> Disabling the attribute improves performance in my tests. I can not
>> find any document about these keys. What i know is the decoded
>> cvpixelbuffer is backed by iosurface, and the api to create texture
>> from iosurface is available since macOS10.6
> Maybe you can add a flag to make them optional?
> They are documented at:
> https://developer.apple.com/documentation/corevideo/kcvpixelbufferopenglescompatibilitykey
> https://developer.apple.com/documentation/corevideo/kcvpixelbufferiosurfaceopengltexturecompatibilitykey
> Things may still work without them but I would like to follow the documentation whenever possible. And I think iOS DOES require the flag for those buffers to be consumed through GLES.

Your links explain nothing. iOS does not need it as mentioned in the
patch. I tested on iphone4s+iOS9 and some new devices.

> Pure speculation here. GPU texture units usually require special memory layouts that may be suboptimal for the decoder. When these constraints are not met either the driver has to perform the conversion or you pay the penalty at access time.

Not sure about memory layout. The fact is on mac the performance is
much better without it. No difference on ios.

More information about the ffmpeg-devel mailing list