[FFmpeg-trac] #9599(undetermined:reopened): VideoToolbox VP9 hwaccel freezes ffmpeg
FFmpeg
trac at avcodec.org
Fri Aug 18 04:47:33 EEST 2023
#9599: VideoToolbox VP9 hwaccel freezes ffmpeg
-------------------------------------+-------------------------------------
Reporter: Eric Karnes | Owner: (none)
Type: defect | Status: reopened
Priority: normal | Component:
| undetermined
Version: git-master | Resolution:
Keywords: | Blocked By:
videotoolbox vp9 |
Blocking: | Reproduced by developer: 0
Analyzed by developer: 0 |
-------------------------------------+-------------------------------------
Comment (by low-batt):
//Note// that there are **two** problems here, the failure to decode the
frame and the **severe** problem with permanently blocking the thread,
preventing a fallback to software decoding.
Does FFmpeg have any Apple contacts? I've investigated this a bit and I'm
not seeing a good way for FFmpeg to workaround Apple's bad error handling.
See below for more details.
This problem is important as the [https://github.com/mpv-player/mpv mpv]
project added VP9 to the default value of its white-list of codecs that
are considered safe to use with hardware decoding:
[https://mpv.io/manual/stable/#options-hwdec-codecs hwdec-codecs]
The [https://github.com/iina/iina IINA] project was tripped up by this
after upgrading mpv and FFmpeg and had to quickly issue a new release with
a workaround to disable VP9 hardware decode on Intel Macs. See IINA issue
[https://github.com/iina/iina/issues/4486 #4486].
The open VLC issue [https://code.videolan.org/videolan/vlc/-/issues/27874
27874] references this FFmpeg ticket. That issue also references the
closed Chromium issue
[https://bugs.chromium.org/p/chromium/issues/detail?id=1115334 1115334].
The Chromium issue contains this interesting post (from 2020):
> After discussion with Apple this is due to the fact that they require
VP9 alt-ref frames to be packaged as superframes. I've written some code
to do this. On their side, they expect to make it an error when alt-ref
frames are sent unpacked.
As that was back in 2020 the error handling must have been updated by now,
right Apple? And if
[https://developer.apple.com/documentation/videotoolbox Video Toolbox] has
this restriction why in the case at hand is hardware decoding working
under Apple Silicon? My development machine is a MacBookPro18,2 with a M1
Max processor under macOS 13.5 and the problem does not reproduce on that
Mac.
In order to investigate this I purchased a reconditioned MacBookPro16,1
with a 6-Core Intel Core i7 processor. On that Mac using the file
[https://github.com/chromium/chromium/blob/main/media/test/data/buck-
1280x720-vp9.webm buck-1280x720-vp9.webm] from the above mentioned
Chromium issue FFmpeg hangs:
{{{
low-batt at urp ffmpeg (master *%=)$ ffmpeg -hide_banner -hwaccel
videotoolbox -i buck-1280x720-vp9.webm -f null -
Input #0, matroska,webm, from 'buck-1280x720-vp9.webm':
Metadata:
encoder : libwebm-0.2.1.0
Duration: 00:00:00.17, start: 0.000000, bitrate: 856 kb/s
Stream #0:0(eng): Video: vp9 (Profile 0), yuv420p(tv), 1280x720, SAR 1:1
DAR 16:9, 29.92 fps, 29.92 tbr, 1k tbn (default)
Stream mapping:
Stream #0:0 -> #0:0 (vp9 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
^C^C^CReceived > 3 system signals, hard exiting
}}}
FFmpeg is hanging in the method `videotoolbox_session_decode_frame` at the
call to
[https://developer.apple.com/documentation/videotoolbox/1536066-vtdecompressionsessionwaitforasy
VTDecompressionSessionWaitForAsynchronousFrames] in this code:
{{{
status = VTDecompressionSessionDecodeFrame(videotoolbox->session,
sample_buf,
0, // decodeFlags
NULL, //
sourceFrameRefCon
0); // infoFlagsOut
if (status == noErr)
status =
VTDecompressionSessionWaitForAsynchronousFrames(videotoolbox->session);
}}}
This code seems odd to me because the Apple documentation for
[https://developer.apple.com/documentation/videotoolbox/1536071-vtdecompressionsessiondecodefram
VTDecompressionSessionDecodeFrame(\_:sampleBuffer:flags:frameRefcon:infoFlagsOut:)]
says:
> **decodeFlags**
> A bitfield of directives to the decompression session and decoder.
>
> The
[https://developer.apple.com/documentation/videotoolbox/vtdecodeframeflags/kvtdecodeframe_enableasynchronousdecompression
kVTDecodeFrame_EnableAsynchronousDecompression] bit indicates whether the
video decoder may decompress the frame asynchronously.
>
> The
[https://developer.apple.com/documentation/videotoolbox/vtdecodeframeflags/kvtdecodeframe_enabletemporalprocessing
kVTDecodeFrame_EnableTemporalProcessing] bit indicates whether the decoder
may delay calls to the output callback so as to enable processing in
temporal (display) order.
>
> If both flags are clear, the decompression completes and your output
callback function will be called before VTDecompressionSessionDecodeFrame
returns.
For the `decodeFlags` parameter the above code passes 0, requesting
synchronous processing. Given that, why is FFmpeg calling
`VTDecompressionSessionWaitForAsynchronousFrames`?
A quick test on my M1 Mac with that call commented out it seemed like
decoding worked. On the Intel Mac FFmpeg quickly froze in
`videotoolbox_stop` on the call to
[https://developer.apple.com/documentation/videotoolbox/1536093-vtdecompressionsessioninvalidate
VTDecompressionSessionInvalidate]:
{{{
+ 2348 ff_videotoolbox_common_end_frame (in ffmpeg) + 191
[0x10128255f]
+ 2348 VTDecompressionSessionInvalidate (in VideoToolbox) + 83
[0x7ff815d41a79]
+ 2347 ??? (in VideoToolbox) load address 0x7ff815d38000 +
0x28c356 [0x7ff815fc4356]
+ ! 2347 FigSemaphoreWaitRelative (in CoreMedia) + 152
[0x7ff8110fb066]
+ ! 2347 WaitOnConditionTimed (in CoreMedia) + 78
[0x7ff81110e3ff]
+ ! 2347 _pthread_cond_wait (in libsystem_pthread.dylib) +
1242 [0x7ff80783d758]
+ ! 2347 __psynch_cvwait (in libsystem_kernel.dylib) + 10
[0x7ff8078010fe]
}}}
With the call to `VTDecompressionSessionInvalidate` commented out, FFmpeg
froze on the following call to
[https://developer.apple.com/documentation/corefoundation/1521153-cfrelease
CFRelease] which itself called `VTDecompressionSessionInvalidate`.
Adding some print statements shows the first frame is decoded
successfully. But the second call to `VTDecompressionSessionDecodeFrame`
returns without calling `videotoolbox_decoder_callback`. Based on that
behavior it seems there was a failure trying to decode the second frame
and yet `VTDecompressionSessionDecodeFrame` returned a successful
`OSStatus`. It also seems it did so without having signaled the internal
semaphore that indicates `VTDecompressionSessionDecodeFrame` is in
progress.
If FFmpeg wanted to continue to use
`VTDecompressionSessionWaitForAsynchronousFrames` it could detect that no
frame was generated and assume `VTDecompressionSessionDecodeFrame` failed
even though it returned success. But with the method
`VTDecompressionSessionInvalidate` also hanging I don't see how FFmpeg can
workaround the mismanagement of the semaphore by `Video Toolbox`.
Hopefully somebody has a contact at Apple that can explain why the error
handing in `VTDecompressionSessionDecodeFrame` is so badly broken and why
decoding is working fine under Apple Silicon.
This is what the output of that test looked like:
{{{
low-batt at urp ffmpeg (master *%=)$ ./ffmpeg -hide_banner -hwaccel
videotoolbox -i buck-1280x720-vp9.webm -f null -
Input #0, matroska,webm, from 'buck-1280x720-vp9.webm':
Metadata:
encoder : libwebm-0.2.1.0
Duration: 00:00:00.17, start: 0.000000, bitrate: 856 kb/s
Stream #0:0(eng): Video: vp9 (Profile 0), yuv420p(tv), 1280x720, SAR 1:1
DAR 16:9, 29.92 fps, 29.92 tbr, 1k tbn (default)
Stream mapping:
Stream #0:0 -> #0:0 (vp9 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Calling VTDecompressionSessionDecodeFrame
In videotoolbox_decoder_callback OSStatus is: 0
Returned from VTDecompressionSessionDecodeFrame OSStatus is: 0
Calling VTDecompressionSessionWaitForAsynchronousFrames
Calling VTDecompressionSessionDecodeFrame
Returned from VTDecompressionSessionDecodeFrame OSStatus is: 0
Calling VTDecompressionSessionWaitForAsynchronousFrames
}}}
--
Ticket URL: <https://trac.ffmpeg.org/ticket/9599#comment:18>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker
More information about the FFmpeg-trac
mailing list