[FFmpeg-trac] #10771(ffmpeg:new): Encoding with hevc_videotoolbox (hardware) using Mac OS Sonoma 14.2.1 on M2 fails with error

[FFmpeg]

#10771: Encoding with hevc_videotoolbox (hardware) using Mac OS Sonoma 14.2.1 on M2
fails with error
-------------------------------------+-------------------------------------
             Reporter:  f.fischer    |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:  ffmpeg       |                  Version:
             Keywords:               |  unspecified
  hevc_videotoolbox                  |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 **Summary of the bug:**

 Encoding any input-file using hevc_videotoolbox on Mac OS Sonoma 14.2.1
 using an Apple M2 MBP / i7 MBP / M1 MacStudio fails with "Error: cannot
 create compression session: -12908".

 **What I wanted to accomplish:**

 Create a preview file of a high res videofile (in my case AVC-I 100 in MXF
 container, but any other container / codec combination I tested failed as
 well).

 **What problem I encountered:**

 ffmpeg stoppes the encoding with named error and terminates the thread.
 Detail of the error encountered (excerpt of the logfile attached):

 ''[hevc_videotoolbox @ 0x7fa551f0c440] Error: cannot create compression
 session: -12908
 [hevc_videotoolbox @ 0x7fa551f0c440] Try -allow_sw 1. The hardware encoder
 may be busy, or not supported.
 [vost#0:0/hevc_videotoolbox @ 0x7fa551f0c000] Error while opening encoder
 - maybe incorrect parameters such as bit_rate, rate, width or height.
 [vf#0:0 @ 0x7fa551f0cd00] Error sending frames to consumers: Generic error
 in an external library
 [vf#0:0 @ 0x7fa551f0cd00] Task finished with error code: -542398533
 (Generic error in an external library)
 (...)
 [vf#0:0 @ 0x7fa551f0cd00] Terminating thread with return code -542398533
 (Generic error in an external library)''

 Please find attached a full terminal output with -v 9 -loglevel 99.

 **How to reproduce:**

 Use i7 / M1 / M2 Apple computer (other chips not tested) with Mac OS
 Sonoma (tested up to 14.2.1) and use a prompt with -c:v hevc_videotoolbox.

 **The exact prompt I used:**

 ffmpeg -i "/Users/devuser/Documents/Testclips/Testclip1.mxf" -c:a aac -c:v
 hevc_videotoolbox -y "/Users/devuser/Documents/Testclips/Testclip1.mp4"

 fmpeg version N-113169-ge1c1dc8347-tessus (UTC compile date: 2024-01-01)
 as offered here: https://evermeet.cx/ffmpeg/
 with
 configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-
 version=tessus --enable-avisynth --enable-fontconfig --enable-gpl
 --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d
 --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-
 libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-
 libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-
 libopus --enable-librubberband --enable-libshine --enable-libsnappy
 --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame
 --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-
 libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-
 libx265 --enable-libxavs --enable-libxml2 --enable-libxvid --enable-
 libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-
 flags=--static --disable-ffplay

 **What I did to prevent the issue / work around it:**

 - added profile:v main or -profile:v main10 to check if there is a problem
 with it - no success, same error
 - added -pix_fmt yuv420p10le to see if I can provoke ffmpeg to check the
 value and fall back to the right colorprofile (8 bit in this case)
 - used -allow_sw 1 as suggested to enable software rendering, which works,
 but reduces rendering speed to about 25% of hardware rendering
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10771>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker