[FFmpeg-trac] #9803(avcodec:new): VideoToolbox H264 RTSP stream: empty image buffer on iOS (error -12909)
FFmpeg
trac at avcodec.org
Sat Jun 4 18:31:44 EEST 2022
#9803: VideoToolbox H264 RTSP stream: empty image buffer on iOS (error -12909)
-------------------------------------+-------------------------------------
Reporter: damiaan | Type: defect
Status: new | Priority: normal
Component: avcodec | Version:
Keywords: | unspecified
videotoolbox H264 ios RTSP | Blocked By:
Blocking: | Reproduced by developer: 0
Analyzed by developer: 0 |
-------------------------------------+-------------------------------------
Summary of the bug:
When playing a H264 encoded RTSP stream on iOS 15.4.1 with
AV_PIX_FMT_VIDEOTOOLBOX, the stream fails to play. The Videotoolbox code
path results in a OSStatus -12909 passed into the
videotoolbox_decoder_callback. The image_buffer argument is also NULL.
The issue occurs with both FFMPEG version 5.0.1 and a version built from
the master branch, commit hash
https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/c6364b711bad1fe2fbd90e5b2798f87080ddf5ea.
I have also tested the same RTSP stream against ffmpeg running on MacOS
12.3.1 FFMPEG version 4.0 fails with the following error log, repeated for
each frame:
{{{
cur_dts is invalid (this is harmless if it occurs once at the start per
stream)
[h264 @ 0x7fb517816e00] nal_unit_type: 9, nal_ref_idc: 0
[h264 @ 0x7fb517816e00] nal_unit_type: 8, nal_ref_idc: 1
[h264 @ 0x7fb517816e00] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x7fb517816e00] nal_unit_type: 1, nal_ref_idc: 1
[h264 @ 0x7fb517816e00] ct_type:0 pic_struct:1
[h264 @ 0x7fb517816e00] VideoToolbox session not available.
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be
initialized.
[h264 @ 0x7fb517816e00] decode_slice_header error
[h264 @ 0x7fb517816e00] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
cur_dts is invalid (this is harmless if it occurs once at the start per
stream)
[h264 @ 0x7fb517817400] nal_unit_type: 9, nal_ref_idc: 0
[h264 @ 0x7fb517817400] nal_unit_type: 8, nal_ref_idc: 1
[h264 @ 0x7fb517817400] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x7fb517817400] nal_unit_type: 1, nal_ref_idc: 1
[h264 @ 0x7fb517817400] ct_type:0 pic_struct:2
[h264 @ 0x7fb517817400] VideoToolbox session not available.
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be
initialized.
[h264 @ 0x7fb517817400] decode_slice_header error
[h264 @ 0x7fb517817400] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
cur_dts is invalid (this is harmless if it occurs once at the start per
stream)
[h264 @ 0x7fb517817a00] nal_unit_type: 9, nal_ref_idc: 0
[h264 @ 0x7fb517817a00] nal_unit_type: 8, nal_ref_idc: 1
[h264 @ 0x7fb517817a00] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x7fb517817a00] nal_unit_type: 1, nal_ref_idc: 0
[h264 @ 0x7fb517817a00] ct_type:0 pic_struct:1
[h264 @ 0x7fb517817a00] VideoToolbox session not available.
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be
initialized.
[h264 @ 0x7fb517817a00] decode_slice_header error
[h264 @ 0x7fb517817a00] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
cur_dts is invalid (this is harmless if it occurs once at the start per
stream)
[h264 @ 0x7fb517818000] nal_unit_type: 9, nal_ref_idc: 0
[h264 @ 0x7fb517818000] nal_unit_type: 8, nal_ref_idc: 1
[h264 @ 0x7fb517818000] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x7fb517818000] nal_unit_type: 1, nal_ref_idc: 0
[h264 @ 0x7fb517818000] ct_type:0 pic_struct:2
[h264 @ 0x7fb517818000] VideoToolbox session not available.
}}}
FFMPEG 5.0.1 on MacOS succeeds with the following log (trimmed down to a
couple of frames):
{{{
[NULL @ 0x7fe0bc904e40] ct_type:0 pic_struct:1
[h264 @ 0x7fe09bf61600] ct_type:0 pic_struct:2
[h264 @ 0x7fe09bf6b580] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x7fe09bf6b580] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x7fe09bf6b580] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x7fe09bf6b580] nal_unit_type: 1(Coded slice of a non-IDR
picture), nal_ref_idc: 0
[NULL @ 0x7fe0bc904e40] ct_type:0 pic_struct:2
[h264 @ 0x7fe09bf75500] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x7fe09bf75500] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x7fe09bf75500] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x7fe09bf75500] nal_unit_type: 1(Coded slice of a non-IDR
picture), nal_ref_idc: 0
[NULL @ 0x7fe0bc904e40] ct_type:0 pic_struct:1
[h264 @ 0x7fe09bf75500] ct_type:0 pic_struct:2
[h264 @ 0x7fe09bf7f480] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x7fe09bf7f480] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x7fe09bf7f480] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x7fe09bf7f480] nal_unit_type: 1(Coded slice of a non-IDR
picture), nal_ref_idc: 1
[NULL @ 0x7fe0bc904e40] ct_type:0 pic_struct:2
[h264 @ 0x7fe09bf89400] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x7fe09bf89400] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x7fe09bf89400] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x7fe09bf89400] nal_unit_type: 1(Coded slice of a non-IDR
picture), nal_ref_idc: 1
}}}
The MacOS command I use to run FFMPEG is as follows:
{{{
ffmpeg -benchmark -hwaccel videotoolbox -i ~/Desktop/test.mp4 -f null
-loglevel debug -
}}}
The iOS debug log is as follows:
{{{
[rtsp @ 0x159fdf0d0] SDP:
v=0
s=udp://239.192.3.33:5000
c=IN IP4 10.130.142.80
m=video 0 RTP/AVP 96
a=control:trackID=0
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1
[rtsp @ 0x159fdf0d0] video codec set to: h264
[rtsp @ 0x159fdf0d0] RTP Packetization Mode: 1
[rtsp @ 0x159fdf0d0] setting jitter buffer size to 0
[rtsp @ 0x159fdf0d0] hello state=0
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15b6579b0] nal_unit_type: 7(SPS), nal_ref_idc: 1
[h264 @ 0x15b6579b0] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15b6579b0] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15b6579b0] nal_unit_type: 5(IDR), nal_ref_idc: 1
[h264 @ 0x15b6579b0] Format yuv420p chosen by get_format().
[h264 @ 0x15b6579b0] Reinit context to 720x480, pix_fmt: yuv420p
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[rtsp @ 0x159fdf0d0] first_dts 0 not matching first dts NOPTS (pts NOPTS,
duration 1501) in the queue
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[rtsp @ 0x159fdf0d0] All info found
[rtsp @ 0x159fdf0d0] rfps: 59.000000 0.009195
[rtsp @ 0x159fdf0d0] rfps: 59.000000 0.009195
[rtsp @ 0x159fdf0d0] rfps: 60.000000 0.000000
[rtsp @ 0x159fdf0d0] rfps: 120.000000 0.000000
[rtsp @ 0x159fdf0d0] rfps: 240.000000 0.000001
[rtsp @ 0x159fdf0d0] rfps: 59.940060 0.000031
[rtsp @ 0x159fdf0d0] rfps: 59.940060 0.000031
[h264 @ 0x15c660c40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 7(SPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 5(IDR), nal_ref_idc: 1
[h264 @ 0x15c660c40] Format videotoolbox_vld chosen by get_format().
[h264 @ 0x15c660c40] Format videotoolbox_vld requires hwaccel
initialisation.
[h264 @ 0x15c660c40] Reinit context to 720x480, pix_fmt: videotoolbox_vld
[h264 @ 0x15c660c40] vt decoder cb: output image buffer is null
[h264 @ 0x15c660c40] hardware accelerator failed to decode picture
[h264 @ 0x15c660c40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 1(Coded slice of a non-IDR picture),
nal_ref_idc: 1
[h264 @ 0x15c660c40] ct_type:0 pic_struct:2
[h264 @ 0x15c660c40] no picture
[h264 @ 0x15c660c40] VideoToolbox decoder needs reconfig, restarting..
[h264 @ 0x15c660c40] vt decoder cb: output image buffer is null
[h264 @ 0x15c660c40] hardware accelerator failed to decode picture
[h264 @ 0x15c660c40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 1(Coded slice of a non-IDR picture),
nal_ref_idc: 1
[h264 @ 0x15c660c40] VideoToolbox decoder needs reconfig, restarting..
[h264 @ 0x15c660c40] vt decoder cb: output image buffer is null
[h264 @ 0x15c660c40] hardware accelerator failed to decode picture
[h264 @ 0x15c660c40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 1(Coded slice of a non-IDR picture),
nal_ref_idc: 1
[h264 @ 0x15c660c40] ct_type:0 pic_struct:2
[h264 @ 0x15c660c40] VideoToolbox decoder needs reconfig, restarting..
[h264 @ 0x15c660c40] vt decoder cb: output image buffer is null
[h264 @ 0x15c660c40] hardware accelerator failed to decode picture
[h264 @ 0x15c660c40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 1(Coded slice of a non-IDR picture),
nal_ref_idc: 0
[h264 @ 0x15c660c40] VideoToolbox decoder needs reconfig, restarting..
[h264 @ 0x15c660c40] vt decoder cb: output image buffer is null
[h264 @ 0x15c660c40] hardware accelerator failed to decode picture
}}}
This log is from FFMPEG 5.0.1, not from the master commit referenced
above. The last few commits add extra logging, so a log from that FFPMEG
build will list the error code -12909 along with the line "vt decoder cb:
output image buffer is null".
No particular FFMPEG options are used on iOS in the AVDictionary options
dictionary, except for forcing the stream to be transported over TCP.
I have captured the raw RTSP stream into a small (1MB) MP4 file that is
attached to this ticket. I am very confused as to why the issue no longer
occurs in FFMPEG 5.0.1 running on MacOS 12.3.1, but it does occur in
FFMPEG 5.0.1 running on iOS 15.4.1 linked against iOS 14.0 SDK.
If there is any more information I can provide please let me know. I am
happy to contribute from a technical perspective to help solve this issue.
--
Ticket URL: <https://trac.ffmpeg.org/ticket/9803>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker
More information about the FFmpeg-trac
mailing list