[FFmpeg-devel] [PATCH 1/2] avcodec/videotoolboxenc: fix undefined behavior with rc_max_rate=0

[Thomas Guillem]

On Fri, Jul 6, 2018, at 09:21, Kári Helgason wrote:
> Sorry for jumping in with a slightly OT question,
> 
> Am I understanding correctly that you have noticed that when using kVTCompre
> ssionPropertyKey_DataRateLimits you will sometimes get random garbage from
> the encoder in the output callback?

Not garbage, but the output callback will be called with a 0 error_status (OK then) but without any sample_buffer.
Then, a crash within the VT lib will follow.

> 
> How frequently does this happen, and can you reproduce it reliably, or do
> you only see it in production?
> 
> Has a radar been filed with Apple about this?

This issue doesn't reproduce on the iOS 12 beta (with the same device), so I guess they are aware or it has been fixed.

> 
> On Wed, Jul 4, 2018 at 1:44 PM Thomas Guillem <thomas at gllm.fr> wrote:
> 
> > On Wed, Jul 4, 2018, at 09:05, Thomas Guillem wrote:
> > > On macOS, a zero rc_max_rate cause an error from
> > > VTSessionSetProperty(kVTCompressionPropertyKey_DataRateLimits).
> > >
> > > on iOS (depending on device/version), a zero rc_max_rate cause invalid
> > > arguments from the vtenc_output_callback after few frames and then a
> > crash
> > > within the VideoToolbox library.
> >
> > In fact, when setting a correct max_rate on iOS, you could still get
> > random crashes the same way. It's happening on ios 11.4 but seems to be OK
> > on iOS 12 Beta.
> >
> > > ---
> > >  libavcodec/videotoolboxenc.c | 2 ++
> > >  1 file changed, 2 insertions(+)
> > >
> > > diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
> > > index ac847358ab..aa9aae7e05 100644
> > > --- a/libavcodec/videotoolboxenc.c
> > > +++ b/libavcodec/videotoolboxenc.c
> > > @@ -1019,6 +1019,7 @@ static int vtenc_create_encoder(AVCodecContext
> >  *avctx,
> > >
> > >      if (vtctx->codec_id == AV_CODEC_ID_H264) {
> > >          // kVTCompressionPropertyKey_DataRateLimits is not available
> > > for HEVC
> > > +        if (max_rate > 0) {
> > >          bytes_per_second_value = max_rate >> 3;
> > >          bytes_per_second = CFNumberCreate(kCFAllocatorDefault,
> > >                                            kCFNumberSInt64Type,
> > > @@ -1058,6 +1059,7 @@ static int vtenc_create_encoder(AVCodecContext
> > > *avctx,
> > >              av_log(avctx, AV_LOG_ERROR, "Error setting max bitrate
> > > property: %d\n", status);
> > >              return AVERROR_EXTERNAL;
> > >          }
> > > +        }
> > >
> > >          if (profile_level) {
> > >              status = VTSessionSetProperty(vtctx->session,
> > > --
> > > 2.18.0
> > >
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel at ffmpeg.org
> > > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel