[FFmpeg-devel] [PATCH 1/2] avcodec/videotoolboxenc: fix undefined behavior with rc_max_rate=0

[Thomas Guillem]

On Wed, Jul 4, 2018, at 09:05, Thomas Guillem wrote:
> On macOS, a zero rc_max_rate cause an error from
> VTSessionSetProperty(kVTCompressionPropertyKey_DataRateLimits).
> 
> on iOS (depending on device/version), a zero rc_max_rate cause invalid
> arguments from the vtenc_output_callback after few frames and then a crash
> within the VideoToolbox library.

In fact, when setting a correct max_rate on iOS, you could still get random crashes the same way. It's happening on ios 11.4 but seems to be OK on iOS 12 Beta.

> ---
>  libavcodec/videotoolboxenc.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
> index ac847358ab..aa9aae7e05 100644
> --- a/libavcodec/videotoolboxenc.c
> +++ b/libavcodec/videotoolboxenc.c
> @@ -1019,6 +1019,7 @@ static int vtenc_create_encoder(AVCodecContext   *avctx,
>  
>      if (vtctx->codec_id == AV_CODEC_ID_H264) {
>          // kVTCompressionPropertyKey_DataRateLimits is not available 
> for HEVC
> +        if (max_rate > 0) {
>          bytes_per_second_value = max_rate >> 3;
>          bytes_per_second = CFNumberCreate(kCFAllocatorDefault,
>                                            kCFNumberSInt64Type,
> @@ -1058,6 +1059,7 @@ static int vtenc_create_encoder(AVCodecContext   
> *avctx,
>              av_log(avctx, AV_LOG_ERROR, "Error setting max bitrate 
> property: %d\n", status);
>              return AVERROR_EXTERNAL;
>          }
> +        }
>  
>          if (profile_level) {
>              status = VTSessionSetProperty(vtctx->session,
> -- 
> 2.18.0
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel