[FFmpeg-devel] [PATCH] Check rc_buffer_size value using integer comparison

[Michael Niedermayer]

On Tue, Sep 07, 2010 at 08:25:31PM +0100, Mans Rullgard wrote:
> For certain input values, the floating-point value calculated
> for bit_rate*time_base may, due to rounding, become minutely
> larger than rc_buffer_size, thus causing the configuration to
> be rejected in error.  Converting to int before the comparison
> truncates any such overshoot.
> 
> This fixes the mxf regression test with gcc 4.5 on x86_32.
> ---
>  libavcodec/mpegvideo_enc.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c
> index 9f8682c..f55883b 100644
> --- a/libavcodec/mpegvideo_enc.c
> +++ b/libavcodec/mpegvideo_enc.c
> @@ -351,7 +351,7 @@ av_cold int MPV_encode_init(AVCodecContext *avctx)
>          av_log(avctx, AV_LOG_INFO, "impossible bitrate constraints, this will fail\n");
>      }
>  
> -    if(avctx->rc_buffer_size && avctx->bit_rate*av_q2d(avctx->time_base) > avctx->rc_buffer_size){
> +    if(avctx->rc_buffer_size && (int)(avctx->bit_rate*av_q2d(avctx->time_base)) > avctx->rc_buffer_size){

should be

avctx->bit_rate*(int64_t)avctx->time_base.num > avctx->rc_buffer_size * (int64_t)avctx->time_base.den

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Thouse who are best at talking, realize last or never when they are wrong.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100907/8552edf6/attachment.pgp>