[FFmpeg-trac] #8202(undetermined:new): outside the range of unsigned long at libavcodec/ratecontrol.c

[FFmpeg]

#8202: outside the range of unsigned long at libavcodec/ratecontrol.c
-------------------------------------+-------------------------------------
             Reporter:  Suhwan       |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:               |                  Version:  git-
  undetermined                       |  master
             Keywords:  ubsan        |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 Summary of the bug:
 There is outside the range of unsigned long at
 libavcodec/ratecontrol.c:967:37
 {{{
 libavcodec/ratecontrol.c:967:37: runtime error: -4.52978e+06 is outside
 the range of representable values of type 'unsigned long'
 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
 libavcodec/ratecontrol.c:967:37 in

 #0  ff_rate_estimate_qscale (s=0x626000000100, dry_run=0)
     at libavcodec/ratecontrol.c:967
 #1  0x000000000437e887 in estimate_qp (s=0x626000000100, dry_run=0)
     at libavcodec/mpegvideo_enc.c:3607
 #2  0x000000000436ae4e in encode_picture (s=0x626000000100,
 picture_number=6)
     at libavcodec/mpegvideo_enc.c:3807
 #3  0x000000000433e284 in ff_mpv_encode_picture (avctx=0x619000002380,
     pkt=0x60e000010780, pic_arg=<optimized out>,
 got_packet=0x7fffffffae00)
     at libavcodec/mpegvideo_enc.c:1868
 #4  0x000000000313a8d1 in avcodec_encode_video2 (avctx=0x619000002380,
     avpkt=<optimized out>, frame=0x616000050480, got_packet_ptr=<optimized
 out>)
     at libavcodec/encode.c:302
 #5  0x000000000313da5f in do_encode (avctx=<optimized out>,
 frame=0x616000050480,
     got_packet=0x7fffffffae00) at libavcodec/encode.c:371
 #6  0x000000000313d13f in avcodec_send_frame (avctx=<optimized out>,
     frame=0x616000050480) at libavcodec/encode.c:420
 #7  0x000000000065b93d in do_video_out (of=0x60a0000002c0,
 ost=0x616000001e80,
     next_picture=0x616000050480, sync_ipts=4.9406564584124654e-324)
     at fftools/ffmpeg.c:1287
 #8  0x00000000006536ee in reap_filters (flush=0) at fftools/ffmpeg.c:1504
 #9  0x00000000005e7433 in transcode_step () at fftools/ffmpeg.c:4648
 #10 transcode () at fftools/ffmpeg.c:4692
 #11 0x00000000005db6ec in main (argc=<optimized out>, argv=<optimized
 out>)
     at fftools/ffmpeg.c:4894
 }}}

 How to reproduce:
 {{{
 % ./ffmpeg_g -stream_loop 23 -y -i flower-palette-02.tif -target vcd
 -loglevel 0 -map 0 -c:a:48 pcm_f64le -c:v:0 h263p -ab 420k -ac 3 -strict 3
 tmp.x11grab

 ffmpeg version N-94982-gea673a0edb Copyright (c) 2000-2019 the FFmpeg
 developers
   built with clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
   configuration: --cc=clang --cxx=clang++ --ld=clang --enable-debug
 --toolchain=clang-usan
 }}}

--
Ticket URL: <https://trac.ffmpeg.org/ticket/8202>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker