[FFmpeg-trac] #8263(undetermined:new): signed integer overflow at libavfilter/vf_convolution.c:139

[FFmpeg]

#8263: signed integer overflow at libavfilter/vf_convolution.c:139
-------------------------------------+-------------------------------------
             Reporter:  Suhwan       |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:               |                  Version:  git-
  undetermined                       |  master
             Keywords:  ubsan        |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 Summary of the bug:
 There are 3 signed integer overflow and 1 outside the range of
 representable values of type 'int' at libavfilter/vf_convolution.c:139

 I compiled ffmpeg with "--toolchain=clang-usan" to check the undefined-
 behaviours and attached log file.
 How to reproduce:
 {{{
 % ffmpeg_g -y -i $PoC -filter_complex roberts -target dvd -loglevel 0
 tmp.subviewer1

 ffmpeg version N-95336-g4f4334bcbc Copyright (c) 2000-2019 the FFmpeg
 developers
 built with clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
 configuration: --cc=clang --cxx=clang++ --ld=clang --enable-debug
 --toolchain=clang-usan
 }}}

 Here's UBSAN log

 {{{
 libavfilter/vf_convolution.c:139:48: runtime error: signed integer
 overflow: -65535 * -65535 cannot be represented in type 'int'
 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
 libavfilter/vf_convolution.c:139:48 in
 libavfilter/vf_convolution.c:139:36: runtime error: signed integer
 overflow: 65535 * 65535 cannot be represented in type 'int'
 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
 libavfilter/vf_convolution.c:139:36 in
 libavfilter/vf_convolution.c:139:26: runtime error: -nan is outside the
 range of representable values of type 'int'
 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
 libavfilter/vf_convolution.c:139:26 in
 libavfilter/vf_convolution.c:139:42: runtime error: signed integer
 overflow: 1712056129 + 1886425489 cannot be represented in type 'int'
 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
 libavfilter/vf_convolution.c:139:42 in
 }}}
 Please confirm.
 Thanks

--
Ticket URL: <https://trac.ffmpeg.org/ticket/8263>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker