[FFmpeg-trac] #5150(avcodec:new): signed integer overflow in ff_h264_decode_mb_cabac()

[FFmpeg]

#5150: signed integer overflow in ff_h264_decode_mb_cabac()
------------------------------------+-----------------------------------
             Reporter:  tsmith      |                    Owner:
                 Type:  defect      |                   Status:  new
             Priority:  normal      |                Component:  avcodec
              Version:  git-master  |               Resolution:
             Keywords:              |               Blocked By:
             Blocking:              |  Reproduced by developer:  0
Analyzed by developer:  0           |
------------------------------------+-----------------------------------

Comment (by tsmith):

 It seems this issue is triggered when calculating a value for 'const int
 index'.
 {{{const int index= 4*i + block_width*j;}}}

 Worst case this could lead to reading or writing from somewhere that was
 unintended. And in that case corrupted input or not it needs to be
 prevented to avoid potential attacks (assuming that it can access
 interesting data). On the other hand it may just turn out to be a blip in
 the output (which really doesn't matter if the input is corrupted to begin
 with). Also I'm not sure if this can be triggered by a valid file I just
 assume it can.

 The real issue here is that signed integer overflows are undefined in C
 (and C++). Since it is undefined this allows the compiler to make certain
 assumptions and optimization that can result in strange behavior. With
 signed integers it is not guaranteed that INT_MAX + 1 will result in
 INT_MIN.

 It is nice to have issues like this fixed from a bug hunting perspective
 but I understand there is more to it than that. If this truly is benign
 and not worth fixing (this also goes for similar overflows I have logged)
 then I will let you guys make the call here.

 Perhaps using '-fwrapv'[1] to make signed int overflows defined is an
 option?  This also makes them operate in the manner that most people
 assume. This may also prevent the use of some optimizations, decreasing
 performance and this is also bad, likely worse.

 [1]https://gcc.gnu.org/onlinedocs/gcc-4.0.2/gcc/Code-Gen-Options.html

--
Ticket URL: <https://trac.ffmpeg.org/ticket/5150#comment:2>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker