[FFmpeg-trac] #9799(avfilter:open): Rotate filter to 16-bit per channel image causes false color

[FFmpeg]

#9799: Rotate filter to 16-bit per channel image causes false color
--------------------------------------+------------------------------------
             Reporter:  hayamdk       |                    Owner:  (none)
                 Type:  defect        |                   Status:  open
             Priority:  important     |                Component:  avfilter
              Version:  git-master    |               Resolution:
             Keywords:  rotate ubsan  |               Blocked By:
             Blocking:                |  Reproduced by developer:  1
Analyzed by developer:  0             |
--------------------------------------+------------------------------------
Changes (by Carl Eugen Hoyos):

 * keywords:   => rotate ubsan
 * status:  reopened => open
 * version:  unspecified => git-master
 * component:  undetermined => avfilter
 * reproduced:  0 => 1

Comment:

 Please test if the following change fixes the issue:
 {{{
 diff --git a/libavfilter/vf_rotate.c b/libavfilter/vf_rotate.c
 index 4429e3d543..d319dfe3d9 100644
 --- a/libavfilter/vf_rotate.c
 +++ b/libavfilter/vf_rotate.c
 @@ -269,8 +269,8 @@ static uint8_t *interpolate_bilinear16(uint8_t
 *dst_color,
          int s01 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize *
 int_y ]);
          int s10 = AV_RL16(&src[src_linestep * int_x  + i + src_linesize *
 int_y1]);
          int s11 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize *
 int_y1]);
 -        int s0 = (((1<<16) - frac_x)*s00 + frac_x*s01);
 -        int s1 = (((1<<16) - frac_x)*s10 + frac_x*s11);
 +        int64_t s0 = (((int64_t)(1<<16) - frac_x)*s00 +
 (int64_t)frac_x*s01);
 +        int64_t s1 = (((int64_t)(1<<16) - frac_x)*s10 +
 (int64_t)frac_x*s11);

          AV_WL16(&dst_color[i], ((int64_t)((1<<16) - frac_y)*s0 +
 (int64_t)frac_y*s1) >> 32);
      }
 }}}
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/9799#comment:26>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker