[FFmpeg-devel] [FFmpeg-cvslog] swscale/output: fix signed integer overflow for ya16

[Michael Niedermayer]

On Thu, Sep 26, 2019 at 01:57:52PM +0000, Paul B Mahol wrote:
> ffmpeg | branch: master | Paul B Mahol <onemda at gmail.com> | Thu Sep 26 15:55:03 2019 +0200| [21838cad2fc44023ad85e35d5c677e2f8d29a0ef] | committer: Paul B Mahol
> 
> swscale/output: fix signed integer overflow for ya16
> 
> Fixes #7666.
> 
> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=21838cad2fc44023ad85e35d5c677e2f8d29a0ef
> ---
> 
>  libswscale/output.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/libswscale/output.c b/libswscale/output.c
> index 7eb4644ce6..0e20a0a6b8 100644
> --- a/libswscale/output.c
> +++ b/libswscale/output.c
> @@ -904,7 +904,7 @@ yuv2ya16_X_c_template(SwsContext *c, const int16_t *lumFilter,
>  
>      for (i = 0; i < dstW; i++) {
>          int j;
> -        int Y = 1 << 18;
> +        int64_t Y = 1 << 18;
>          int64_t A = 0xffff<<14;

This fix is not optimal, 64bit intermediates should not be needed

Ill post a patch which does this without the 64bit

also the alpha code looks somewhat wrong, ive a fix but no testcase
will post that too

Thanks

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Elect your leaders based on what they did after the last election, not
based on what they say before an election.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20190926/52927287/attachment.sig>