[FFmpeg-devel] [PATCH] lavfi/vf_deshake: fix segfaults #2443

[Michael Niedermayer]

On Mon, Apr 15, 2013 at 01:04:03AM -0300, João Bernardo wrote:
>     Fixed both segfaults from ticket #2443 with deshake filter:

>      1st - Array 128x128 not big enough for rx=64

fixed


>      2nd - SSE2 instruction PSADBW need memory aligned for 128bit operands
> (XMM)
> 
> diff --git a/libavcodec/x86/motion_est.c b/libavcodec/x86/motion_est.c
> index 3ffb002..d828d8a 100644
> --- a/libavcodec/x86/motion_est.c
> +++ b/libavcodec/x86/motion_est.c
> @@ -104,8 +104,10 @@ static int sad16_sse2(void *v, uint8_t *blk2, uint8_t
> *blk1, int stride, int h)
>          "1:                             \n\t"
>          "movdqu (%1), %%xmm0            \n\t"
>          "movdqu (%1, %4), %%xmm1        \n\t"
> -        "psadbw (%2), %%xmm0            \n\t"
> -        "psadbw (%2, %4), %%xmm1        \n\t"
> +        "movdqu (%2), %%xmm2            \n\t"
> +        "movdqu (%2, %4), %%xmm3        \n\t"
> +        "psadbw %%xmm2, %%xmm0          \n\t"
> +        "psadbw %%xmm3, %%xmm1          \n\t"

The input to this function must be aligned to the blocksize of 8 or 16
the caller is buggy if it calls this function on misaligned data,
the alignment requirement exists to maximize speed and to simplify
SIMD implementions

please see dsputil.h:
typedef int (*me_cmp_func)(void /*MpegEncContext*/ *s, uint8_t *blk1/*align width (8 or 16)*/, uint8_t *blk2/*align 1*/, int line_size, int h)/* __attribute__ ((const))*/;


[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

It is what and why we do it that matters, not just one of them.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20130415/84d4d78f/attachment.asc>