id	summary	reporter	owner	description	type	status	priority	component	version	resolution	keywords	cc	blockedby	blocking	reproduced	analyzed
1752	hqdn3d crash (assembly)	Cigaes		"A particular combination of pixels cause hqdn3d to crash.

How to reproduce:
{{{
$ ./ffmpeg_g -loglevel debug -s 2x4 -pix_fmt yuv420p -i /tmp/t.raw -vf hqdn3d -f null -
ffmpeg version N-44586-gb90210e Copyright (c) 2000-2012 the FFmpeg developers
  built on Sep 19 2012 12:24:19 with gcc 4.7 (Debian 4.7.1-7)
  configuration: --enable-shared --disable-static --enable-gpl --enable-libx264 --enable-libass --enable-libfreetype --assert-level=1
  libavutil      51. 73.101 / 51. 73.101
  libavcodec     54. 56.100 / 54. 56.100
  libavformat    54. 27.101 / 54. 27.101
  libavdevice    54.  2.100 / 54.  2.100
  libavfilter     3. 16.104 /  3. 16.104
  libswscale      2.  1.101 /  2.  1.101
  libswresample   0. 15.100 /  0. 15.100
  libpostproc    52.  0.100 / 52.  0.100
[AVIOContext @ 0x1a8caa0] Statistics: 12 bytes read, 0 seeks
Input #0, image2, from '/tmp/t.raw':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0, 1, 1/25: Video: rawvideo (I420 / 0x30323449), yuv420p, 2x4, 1/25, 25 tbr, 25 tbn, 25 tbc
[Parsed_hqdn3d_0 @ 0x1a8cd40] ls:4.000000 cs:3.000000 lt:6.000000 ct:4.500000
[buffer @ 0x1a8ea00] Setting entry with key 'video_size' to value '2x4'
[buffer @ 0x1a8ea00] Setting entry with key 'pix_fmt' to value '0'
[buffer @ 0x1a8ea00] Setting entry with key 'time_base' to value '1/25'
[buffer @ 0x1a8ea00] Setting entry with key 'pixel_aspect' to value '0/1'
[buffer @ 0x1a8ea00] Setting entry with key 'sws_param' to value 'flags=2'
[buffer @ 0x1a8ea00] Setting entry with key 'frame_rate' to value '25/1'
[graph 0 input from stream 0:0 @ 0x1a8ce40] w:2 h:4 pixfmt:yuv420p tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf54.27.101
    Stream #0:0, 0, 1/90000: Video: rawvideo (I420 / 0x30323449), yuv420p, 2x4, 1/25, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
Press [q] to stop, [?] for help
zsh: segmentation fault
}}}

The sample file contains:

{{{
0000000: b586 1c00 0000 3c8f 7f7f 7f7f
}}}

valgrind says:

{{{
==25957== Invalid read of size 2
==25957==    at 0x50B965E: ??? (hqdn3d.asm:103)
==25957==    by 0xE5877C7: ???
==25957==    by 0x50A2724: end_frame (vf_hqdn3d.c:115)
==25957==    by 0x50B1BC0: ff_end_frame (video.c:342)
==25957==    by 0x506759A: request_frame (buffersrc.c:379)
==25957==    by 0x5067785: av_buffersrc_add_ref (buffersrc.c:152)
==25957==    by 0x5067967: av_buffersrc_add_frame (buffersrc.c:91)
==25957==    by 0x416BF6: decode_video (ffmpeg.c:1646)
==25957==    by 0x4093E8: main (ffmpeg.c:1761)
==25957==  Address 0xffffffffee57aee0 is not stack'd, malloc'd or (recently) free'd
}}}

gdb says:

{{{
Program received signal SIGSEGV, Segmentation fault.
ff_hqdn3d_row_8_x86.loop2 () at libavfilter/x86/hqdn3d.asm:103
103     HQDN3D_ROW 8
(gdb) where
#0  ff_hqdn3d_row_8_x86.loop2 () at libavfilter/x86/hqdn3d.asm:103
#1  0x00000000006329c8 in ?? ()
#2  0x00002aaaaaf41725 in denoise_spatial (temporal=0x645480, spatial=0x641420, depth=8, dstride=32, sstride=<optimized out>, 
    h=4, w=2, frame_ant=0xffffffff, line_ant=0x635080, dst=<optimized out>, src=<optimized out>, hqdn3d=0x632bc0)
    at libavfilter/vf_hqdn3d.c:115
#3  denoise_depth (depth=8, temporal=0x643480, spatial=<optimized out>, dstride=32, sstride=<optimized out>, 
    h=<optimized out>, w=<optimized out>, frame_ant_ptr=<optimized out>, line_ant=0x635080, dst=<optimized out>, 
    src=<optimized out>, hqdn3d=0x632bc0) at libavfilter/vf_hqdn3d.c:153
#4  end_frame (inlink=<optimized out>) at libavfilter/vf_hqdn3d.c:338

rax            0x645480 6575232
rbx            0xffffffff       4294967295
rcx            0x6329ca 6498762
rdx            0x635082 6508674
rsi            0x636581 6514049
rdi            0x636581 6514049
rbp            0x1      0x1
rsp            0x7fffffffc940   0x7fffffffc940
r8             0x0      0
r9             0x641420 6558752
r10            0x7      7
r11            0xfffffffff0000000       -268435456
r12            0x1      1
r13            0x635080 6508672
r14            0x641420 6558752
r15            0x645480 6575232
rip            0x2aaaaaf5865e   0x2aaaaaf5865e <ff_hqdn3d_row_8_x86.loop2+52>
eflags         0x10296  [ PF AF SF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
}}}

The crash does not happen if assembly is disabled. The arch setting is ARCH_X86_64.

(The crash also happens with a real-world image, I just cropped very tightly.)"	defect	closed	normal	avfilter	git-master	fixed	hqdn3d asm crash segv				1	0