[FFmpeg-trac] #1752(avfilter:new): hqdn3d crash (assembly)
FFmpeg
trac at avcodec.org
Wed Sep 19 12:34:32 CEST 2012
#1752: hqdn3d crash (assembly)
-------------------------------------+-------------------------------------
Reporter: Cigaes | Owner:
Type: defect | Status: new
Priority: normal | Component: avfilter
Version: git- | Keywords: hqdn3d asm
master | crash segv
Blocked By: | Blocking:
Reproduced by developer: 0 | Analyzed by developer: 0
-------------------------------------+-------------------------------------
A particular combination of pixels cause hqdn3d to crash.
How to reproduce:
{{{
$ ./ffmpeg_g -loglevel debug -s 2x4 -pix_fmt yuv420p -i /tmp/t.raw -vf
hqdn3d -f null -
ffmpeg version N-44586-gb90210e Copyright (c) 2000-2012 the FFmpeg
developers
built on Sep 19 2012 12:24:19 with gcc 4.7 (Debian 4.7.1-7)
configuration: --enable-shared --disable-static --enable-gpl --enable-
libx264 --enable-libass --enable-libfreetype --assert-level=1
libavutil 51. 73.101 / 51. 73.101
libavcodec 54. 56.100 / 54. 56.100
libavformat 54. 27.101 / 54. 27.101
libavdevice 54. 2.100 / 54. 2.100
libavfilter 3. 16.104 / 3. 16.104
libswscale 2. 1.101 / 2. 1.101
libswresample 0. 15.100 / 0. 15.100
libpostproc 52. 0.100 / 52. 0.100
[AVIOContext @ 0x1a8caa0] Statistics: 12 bytes read, 0 seeks
Input #0, image2, from '/tmp/t.raw':
Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
Stream #0:0, 1, 1/25: Video: rawvideo (I420 / 0x30323449), yuv420p,
2x4, 1/25, 25 tbr, 25 tbn, 25 tbc
[Parsed_hqdn3d_0 @ 0x1a8cd40] ls:4.000000 cs:3.000000 lt:6.000000
ct:4.500000
[buffer @ 0x1a8ea00] Setting entry with key 'video_size' to value '2x4'
[buffer @ 0x1a8ea00] Setting entry with key 'pix_fmt' to value '0'
[buffer @ 0x1a8ea00] Setting entry with key 'time_base' to value '1/25'
[buffer @ 0x1a8ea00] Setting entry with key 'pixel_aspect' to value '0/1'
[buffer @ 0x1a8ea00] Setting entry with key 'sws_param' to value 'flags=2'
[buffer @ 0x1a8ea00] Setting entry with key 'frame_rate' to value '25/1'
[graph 0 input from stream 0:0 @ 0x1a8ce40] w:2 h:4 pixfmt:yuv420p tb:1/25
fr:25/1 sar:0/1 sws_param:flags=2
Output #0, null, to 'pipe:':
Metadata:
encoder : Lavf54.27.101
Stream #0:0, 0, 1/90000: Video: rawvideo (I420 / 0x30323449), yuv420p,
2x4, 1/25, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
Press [q] to stop, [?] for help
zsh: segmentation fault
}}}
The sample file contains:
{{{
0000000: b586 1c00 0000 3c8f 7f7f 7f7f
}}}
valgrind says:
{{{
==25957== Invalid read of size 2
==25957== at 0x50B965E: ??? (hqdn3d.asm:103)
==25957== by 0xE5877C7: ???
==25957== by 0x50A2724: end_frame (vf_hqdn3d.c:115)
==25957== by 0x50B1BC0: ff_end_frame (video.c:342)
==25957== by 0x506759A: request_frame (buffersrc.c:379)
==25957== by 0x5067785: av_buffersrc_add_ref (buffersrc.c:152)
==25957== by 0x5067967: av_buffersrc_add_frame (buffersrc.c:91)
==25957== by 0x416BF6: decode_video (ffmpeg.c:1646)
==25957== by 0x4093E8: main (ffmpeg.c:1761)
==25957== Address 0xffffffffee57aee0 is not stack'd, malloc'd or
(recently) free'd
}}}
gdb says:
{{{
Program received signal SIGSEGV, Segmentation fault.
ff_hqdn3d_row_8_x86.loop2 () at libavfilter/x86/hqdn3d.asm:103
103 HQDN3D_ROW 8
(gdb) where
#0 ff_hqdn3d_row_8_x86.loop2 () at libavfilter/x86/hqdn3d.asm:103
#1 0x00000000006329c8 in ?? ()
#2 0x00002aaaaaf41725 in denoise_spatial (temporal=0x645480,
spatial=0x641420, depth=8, dstride=32, sstride=<optimized out>,
h=4, w=2, frame_ant=0xffffffff, line_ant=0x635080, dst=<optimized
out>, src=<optimized out>, hqdn3d=0x632bc0)
at libavfilter/vf_hqdn3d.c:115
#3 denoise_depth (depth=8, temporal=0x643480, spatial=<optimized out>,
dstride=32, sstride=<optimized out>,
h=<optimized out>, w=<optimized out>, frame_ant_ptr=<optimized out>,
line_ant=0x635080, dst=<optimized out>,
src=<optimized out>, hqdn3d=0x632bc0) at libavfilter/vf_hqdn3d.c:153
#4 end_frame (inlink=<optimized out>) at libavfilter/vf_hqdn3d.c:338
rax 0x645480 6575232
rbx 0xffffffff 4294967295
rcx 0x6329ca 6498762
rdx 0x635082 6508674
rsi 0x636581 6514049
rdi 0x636581 6514049
rbp 0x1 0x1
rsp 0x7fffffffc940 0x7fffffffc940
r8 0x0 0
r9 0x641420 6558752
r10 0x7 7
r11 0xfffffffff0000000 -268435456
r12 0x1 1
r13 0x635080 6508672
r14 0x641420 6558752
r15 0x645480 6575232
rip 0x2aaaaaf5865e 0x2aaaaaf5865e
<ff_hqdn3d_row_8_x86.loop2+52>
eflags 0x10296 [ PF AF SF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
}}}
The crash does not happen if assembly is disabled. The arch setting is
ARCH_X86_64.
(The crash also happens with a real-world image, I just cropped very
tightly.)
--
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1752>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker
More information about the FFmpeg-trac
mailing list