[FFmpeg-trac] #2675(postproc:new): postproc crashes with -fstack-protector-all

FFmpeg trac at avcodec.org
Sun Jun 16 16:26:12 CEST 2013


#2675: postproc crashes with -fstack-protector-all
------------------------------------+------------------------------------
             Reporter:  cehoyos     |                    Owner:  michael
                 Type:  defect      |                   Status:  new
             Priority:  normal      |                Component:  postproc
              Version:  git-master  |               Resolution:
             Keywords:  crash       |               Blocked By:
             Blocking:              |  Reproduced by developer:  0
Analyzed by developer:  0           |
------------------------------------+------------------------------------

Comment (by cehoyos):

 The reason is apparently that gcc miscompiles dering_SSE2() and
 dering_MMX2() in libpostproc/postproc_template.c:
 {{{
 1182        "movq %%mm6, %%mm0                      \n\t" // max
 1183        "psubb %%mm7, %%mm6                     \n\t" // max - min
 1184        "push %4                              \n\t"
 1184        "movd %%mm6, %k4                        \n\t"
 1185        "cmpb "MANGLE(deringThreshold)", %b4    \n\t"
 1186        "pop %4                               \n\t"

 ...

         : : "r" (src), "r" ((x86_reg)stride), "m" (c->pQPb),
 "m"(c->pQPb2), "q"(tmp)
         : "%"REG_a, "%"REG_d
 }}}
 {{{
    0x0000000000bd40eb <dering_SSE2+187>:        movq   %mm6,%mm0
    0x0000000000bd40ee <dering_SSE2+190>:        psubb  %mm7,%mm6
    0x0000000000bd40f1 <dering_SSE2+193>:        push   %rsp
    0x0000000000bd40f2 <dering_SSE2+194>:        movd   %mm6,%esp
    0x0000000000bd40f5 <dering_SSE2+197>:        cmp    0xe3c6c0,%spl
 => 0x0000000000bd40fd <dering_SSE2+205>:        pop    %rsp
 }}}

-- 
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/2675#comment:1>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker


More information about the FFmpeg-trac mailing list