encoding bgra ljpeg crash

[richardpl]

Summary of the bug:
How to reproduce:

> ./ffmpeg -i ~/dxtory_mic.avi -pix_fmt bgra /tmp/out.ljpg                                     
ffmpeg version git-2013-01-28-3c98205 Copyright (c) 2000-2013 the FFmpeg developers
  built on Jan 28 2013 19:33:22 with FreeBSD clang version 3.1 (branches/release_31 156863) 20120523
  configuration: --cc=clang --as=clang --extra-cflags=-fcolor-diagnostics --extra-cflags=-I/usr/local/incl
ude --extra-ldflags=-L/usr/local/lib --samples=../fate-suite --disable-debug --enable-static --disable-pos
tproc --disable-indev=v4l2 --disable-indev=alsa --disable-indev=bktr --disable-indev=dshow --disable-indev
=dv1394 --disable-indev=fbdev --disable-indev=jack --disable-indev=libcdio --disable-indev=libdc1394 --dis
able-indev=openal --disable-indev=pulse --disable-indev=sndio --disable-indev=vfwcap --disable-outdev=sndi
o --disable-outdev=alsa --disable-filter=mp --disable-doc --enable-nonfree --enable-openssl --enable-gpl -
-enable-x11grab --disable-shared
  libavutil      52. 17.100 / 52. 17.100
  libavcodec     54. 91.100 / 54. 91.100
  libavformat    54. 61.104 / 54. 61.104
  libavdevice    54.  3.102 / 54.  3.102
  libavfilter     3. 34.101 /  3. 34.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
Guessed Channel Layout for  Input Stream #0.1 : stereo
Guessed Channel Layout for  Input Stream #0.2 : stereo
Input #0, avi, from '/home/r/dxtory_mic.avi':
  Metadata:
    encoder         : DxtoryCore ver2.0.0.109
    ISRC            : Video:YUV420 Audio0:Lautsprecher (Realtek High Definition Audio) Audio1:Mikrofon (Re
altek High Definition Audio)
  Duration: 00:00:07.48, start: 0.000000, bitrate: 1496 kb/s
    Stream #0:0: Video: dxtory (xtor / 0x726F7478), yuv420p, 1280x720, 25 tbr, 25 tbn, 25 tbc
    Stream #0:1: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s
    Stream #0:2: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s
Output #0, image2, to '/tmp/out.ljpg':
  Metadata:
    ISRC            : Video:YUV420 Audio0:Lautsprecher (Realtek High Definition Audio) Audio1:Mikrofon (Re
altek High Definition Audio)
    encoder         : Lavf54.61.104
    Stream #0:0: Video: ljpeg, bgra, 1280x720, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (dxtory -> ljpeg)
Press [q] to stop, [?] for help
Segmentation fault

[cehoyos]

Regression since 642a655

[cehoyos]

(gdb) r -i tests/lena.pnm out.ljpg
Starting program: ffmpeg_g -i tests/lena.pnm out.ljpg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ffmpeg version N-49414-g11c99c7 Copyright (c) 2000-2013 the FFmpeg developers
  built on Jan 29 2013 14:43:08 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl --disable-indev=jack
  libavutil      52. 17.100 / 52. 17.100
  libavcodec     54. 91.100 / 54. 91.100
  libavformat    54. 61.104 / 54. 61.104
  libavdevice    54.  3.102 / 54.  3.102
  libavfilter     3. 34.101 /  3. 34.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, image2, from 'tests/lena.pnm':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: ppm, rgb24, 256x256, 25 tbr, 25 tbn, 25 tbc
Output #0, image2, to 'out.ljpg':
  Metadata:
    encoder         : Lavf54.61.104
    Stream #0:0: Video: ljpeg, bgr24, 256x256, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (ppm -> ljpeg)
Press [q] to stop, [?] for help

Program received signal SIGSEGV, Segmentation fault.
encode_picture_lossless (avctx=0x15fb600, pkt=0x7fffffffd920, pict=<optimized out>,
    got_packet=0x7fffffffd814) at libavcodec/ljpegenc.c:82
82                  buffer[0][i]= 1 << (9 - 1);
(gdb) bt
#0  encode_picture_lossless (avctx=0x15fb600, pkt=0x7fffffffd920, pict=<optimized out>,
    got_packet=0x7fffffffd814) at libavcodec/ljpegenc.c:82
#1  0x0000000000999e08 in avcodec_encode_video2 (avctx=avctx@entry=0x15fb600,
    avpkt=avpkt@entry=0x7fffffffd920, frame=frame@entry=0x7fffffffd980,
    got_packet_ptr=got_packet_ptr@entry=0x7fffffffd814) at libavcodec/utils.c:1439
#2  0x000000000045c51d in do_video_out (in_picture=0x1648c20, ost=0x1600d20, s=0x15faca0)
    at ffmpeg.c:948
#3  reap_filters () at ffmpeg.c:1101
#4  0x000000000044eaa5 in transcode_step () at ffmpeg.c:3093
#5  transcode () at ffmpeg.c:3136
#6  main (argc=<optimized out>, argv=<optimized out>) at ffmpeg.c:3311
(gdb) disass $pc-32,$pc+32
Dump of assembler code from 0x84b9b4 to 0x84b9f4:
   0x000000000084b9b4 <encode_picture_lossless+1396>:   decl   0x98249c(%rbx)
   0x000000000084b9ba <encode_picture_lossless+1402>:   add    %al,(%rax)
   0x000000000084b9bc <encode_picture_lossless+1404>:   mov    0x1838(%r13),%r8
   0x000000000084b9c3 <encode_picture_lossless+1411>:   movslq 0x12d8(%r13),%rdx
   0x000000000084b9ca <encode_picture_lossless+1418>:   test   %ebx,%ebx
   0x000000000084b9cc <encode_picture_lossless+1420>:   mov    %r8,0x88(%rsp)
=> 0x000000000084b9d4 <encode_picture_lossless+1428>:   movw   $0x100,(%r8)
   0x000000000084b9da <encode_picture_lossless+1434>:   movw   $0x100,0x2(%r8)
   0x000000000084b9e1 <encode_picture_lossless+1441>:   movw   $0x100,0x4(%r8)
   0x000000000084b9e8 <encode_picture_lossless+1448>:   jle    0x84b8ec <encode_picture_lossless+1196>
   0x000000000084b9ee <encode_picture_lossless+1454>:   mov    0x70(%rsp),%r8d
   0x000000000084b9f3 <encode_picture_lossless+1459>:   mov    %rdx,0xa8(%rsp)
End of assembler dump.
(gdb) info register
rax            0x7ffff7e77040   140737352527936
rbx            0x100    256
rcx            0x10f0   4336
rdx            0x300    768
rsi            0x8      8
rdi            0x7ffff7e7725b   140737352528475
rbp            0x7fffffffd980   0x7fffffffd980
rsp            0x7fffffffd670   0x7fffffffd670
r8             0x0      0
r9             0x11f    287
r10            0xfa     250
r11            0x7ffff7e77237   140737352528439
r12            0x0      0
r13            0x15fba60        23050848
r14            0x7fffffffd920   140737488345376
r15            0x15fccf8        23055608
rip            0x84b9d4 0x84b9d4 <encode_picture_lossless+1428>
eflags         0x10206  [ PF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0