sws_scale overwrites out[0][-1] on big endian

[cehoyos]

Current FFmpeg outputs a warning message from free() on ppc OSX, this is a regression since 27744fe / 77cfb2

$ ffmpeg -i tests/lena.pnm -vf format=rgba out.png
ffmpeg version N-49548-ga60530e Copyright (c) 2000-2013 the FFmpeg developers
  built on Feb  3 2013 00:48:44 with gcc 4.2.1 (GCC) (Apple Inc. build 5577)
  configuration: --cc=gcc-4.2 --enable-gpl
  libavutil      52. 17.100 / 52. 17.100
  libavcodec     54. 91.100 / 54. 91.100
  libavformat    54. 61.104 / 54. 61.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 35.100 /  3. 35.100
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, image2, from 'tests/lena.pnm':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: ppm, rgb24, 256x256, 25 tbr, 25 tbn, 25 tbc
Output #0, image2, to 'out.png':
  Metadata:
    encoder         : Lavf54.61.104
    Stream #0:0: Video: png, rgba, 256x256, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (ppm -> png)
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A
video:191kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.011248%
ffmpeg(42660) malloc: *** error for object 0x197d011: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug

(gdb) r -i tests/lena.pnm -vf format=rgba out.png
Starting program: ffmpeg_g -i tests/lena.pnm -vf format=rgba out.png
Reading symbols for shared libraries . done
ffmpeg version N-49548-ga60530e Copyright (c) 2000-2013 the FFmpeg developers
  built on Feb  3 2013 00:48:44 with gcc 4.2.1 (GCC) (Apple Inc. build 5577)
  configuration: --cc=gcc-4.2 --enable-gpl
  libavutil      52. 17.100 / 52. 17.100
  libavcodec     54. 91.100 / 54. 91.100
  libavformat    54. 61.104 / 54. 61.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 35.100 /  3. 35.100
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, image2, from 'tests/lena.pnm':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: ppm, rgb24, 256x256, 25 tbr, 25 tbn, 25 tbc
Output #0, image2, to 'out.png':
  Metadata:
    encoder         : Lavf54.61.104
    Stream #0:0: Video: png, rgba, 256x256, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (ppm -> png)
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A
video:191kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.011248%
ffmpeg_g(42676) malloc: *** error for object 0x1186011: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug

Breakpoint 1, 0x96da3e68 in malloc_error_break ()
(gdb) bt
#0  0x96da3e68 in malloc_error_break ()
#1  0x96d9ead0 in szone_error ()
#2  0x0068ddb8 in av_freep (arg=0x100b5f0) at libavutil/mem.c:181
#3  0x00034c74 in ff_free_pool (pool=0x100b550) at libavfilter/buffer.c:107
#4  0x0003206c in avfilter_free (filter=0x100ab10) at libavfilter/avfilter.c:162
#5  0x00032a04 in avfilter_graph_free (graph=0x100a128) at libavfilter/avfiltergraph.c:67
#6  0x0000c354 in exit_program () at ffmpeg.c:412
#7  0x96ca6968 in __cxa_finalize ()
#8  0x96ca6828 in exit ()
#9  0x00016b98 in main (argc=<value temporarily unavailable, due to optimizations>, argv=<value temporarily unavailable, due to optimizations>) at ffmpeg.c:3319

[michael]

Please post the alloc related stuff from config.h
(HAVE_POSIX_MEMALIGN, HAVE_ALIGNED_MALLOC, HAVE_MEMALIGN)
i suspect this is caused by malloc(0) somewhere

[cehoyos]

$ $ grep -i alloc config.h
#define HAVE_ALIGNED_MALLOC 0
#define HAVE_MALLOC_H 0
#define HAVE_VIRTUALALLOC 0
$ grep -i align config.h
#define HAVE_FAST_UNALIGNED 1
#define HAVE_ALIGNED_MALLOC 0
#define HAVE_ALIGNED_STACK 1
#define HAVE_LOCAL_ALIGNED_16 1
#define HAVE_LOCAL_ALIGNED_8 1
#define HAVE_MEMALIGN 0
#define HAVE_POSIX_MEMALIGN 0
#define CONFIG_FAST_UNALIGNED 1
#define CONFIG_MEMALIGN_HACK 1

[michael]

patch posted, please test

[cehoyos]

The problem is that before sws_scale() gets called from scale_slice(), out[0][-1] contains the offset for mem_align_hack from the aligned to the originally allocated position. This value gets overwritten in sws_scale() leading to an invalid address being used later for free().

[cehoyos]

The invalid write in sws_scale() on big-endian is a regression since 82e5f86 - reproducible with the following command line:
$ ffmpeg -i tests/lena.pnm -pix_fmt rgba -f null -

[cehoyos]

Using attached patch, I get the following output:

$ ffmpeg -i tests/lena.pnm -pix_fmt rgba -f null -
ffmpeg version N-49673-gb5884db Copyright (c) 2000-2013 the FFmpeg developers
  built on Feb  7 2013 13:35:12 with gcc 4.2.1 (GCC) (Apple Inc. build 5577)
  configuration: --cc=gcc-4.2 --enable-memalign-hack
  libavutil      52. 17.101 / 52. 17.101
  libavcodec     54. 91.101 / 54. 91.101
  libavformat    54. 61.104 / 54. 61.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 35.101 /  3. 35.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
Input #0, image2, from 'tests/lena.pnm':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: ppm, rgb24, 256x256, 25 tbr, 25 tbn, 25 tbc
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf54.61.104
    Stream #0:0: Video: rawvideo (RGBA / 0x41424752), rgba, 256x256, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (ppm -> rawvideo)
Press [q] to stop, [?] for help
sws_scale() begin - dst: 0x194c010, dst[-1]: 16
sws_scale() end   - dst: 0x194c010, dst[-1]: 255
[null @ 0x1009e10] Encoder did not produce proper pts, making some up.
frame=    1 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A
video:0kB audio:0kB subtitle:0 global headers:0kB muxing overhead -134.375000%
ffmpeg(33692) malloc: *** error for object 0x194c011: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug

[cehoyos]

Fixed by Michael.