[FFmpeg-devel] Problem with implementation of AC3 fixed point decoder

[Babic, Nedeljko]

Hello,

Few weeks ago I posted patche with implementation of AC3 fixed point decoder for review.
In the meantime code for AC3 floating point decoder changed and now only planar float is 
outputted, decoding is done directly into output buffers, etc.

I am trying to apply changes to AC3 fixed point code so I can prepare new patche for review 
and I have a problem. 

It looks like decoding (fixed point) is working ok after my changes, but when it exits from decoder 
it breaks on malloc in memalign (in avfilter_get_audio_buffer_ref_from_arrays_channels) with 
“malloc(): memory corruption: 0x00000000014798e0 ***”.

This happens only for fixed point decoder (I didn’t break floating point decoder with my changes) 
when I use AV_SAMPLE_FMT_S16P instead of AV_SAMPLE_FMT_FLTP.

I even tried to change only sample_fmt field in structure AVCodecContext from AV_SAMPLE_FMT_FLTP 
to AV_SAMPLE_FMT_S16P and to run floating point decoder but then it also breaks in the same way.

I also tried to set field sample_fmt to AV_SAMPLE_FMT_FLTP for fixed point decoder to see what will 
happen and then it decodes, but there are only zeroes on all channels in output file as if output 
buffers are wrong.

I am not quite sure that I understand how work with planar data is done in ffmpeg (and I think that 
the problem is there somewhere) and I guess that I am missing something outside of decoder.
 
Is it enough to change value of field sample_fmt (in struct AVCodecContext) from  AV_SAMPLE_FMT_FLTP to AV_SAMPLE_FMT_S16P (if decoding is done ok) or something else should be set?

My configuration is:
C compiler                    gcc
ARCH                            x86 (generic)
big-endian                     no
runtime cpu detection     yes
yasm                             no
MMX enabled                yes
MMXEXT enabled          yes
3DNow! enabled            yes
3DNow! extended enabled   yes
SSE enabled                   yes
SSSE3 enabled                yes
AVX enabled                  yes
FMA4 enabled                yes
CMOV enabled               yes
CMOV is fast                  yes
EBX available                  yes
EBP available                  yes
debug symbols              yes
strip symbols                 yes
optimize for size            no
optimizations                 yes
static                             yes
shared                          no
postprocessing support yes
new filter support          yes
network support            yes
threading support         pthreads
safe bitstream reader    yes
SDL support                 yes
libdxva2 enabled          no
libva enabled                no
libvdpau enabled          no
AVISynth enabled          no
frei0r enabled               no
gnutls enabled              no
libaacplus enabled        no
libass enabled              no
libcaca enabled            no
libcdio support            no
libcelt enabled             no
libdc1394 support        no
libfaac enabled            no
libfdk-aac enabled       no
libgsm enabled            no
libiec61883 support     no
libilbc enabled             no
libmodplug enabled     no
libmp3lame enabled    no
libnut enabled             no
libopencore-amrnb support no
libopencore-amrwb support no
libopencv support       no
libopenjpeg enabled   no
libopus enabled          no
libpulse enabled         no
librtmp enabled          no
libschroedinger enabled   no
libsoxr enabled           no
libspeex enabled         no
libstagefright-h264 enabled    no
libtheora enabled         no
libtwolame enabled      no
libutvideo enabled       no
libv4l2 enabled            no
libvo-aacenc support   no
libvo-amrwbenc support    no
libvorbis enabled         no
libvpx enabled             no
libx264 enabled           no
libxavs enabled           no
libxvid enabled            no
openal enabled           no
openssl enabled          no
zlib enabled                yes
bzlib enabled              yes
texi2html enabled        no
perl enabled               yes
pod2man enabled       yes
makeinfo enabled        yes

Thanks,
Nedeljko