ffmpeg-0.10.3 fails to build on x32 ABI

[devsk]

Summary of the bug: ffmpeg-0.10.3 fails to build on x32 ABI
How to reproduce:

emerge -v ffmpeg
CC      libavfilter/libmpcodecs/vf_pullup.o
libavfilter/libmpcodecs/vf_ilpack.c: Assembler messages:
libavfilter/libmpcodecs/vf_ilpack.c:116: Error: operand type mismatch for `push'
libavfilter/libmpcodecs/vf_ilpack.c:195: Error: operand type mismatch for `pop'
libavfilter/libmpcodecs/vf_ilpack.c:224: Error: operand type mismatch for `push'
libavfilter/libmpcodecs/vf_ilpack.c:307: Error: operand type mismatch for `pop'
make: *** [libavfilter/libmpcodecs/vf_ilpack.o] Error 1
make: *** Waiting for unfinished jobs....
CC      libavfilter/libmpcodecs/vf_qp.o
libavfilter/libmpcodecs/vf_mcdeint.c: In function 'config':
libavfilter/libmpcodecs/vf_mcdeint.c:199:13: warning: 'avcodec_alloc_context' is deprecated (declared at ./libavcodec/avcodec.h:4025) [-Wdeprecated-declarations]
libavfilter/libmpcodecs/vf_mcdeint.c:227:13: warning: 'avcodec_open' is deprecated (declared at ./libavcodec/avcodec.h:4155) [-Wdeprecated-declarations]
 * ERROR: media-video/ffmpeg-0.10.3 failed (compile phase):
 *   emake failed

Line 116 from libavfilter/libmpcodecs/vf_ilpack.c:

#if HAVE_EBX_AVAILABLE
static void pack_li_0_MMX(unsigned char *dst, unsigned char *y,
    unsigned char *u, unsigned char *v, int w, int us, int vs)
{
    __asm__ volatile (""
        "push %%"REG_BP" \n\t"

REG_BP defined in ./libavutil/x86_cpu.h:

#if ARCH_X86_64
...
#    define REG_SP "rsp"
#    define REG_BP "rbp"
...
#elif ARCH_X86_32
...
#    define REG_SP "esp"
#    define REG_BP "ebp"
...
#else
typedef int x86_reg;

The question is: is it hard to provide support for the new x32 ABI which is now supported in kernel 3.4.x, glibc-2.16, gcc-4.7.1, binutils-2.22?

[cehoyos]

Please report the results for current git head.

[devsk]

The same result with the git head:

libavfilter/libmpcodecs/vf_ilpack.c: Assembler messages:
libavfilter/libmpcodecs/vf_ilpack.c:116: Error: operand type mismatch for `push'
libavfilter/libmpcodecs/vf_ilpack.c:195: Error: operand type mismatch for `pop'
libavfilter/libmpcodecs/vf_ilpack.c:224: Error: operand type mismatch for `push'
libavfilter/libmpcodecs/vf_ilpack.c:307: Error: operand type mismatch for `pop'
make: *** [libavfilter/libmpcodecs/vf_ilpack.o] Error 1
make: *** Waiting for unfinished jobs....
libavfilter/libmpcodecs/vf_mcdeint.c: In function 'filter':
libavfilter/libmpcodecs/vf_mcdeint.c:104:5: warning: 'avcodec_encode_video' is deprecated (declared at ./libavcodec/avcodec.h:4162) [-Wdeprecated-declarations]
libavfilter/libmpcodecs/vf_mcdeint.c: In function 'config':
libavfilter/libmpcodecs/vf_mcdeint.c:199:13: warning: 'avcodec_alloc_context' is deprecated (declared at ./libavcodec/avcodec.h:3327) [-Wdeprecated-declarations]
libavfilter/libmpcodecs/vf_mcdeint.c:227:13: warning: 'avcodec_open' is deprecated (declared at ./libavcodec/avcodec.h:3461) [-Wdeprecated-declarations]
libavfilter/libmpcodecs/vf_pp.c: In function 'put_image':
libavfilter/libmpcodecs/vf_pp.c:150:21: warning: passing argument 1 of 'pp_postprocess' from incompatible pointer type [enabled by default]
In file included from libavfilter/libmpcodecs/vf_pp.c:36:0:
./libpostproc/postprocess.h:80:6: note: expected 'const uint8_t **' but argument is of type 'unsigned char **'
libavfilter/libmpcodecs/vf_pp.c: In function 'vf_open':
libavfilter/libmpcodecs/vf_pp.c:196:13: warning: assignment discards 'const' qualifier from pointer target type [enabled by default]
libavfilter/libmpcodecs/vf_pp.c:173:9: warning: variable 'hex_mode' set but not used [-Wunused-but-set-variable]
 * ERROR: media-video/ffmpeg-9999 failed (compile phase):
 *   emake failed
 * 
 * If you need support, post the output of `emerge --info '=media-video/ffmpeg-9999'`,
 * the complete build log and the output of `emerge -pqv '=media-video/ffmpeg-9999'`.
 * The complete build log is located at '/var/tmp/portage/media-video/ffmpeg-9999/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/media-video/ffmpeg-9999/temp/environment'.
 * Working directory: '/var/tmp/portage/media-video/ffmpeg-9999/work/ffmpeg-9999'
 * S: '/var/tmp/portage/media-video/ffmpeg-9999/work/ffmpeg-9999'
 * 
 * The following package has failed to build or install:
 * 
 *  (media-video/ffmpeg-9999::gentoo, ebuild scheduled for merge), Log file:
 *   '/var/tmp/portage/media-video/ffmpeg-9999/temp/build.log'

[reimar]

As a very first step, configure would need a reliable and portable way to detect x32, and I am not aware of one.
Until then you can use --disable-asm (which will basically make it rather useless speed-wise, but it's quite unlikely all the asm code will be ported any time soon, if ever).