[FFmpeg-devel] [PATCH v2] configure: fix the bigendian test

[David Michael]

There are two issues:

When build systems enable LTO in CFLAGS, the unused global integer
does not make it into the compiled object file.  As a workaround,
check if the compiler understands -fno-lto and append it after
CFLAGS while building the endianness test.

The hexdump output is line-wrapped, so the expected value will not
be matched when its four bytes are split across two lines.  Use the
POSIX "-A n" option to disable printing input offsets and delete
all newline characters to output continuous hex values to grep.

Signed-off-by: David Michael <fedora.dm0 at gmail.com>
---
 configure | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/configure b/configure
index 51e43fbf66..47160b4bbc 100755
--- a/configure
+++ b/configure
@@ -5756,10 +5756,11 @@ done
 check_cc pragma_deprecated "" '_Pragma("GCC diagnostic ignored \"-Wdeprecated-declarations\"")'
 
 # The global variable ensures the bits appear unchanged in the object file.
-test_cc <<EOF || die "endian test failed"
+test_cflags -fno-lto && override_flags=-fno-lto || override_flags=
+test_cc $override_flags <<EOF || die "endian test failed"
 unsigned int endian = 'B' << 24 | 'I' << 16 | 'G' << 8 | 'E';
 EOF
-od -t x1 $TMPO | grep -q '42 *49 *47 *45' && enable bigendian
+od -A n -t x1 $TMPO | tr -d '\n' | grep -q '42 *49 *47 *45' && enable bigendian
 
 check_cc const_nan math.h "struct { double d; } static const bar[] = { { NAN } }"
 
-- 
2.26.2