mxf files without FooterPartition

[zhukov]

git-master
ffmpeg -i test.mxf
outputs the following

[mxf @ 0x7fce6183d800] inconsistent FooterPartition? value: 1132608274 != 4294967296
[mxf @ 0x7fce6183d800] PreviousPartition? points to this partition or forward
[mxf @ 0x7fce6183d800] error reading header metadata

ffmpeg 0.7.8 works fine on the very same file

my rough ideas why that might be:
./libavformat/mxfdec.c:458

footer_partition = avio_rb64(pb);

wants to output -1

i created a small patch

diff --git a/libavformat/mxfdec.c b/libavformat/mxfdec.c
index 10512ef..43f247a 100644
--- a/libavformat/mxfdec.c
+++ b/libavformat/mxfdec.c
@@ -464,7 +464,7 @@ static int mxf_read_partition_pack(void *arg, AVIOContext *pb, int tag, int size

avio_read(pb, op, sizeof(UID));

/* some files don'thave FooterPartition? set in every partition */

• if (footer_partition) {

+ if (footer_partition != 4294967296) {

if (mxf->footer_partition && mxf->footer_partition != footer_partition) {

av_log(mxf->fc, AV_LOG_ERROR, "inconsistent FooterPartition? value: %" PRIi64 " != %" PRIi64 "\n",

mxf->footer_partition, footer_partition);

which did not help

[zhukov]

diff --git a/libavformat/mxfdec.c b/libavformat/mxfdec.c
index 10512ef..43f247a 100644
--- a/libavformat/mxfdec.c
+++ b/libavformat/mxfdec.c
@@ -464,7 +464,7 @@ static int mxf_read_partition_pack(void *arg, AVIOContext *pb, int tag, int size
     avio_read(pb, op, sizeof(UID));
 
     /* some files don'thave FooterPartition set in every partition */
-    if (footer_partition) {
+    if (footer_partition != 4294967296) {
         if (mxf->footer_partition && mxf->footer_partition != footer_partition) {
             av_log(mxf->fc, AV_LOG_ERROR, "inconsistent FooterPartition value: %" PRIi64 " != %" PRIi64 "\n",
                    mxf->footer_partition, footer_partition);

[cehoyos]

Please provide the sample.

[zhukov]

@cehoyos the entire file is over 1GB, first 10MB can be downloaded at  http://dl.dropbox.com/u/1109725/10bit_yuv_j2k.mxf

[cehoyos]

Do other applications play the 10MB file?

[zhukov]

this is what i've done:
ffmpeg-git -i 1GB.mxf

ffmpeg version N-36448-g1edf866, Copyright (c) 2000-2012 the FFmpeg developers
  built on Jan  4 2012 16:32:31 with llvm_gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)
  configuration: --enable-libopenjpeg
  libavutil      51. 34.100 / 51. 34.100
  libavcodec     53. 53.100 / 53. 53.100
  libavformat    53. 29.100 / 53. 29.100
  libavdevice    53.  4.100 / 53.  4.100
  libavfilter     2. 57.101 /  2. 57.101
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0.  5.100 /  0.  5.100
[mxf @ 0x7fa69303d800] inconsistent FooterPartition value: 1132608274 != 4294967296
[mxf @ 0x7fa69303d800] PreviousPartition points to this partition or forward
[mxf @ 0x7fa69303d800] error reading header metadata
1GB.mxf: Operation not permitted

dd if=1GB.mxf of=10MB.mxf bs=1m count=10
ffmpeg-git -i 10MB.mxf

ffmpeg version N-36448-g1edf866, Copyright (c) 2000-2012 the FFmpeg developers
  built on Jan  4 2012 16:32:31 with llvm_gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)
  configuration: --enable-libopenjpeg
  libavutil      51. 34.100 / 51. 34.100
  libavcodec     53. 53.100 / 53. 53.100
  libavformat    53. 29.100 / 53. 29.100
  libavdevice    53.  4.100 / 53.  4.100
  libavfilter     2. 57.101 /  2. 57.101
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0.  5.100 /  0.  5.100
[mxf @ 0x7fd88a83d800] broken or empty index
Input #0, mxf, from '10MB.mxf':
  Duration: 00:00:37.24, start: 0.000000, bitrate: 2252 kb/s
    Stream #0:0: Video: j2k, 1920x1080, 23.98 fps, 23.98 tbr, 23.98 tbn, 23.98 tbc
At least one output file must be specified

ffmpeg-0.7.8 -i 1GB.mxf

ffmpeg version 0.7.8, Copyright (c) 2000-2011 the FFmpeg developers
  built on Dec 19 2011 16:47:49 with gcc 4.2.1 (Apple Inc. build 5666) (dot 3)
  configuration: --prefix=/opt/local --enable-gpl --enable-postproc --enable-swscale --enable-avfilter --enable-libmp3lame --enable-libvorbis --enable-libtheora --enable-libdirac --enable-libschroedinger --enable-libopenjpeg --enable-libxvid --enable-libx264 --enable-libvpx --enable-libspeex --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/gcc-4.2 --arch=x86_64 --enable-yasm
  libavutil    50. 43. 0 / 50. 43. 0
  libavcodec   52.123. 0 / 52.123. 0
  libavformat  52.111. 0 / 52.111. 0
  libavdevice  52.  5. 0 / 52.  5. 0
  libavfilter   1. 80. 0 /  1. 80. 0
  libswscale    0. 14. 1 /  0. 14. 1
  libpostproc  51.  2. 0 / 51.  2. 0
[libopenjpeg @ 0x7ff0ca83fc00] Only first component will be used.
Input #0, mxf, from '1GB.mxf':
  Duration: 00:00:37.24, start: 0.000000, bitrate: 243273 kb/s
    Stream #0.0: Video: libopenjpeg, gray, 1920x1080, 23.98 fps, 23.98 tbr, 23.98 tbn, 23.98 tbc
At least one output file must be specified

ffmpeg-0.7.8 -i 10MB.mxf

ffmpeg version 0.7.8, Copyright (c) 2000-2011 the FFmpeg developers
  built on Dec 19 2011 16:47:49 with gcc 4.2.1 (Apple Inc. build 5666) (dot 3)
  configuration: --prefix=/opt/local --enable-gpl --enable-postproc --enable-swscale --enable-avfilter --enable-libmp3lame --enable-libvorbis --enable-libtheora --enable-libdirac --enable-libschroedinger --enable-libopenjpeg --enable-libxvid --enable-libx264 --enable-libvpx --enable-libspeex --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/gcc-4.2 --arch=x86_64 --enable-yasm
  libavutil    50. 43. 0 / 50. 43. 0
  libavcodec   52.123. 0 / 52.123. 0
  libavformat  52.111. 0 / 52.111. 0
  libavdevice  52.  5. 0 / 52.  5. 0
  libavfilter   1. 80. 0 /  1. 80. 0
  libswscale    0. 14. 1 /  0. 14. 1
  libpostproc  51.  2. 0 / 51.  2. 0
[libopenjpeg @ 0x7f838383fc00] Only first component will be used.
Input #0, mxf, from '10MB.mxf':
  Duration: 00:00:37.24, start: 0.000000, bitrate: 2252 kb/s
    Stream #0.0: Video: libopenjpeg, gray, 1920x1080, 23.98 fps, 23.98 tbr, 23.98 tbn, 23.98 tbc
At least one output file must be specified

[reimar]

Have you tried
ffmpeg -i - < 1GB.mxf
?
And MPlayer can play the 10MB variant, it seems to be fine, I think the demuxer only stumbles over some bad data at the end of the 1GB file.
If you can, you could try to just zero out the middle part of the file (leaving only first 10 MB and maybe last 1 MB, I suspect with the right options dd from /dev/zero should be able to do that), if it still allows to reproduce the issue compress and upload it.

[zhukov]

ffmpeg-git -i - <1GB.mxf
works fine

i created the zeroed-middle-mxf you suggested. different error.

bash-3.2$ /usr/local/bin/ffmpeg -i shallow.mxf 
ffmpeg version N-36448-g1edf866, Copyright (c) 2000-2012 the FFmpeg developers
  built on Jan  4 2012 16:32:31 with llvm_gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)
  configuration: --enable-libopenjpeg
  libavutil      51. 34.100 / 51. 34.100
  libavcodec     53. 53.100 / 53. 53.100
  libavformat    53. 29.100 / 53. 29.100
  libavdevice    53.  4.100 / 53.  4.100
  libavfilter     2. 57.101 /  2. 57.101
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0.  5.100 /  0.  5.100
[mxf @ 0x7fb58183d800] unknown operational pattern: 00h 00h - guessing OP1a
[mxf @ 0x7fb58183d800] invalid KAGSize 0 - guessing 1
[mxf @ 0x7fb58183d800] partition 1: bad ThisPartition = 0
[mxf @ 0x7fb58183d800] IndexSID 129 starts at EditUnit 722 - seeking may not work as expected
Input #0, mxf, from 'shallow.mxf':
  Duration: 00:00:37.24, start: 0.000000, bitrate: 243273 kb/s
    Stream #0:0: Video: j2k, 1920x1080, 23.98 fps, 23.98 tbr, 23.98 tbn, 23.98 tbc
At least one output file must be specified

thought it could be useful:

original error message
[mxf @ 0x7f945a03d800] inconsistent FooterPartition value: 1132608274 != 4294967296
1132608274 is very close to the size of file 1132608530

[reimar]

I just sent a patch to ffmpeg-devel, if you can test it and it works we might not need the sample.

[zhukov]

yes the patch helps to get through the error but the j2k inside wont decode (even i force use of libopenjpeg)

./ffmpeg -loglevel debug -codec libopenjpeg -i ~/Dropbox/Public/10bit_yuv_j2k.mxf -f rawvideo /dev/null
ffmpeg version N-36504-g4278023, Copyright (c) 2000-2012 the FFmpeg developers
  built on Jan  5 2012 15:26:05 with llvm_gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)
  configuration: --enable-libopenjpeg
  libavutil      51. 34.100 / 51. 34.100
  libavcodec     53. 54.100 / 53. 54.100
  libavformat    53. 29.100 / 53. 29.100
  libavdevice    53.  4.100 / 53.  4.100
  libavfilter     2. 57.101 /  2. 57.101
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0.  5.100 /  0.  5.100
[mxf @ 0x7fa24983d800] Format mxf probed with size=2048 and score=100
[mxf @ 0x7fa24983d800] broken or empty index
[libopenjpeg @ 0x7fa24983fc00] err{or,}_recognition separate: 1; 1
[libopenjpeg @ 0x7fa24983fc00] err{or,}_recognition combined: 1; 10001
[libopenjpeg @ 0x7fa24983fc00] detected 8 logical cores
[mxf @ 0x7fa24983d800] parser not found for codec j2k, packets or times may be invalid.
[mxf @ 0x7fa24983d800] Probe buffer size limit 5000000 reached
Input #0, mxf, from '/Users/zhukov/Dropbox/Public/10bit_yuv_j2k.mxf':
  Duration: 00:00:37.24, start: 0.000000, bitrate: 2252 kb/s
    Stream #0:0, 4, 1001/24000: Video: j2k, 1920x1080, 1001/24000, 23.98 fps, 23.98 tbr, 23.98 tbn, 23.98 tbc
File '/dev/null' already exists. Overwrite ? [y/N] y
Video pixel format is unknown, stream cannot be encoded

which was not the case like a month before when i submitted a patch for yuv libopenjpegdec support

[reimar]

footer_partition = avio_rb64(pb);

wants to output -1

The value you give (4294967296) is not even close to -1, this is a 64 bit value (otherwise files > 4 GB wouldn't be possible).
In addition, according to the code the value for "not set" should be 0, not -1.
The code probably should take the file size into account when deciding which to use, however it happens to pick the right one anyway.
As for decoding, it looks like somehow pix_fmt doesn't get set at the right point.
The strange thing is that MPlayer works just fine.
It seems to me that somehow frame multithreading is enabled and that breaks try_decode_frame.

[reimar]

Sent another patch to ffmpeg-devel. MPlayer is not affected because the only missing information is pix_fmt and MPlayer requires that only when it actually needs to display something, while ffmpeg needs it before even starting to do anything at all.

[reimar]

I forgot: -threads 1 should be a work-around against that second problem.

[zhukov]

re second patch ([PATCH] Flush decoders correctly in avformat_find_stream_info())
yes it helps, thanks

re -1 and 4294967296
offtopic but anyway :)

#include <stdio.h>

int main(void) {
    unsigned long long a = 4294967296L;
    unsigned long long a2 = 4294967295L;
    int b = (int) a;
    int b2 = (int) a2;
    printf("%d %llu\n", b ,a);
    printf("%d %llu\n", b2 ,a2);
    return 0;
}

./a.out

0 4294967296
-1 4294967295

[cehoyos]

Fixed by Reimar.