[FFmpeg-devel] Tracking down frame corrupting race conditions on Windows.
dalecurtis at chromium.org
Fri Apr 20 22:49:13 CEST 2012
We've been running into an issue where the decoder (avcodec_decode_video2) is
returning corrupted frames on Win32; some images showing the corruption:
The problem is most easily detected when run under thread sanitizer (TSAN)* with
a hash test. I've created a sample program which simply decodes and hashes
each frame (using the av_sha_* interfaces) to illustrate this:
Without TSAN and 2 threads:
> hash-test.exe bear0.mp4 2
With TSAN and 2 threads:
> tsan\tsan.bat --log-file=tsan.txt -- hash-test.exe bear0.mp4 2
With TSAN and 1 threads:
> tsan\tsan.bat --log-file=tsan.txt -- hash-test.exe bear0.mp4 1
Our assumption is that the output from threads == 1 should be indistinguishable
from output with threads > 1. Some general notes:
- The issue occurs with at least h264, vp8, and theora files.
- We have seen no issues under Linux or Mac, which is not to say there are no
problems. The Windows version of TSAN runs much slower than the Linux
or Mac variants and thus may be more susceptible to the issue.
- Switching to pthreads instead of using w32threads on Windows does not fix
- Given the hash changes with the number of threads, we suspect the problem
is a race condition.
- It's possible TSAN is breaking something fundamental, however past
experience with the tool has shown the worst case to be false positives;
never spurious impacts to the running program...
- FATE will fail its hash tests on almost every single test if TSAN is set as
the target_exec on Windows.
To reproduce the results, the fastest way is to grab the test bundle from here:
The bundle includes a precompiled Win32 hash-test executable, hash-test source
code, and TSAN binaries as well as h264, theora, and vp8 test cases. From there
you just need to add the FFmpeg or LibAV DLL files from the prebuilt servers:
Then to run the test you just run:
hash-test.exe <file> <threads>
tsan\tsan.bat --log-file=tsan.txt -- hash-test.exe <file> <threads>
I highly recommend using --log-file with TSAN, otherwise it will generate a
tremendous amount of warnings due to the code's assumption of atomic integers as
well as other more complicated threading patterns. However, it's possible the
real issue is lurking in one of those warnings.
If you want to build the test yourself you'll need to setup MinGW and build
the shared version of FFmpeg/LibAV, the test can then be compiled from the MinGW
gcc -o hash-test.exe hash-test.c -I. -std=c99 avformat-54.dll avutil-51.dll
acolwell and I have spent a lot of time trying to track down the source of this
corruption on Windows without much success. We're hoping the community might
have some better ideas on where to look. Thanks in advance for any assistance!
*Thread Sanitizer: http://code.google.com/p/data-race-test/wiki/ThreadSanitizer
More information about the ffmpeg-devel