[FFmpeg-devel] [PATCH v2 0/4] libopenmpt 0.3 updates

Jörn Heusipp osmanx at problemloesungsmaschine.de
Wed Jan 10 13:18:00 EET 2018




Compared to the previous patch versions, only part 4 has been changed
(based on the comments received so far).



I also ran more performance tests. FFmpeg is configured via
'./configure --enable-libopenmpt --enable-libmodplug' in all tests.



probetest failures with current libopenmpt 0.3.5-pre.0 (r9426):

testing size=1
testing size=2
testing size=4
testing size=8
Failure of vqf probing code with score=50 type=3 p=C2A size=8
Failure of mov,mp4,m4a,3gp,3g2,mj2 probing code with score=50 type=3 p=FD1 size=8
Failure of 3dostr probing code with score=66 type=3 p=E2 size=8
Failure of mov,mp4,m4a,3gp,3g2,mj2 probing code with score=100 type=3 p=75 size=8
testing size=16
Failure of musx probing code with score=40 type=3 p=A6A size=16
Failure of alias_pix probing code with score=51 type=0 p=CBA size=16
testing size=32
testing size=64
Failure of mpc8 probing code with score=49 type=3 p=85A size=64
testing size=128
testing size=256
testing size=512
testing size=1024
testing size=2048
Failure of libopenmpt probing code with score=76 type=0 p=FED size=2048
testing size=4096
testing size=8192
testing size=16384
testing size=32768
testing size=65536



AMD A4-5000, Ubuntu 14.04, x86-64, GCC 4.8.2, libopenmpt 0.3.5-pre.0 (r9426):

probetest performance:
  1425530947 cycles,           aa
 25976883912 cycles,          aac
 25507211169 cycles,          ac3
  1042217662 cycles,          acm
  1632550096 cycles,          act
  1847361076 cycles,          adp
  3215025217 cycles,          ads
  1751119560 cycles,          adx
  1422549077 cycles,          aea
  1661030689 cycles,         aiff
  1664388004 cycles,          aix
  2876754979 cycles,          amr
 16213648221 cycles,        amrnb
 10357400286 cycles,        amrwb
  1518874168 cycles,          anm
  1848774401 cycles,          apc
  1025286732 cycles,          ape
  1999052278 cycles,         apng
 18252897495 cycles,      aqtitle
  2771276135 cycles,          asf
  2138272095 cycles,        asf_o
 16758768236 cycles,          ass
  1810499177 cycles,          ast
  1597035448 cycles,           au
  1629356130 cycles,          avi
  1664911494 cycles,          avr
  1421498186 cycles,          avs
  1504496122 cycles,  bethsoftvid
  1494735429 cycles,          bfi
  2543270309 cycles,         bink
  2111172403 cycles,          bit
  1500579804 cycles,        bfstm
  1087554907 cycles,        brstm
  1587938487 cycles,          boa
  1715001029 cycles,          c93
  1063478621 cycles,          caf
  7724800501 cycles,    cavsvideo
  1825737620 cycles,         cdxl
  1181800422 cycles,         cine
  3250419221 cycles,       concat
  2222078869 cycles,        dcstr
  1512175182 cycles,          dfa
  1403369419 cycles,        dirac
  1555052671 cycles,        dnxhd
  2044301070 cycles,          dsf
  1164145857 cycles,       dsicin
  1617975398 cycles,          dss
 31075115518 cycles,          dts
  1512886188 cycles,        dtshd
 15304698997 cycles,           dv
 12154318089 cycles,       dvbsub
  1641827312 cycles,       dvbtxt
  1799178653 cycles,          dxa
  2122493980 cycles,           ea
  1412999215 cycles,     ea_cdata
 19832626428 cycles,         eac3
  1469702273 cycles,         epaf
  3151846641 cycles,   ffmetadata
  3846803597 cycles,         fits
  2382096387 cycles,         flac
  1938351100 cycles,         flic
  1750595032 cycles,          flv
  1034144158 cycles,     live_flv
  1419146038 cycles,          4xm
  2298304236 cycles,          frm
  2332352309 cycles,          fsb
  1753160137 cycles,          gdv
  1164270006 cycles,         genh
  4115033986 cycles,          gif
  2162805981 cycles,          gsm
  2425065785 cycles,          gxf
 22958299155 cycles,         h261
 22259377736 cycles,         h263
 21565315903 cycles,         h264
 14131416451 cycles,         hevc
  2294216465 cycles, hls,applehttp
  2026125838 cycles,          hnm
  1599698430 cycles,          ico
  2130406935 cycles,        idcin
  1936330153 cycles,          idf
  1721541470 cycles,          iff
  2700087066 cycles,         ilbc
  1603579794 cycles,    alias_pix
  2559447550 cycles,  brender_pix
  1736901491 cycles,    ingenient
 13309369249 cycles,      ipmovie
  2020584923 cycles,        ircam
  2157010519 cycles,          iss
  1326703719 cycles,          iv8
  1373300900 cycles,          ivf
  3415745402 cycles,          ivr
 28777360214 cycles,      jacosub
  1683261092 cycles,           jv
  1796897802 cycles,        lmlm4
 27908164305 cycles,         loas
  2783000204 cycles,          lrc
  1698706136 cycles,          lvf
  2283438621 cycles,          lxf
 21187997513 cycles,          m4v
  1761691177 cycles, matroska,webm
  1810514534 cycles,        mgsts
 13885029990 cycles,     microdvd
 17194619859 cycles,        mjpeg
 14864883485 cycles,   mjpeg_2000
 20581718455 cycles,          mlp
  1465609046 cycles,          mlv
  2130473395 cycles,           mm
  1381846427 cycles,          mmf
  2189640745 cycles, mov,mp4,m4a,3gp,3g2,mj2
 33954117497 cycles,          mp3
  1705624337 cycles,          mpc
  1372935442 cycles,         mpc8
 21843519866 cycles,         mpeg
 25019569510 cycles,       mpegts
 10100750292 cycles,    mpegvideo
  1807433600 cycles,       mpjpeg
 10712139775 cycles,         mpl2
 11785261566 cycles,        mpsub
  1905445870 cycles,          msf
 10814845066 cycles,     msnwctcp
  1531102489 cycles,         mtaf
  1306615774 cycles,          mtv
  2222380980 cycles,         musx
  1716209923 cycles,           mv
  3881317597 cycles,          mxf
  1545720786 cycles,           nc
  2056314607 cycles,   nistsphere
  1654383657 cycles,          nsp
 18094337108 cycles,          nsv
 14525862175 cycles,          nut
  3159902207 cycles,          nuv
  2727640331 cycles,          ogg
  3262277011 cycles,          oma
  2311236695 cycles,          paf
 10558093771 cycles,          pjs
  1694997950 cycles,          pmp
  1833059413 cycles,          pva
  2055387815 cycles,          pvf
  1136126486 cycles,          qcp
  1631946005 cycles,          r3d
 12636465075 cycles,     realtext
  1678744413 cycles,     redspark
  1689616653 cycles,          rl2
  1684971239 cycles,           rm
  1702641083 cycles,          roq
  2859211090 cycles,          rpl
  2477043341 cycles,          rsd
 29928172868 cycles,        s337m
  8043319530 cycles,         sami
 23903157582 cycles,          sbg
 13817316201 cycles,          scc
 11367926253 cycles,          sdp
  1759022011 cycles,         sdr2
  1323694732 cycles,          sds
  1611367384 cycles,          sdx
  1502062085 cycles,     film_cpk
  2022693698 cycles,          shn
  1879280597 cycles,         siff
  1573119725 cycles,          smk
  2022446701 cycles,       smjpeg
  1746195740 cycles,        smush
  1324175723 cycles,          sol
  1723257601 cycles,          sox
 14435589893 cycles,        spdif
 19961951986 cycles,          srt
  2279919281 cycles,       psxstr
 14460561033 cycles,          stl
  7757448652 cycles,   subviewer1
  9881927027 cycles,    subviewer
  1813503023 cycles,          sup
  2614487132 cycles,         svag
  1788813210 cycles,          swf
  2434490441 cycles,          tak
  5163701269 cycles,  tedcaptions
  1838368024 cycles,          thp
  4031549835 cycles,       3dostr
  1737125038 cycles,   tiertexseq
  1057468914 cycles,          tmv
 21334471493 cycles,       truehd
  1020803156 cycles,          tta
  1749787031 cycles,          txd
  1798977052 cycles,           ty
  2446972639 cycles,          vag
 16496826462 cycles,          vc1
  1457488878 cycles,      vc1test
  2176328303 cycles,         vivo
  1750344002 cycles,          vmd
  2361316510 cycles,       vobsub
  3048150634 cycles,          voc
  1212442370 cycles,          vpk
 15676889881 cycles,      vplayer
  1733430074 cycles,          vqf
  2109001718 cycles,          w64
  1910784666 cycles,          wav
  1793196314 cycles,     wc3movie
  2716316840 cycles,       webvtt
  1930245918 cycles,        wsaud
  1419655493 cycles,          wsd
  1063672420 cycles,        wsvqa
  2199993223 cycles,          wtv
  2963839522 cycles,          wve
  1741739753 cycles,           wv
  1704590788 cycles,           xa
  1789277847 cycles,         xbin
  2039493809 cycles,          xmv
  2570060068 cycles,         xvag
  2268862337 cycles,         xwma
  1351127969 cycles,          yop
  1828935584 cycles, yuv4mpegpipe
  1908523596 cycles,     bmp_pipe
  1658940649 cycles,     dds_pipe
  1533982431 cycles,     dpx_pipe
  1406840553 cycles,     exr_pipe
  1438926647 cycles,     j2k_pipe
  1251023770 cycles,    jpeg_pipe
  1254565974 cycles,  jpegls_pipe
  1710131896 cycles,     pam_pipe
  1646528709 cycles,     pbm_pipe
  1216178831 cycles,     pcx_pipe
  1795627990 cycles,  pgmyuv_pipe
  1555283784 cycles,     pgm_pipe
  1730118897 cycles,  pictor_pipe
  1253719192 cycles,     png_pipe
  1363037586 cycles,     ppm_pipe
  1212431338 cycles,     psd_pipe
  1738071787 cycles,   qdraw_pipe
  1702267451 cycles,     sgi_pipe
  3216491315 cycles,     svg_pipe
  1291987788 cycles, sunrast_pipe
  1268831892 cycles,    tiff_pipe
  1142563696 cycles,    webp_pipe
  1132441750 cycles,     xpm_pipe
  2721827897 cycles,   libmodplug
116347927174 cycles,   libopenmpt

  2837754321           libopenmpt (per module file format)

Overall increase in CPU cycles is about 10% on this system.



AMD FX-8320E, Ubuntu 16.04, x86-64, GCC 5.4.0, libopenmpt 0.3.5-pre.0 (r9426):

probetest performance:
  1431838272 cycles,           aa
 19596017615 cycles,          aac
 22072437919 cycles,          ac3
  1586366499 cycles,          acm
  1633298889 cycles,          act
  2154140183 cycles,          adp
  2378007979 cycles,          ads
  1823790336 cycles,          adx
  1658075845 cycles,          aea
  1727693550 cycles,         aiff
  1749865867 cycles,          aix
  2406511571 cycles,          amr
 16280118538 cycles,        amrnb
  9729762097 cycles,        amrwb
  1679103191 cycles,          anm
  2135510099 cycles,          apc
  1488362086 cycles,          ape
  2080499431 cycles,         apng
  9520540950 cycles,      aqtitle
  2266709135 cycles,          asf
  1868832098 cycles,        asf_o
 12886457939 cycles,          ass
  1485348618 cycles,          ast
  1530423123 cycles,           au
  2076317613 cycles,          avi
  1697698321 cycles,          avr
  1667579531 cycles,          avs
  1724589202 cycles,  bethsoftvid
  1544795352 cycles,          bfi
  2165440010 cycles,         bink
  1949332852 cycles,          bit
  1726887437 cycles,        bfstm
  1817601901 cycles,        brstm
  1931972368 cycles,          boa
  1942186683 cycles,          c93
  1521603271 cycles,          caf
  6700520065 cycles,    cavsvideo
  1776722752 cycles,         cdxl
  1553705877 cycles,         cine
  2273386858 cycles,       concat
  1562976669 cycles,        dcstr
  1717046635 cycles,          dfa
  1664079909 cycles,        dirac
  1669153535 cycles,        dnxhd
  2246099412 cycles,          dsf
  1453517393 cycles,       dsicin
  1500647387 cycles,          dss
 30155546433 cycles,          dts
  1535993877 cycles,        dtshd
 15800426975 cycles,           dv
 11610679333 cycles,       dvbsub
  1841619921 cycles,       dvbtxt
  1830448110 cycles,          dxa
  2092010472 cycles,           ea
  1636380532 cycles,     ea_cdata
 20370397390 cycles,         eac3
  1986625579 cycles,         epaf
  2345789673 cycles,   ffmetadata
  2208460360 cycles,         fits
  2666576040 cycles,         flac
  1853359746 cycles,         flic
  1655594426 cycles,          flv
  1749273757 cycles,     live_flv
  1751364550 cycles,          4xm
  1878271595 cycles,          frm
  2195821183 cycles,          fsb
  2150865173 cycles,          gdv
  1814313220 cycles,         genh
  2731087542 cycles,          gif
  2425559509 cycles,          gsm
  1946161984 cycles,          gxf
 22485157183 cycles,         h261
 18090217832 cycles,         h263
 18123381087 cycles,         h264
 11743837228 cycles,         hevc
  2943741402 cycles, hls,applehttp
  1596751795 cycles,          hnm
  1518915719 cycles,          ico
  2035049448 cycles,        idcin
  1903935166 cycles,          idf
  1759210194 cycles,          iff
  2303812339 cycles,         ilbc
  2256463346 cycles,    alias_pix
  2162931472 cycles,  brender_pix
  1993497708 cycles,    ingenient
 11100984389 cycles,      ipmovie
  1721013983 cycles,        ircam
  2414832178 cycles,          iss
  1563366273 cycles,          iv8
  1720764903 cycles,          ivf
  2705480276 cycles,          ivr
 18444193346 cycles,      jacosub
  1623518888 cycles,           jv
  1546834075 cycles,        lmlm4
 19689388118 cycles,         loas
  2578655283 cycles,          lrc
  1508104094 cycles,          lvf
  2049106502 cycles,          lxf
 14261507075 cycles,          m4v
  2053157819 cycles, matroska,webm
  2104137496 cycles,        mgsts
 16036832726 cycles,     microdvd
 17859866440 cycles,        mjpeg
 11882965398 cycles,   mjpeg_2000
 15763064782 cycles,          mlp
  1819135239 cycles,          mlv
  1704707509 cycles,           mm
  1545905690 cycles,          mmf
  2850154432 cycles, mov,mp4,m4a,3gp,3g2,mj2
 31022110090 cycles,          mp3
  1817347412 cycles,          mpc
  1531970665 cycles,         mpc8
 17122469946 cycles,         mpeg
 22261016387 cycles,       mpegts
  8951992309 cycles,    mpegvideo
  1809929700 cycles,       mpjpeg
 11951089260 cycles,         mpl2
  6551894512 cycles,        mpsub
  1899715798 cycles,          msf
 10830367049 cycles,     msnwctcp
  2044908745 cycles,         mtaf
  1656039910 cycles,          mtv
  1959428195 cycles,         musx
  1986706463 cycles,           mv
  3335345137 cycles,          mxf
  1437248826 cycles,           nc
  1560607671 cycles,   nistsphere
  1743990594 cycles,          nsp
 13637693392 cycles,          nsv
 10394196213 cycles,          nut
  2918921805 cycles,          nuv
  2270768860 cycles,          ogg
  2340087729 cycles,          oma
  2035175754 cycles,          paf
 10801631816 cycles,          pjs
  1600390257 cycles,          pmp
  1624172083 cycles,          pva
  2029585927 cycles,          pvf
  1499654920 cycles,          qcp
  1523644022 cycles,          r3d
  9729201793 cycles,     realtext
  2741283597 cycles,     redspark
  1560979664 cycles,          rl2
  1707424337 cycles,           rm
  1503702579 cycles,          roq
  2502554297 cycles,          rpl
  2324528772 cycles,          rsd
 24221857370 cycles,        s337m
  8451297386 cycles,         sami
 14470356448 cycles,          sbg
 12383160138 cycles,          scc
 10066289314 cycles,          sdp
  1492822405 cycles,         sdr2
  1953547326 cycles,          sds
  1785608501 cycles,          sdx
  1642318183 cycles,     film_cpk
  1674580160 cycles,          shn
  1832283485 cycles,         siff
  2158705275 cycles,          smk
  2242528030 cycles,       smjpeg
  1866202828 cycles,        smush
  1719907973 cycles,          sol
  1603385883 cycles,          sox
 10811633406 cycles,        spdif
 15052457444 cycles,          srt
  1985909974 cycles,       psxstr
 10861785105 cycles,          stl
  6049231980 cycles,   subviewer1
  9438386352 cycles,    subviewer
  1990328115 cycles,          sup
  2353635176 cycles,         svag
  1803115002 cycles,          swf
  2502101683 cycles,          tak
  3174848173 cycles,  tedcaptions
  2109731137 cycles,          thp
  3157368385 cycles,       3dostr
  1534576882 cycles,   tiertexseq
  1498633646 cycles,          tmv
 16064097534 cycles,       truehd
  2038659417 cycles,          tta
  1525132720 cycles,          txd
  2086281061 cycles,           ty
  2335732447 cycles,          vag
 17816993426 cycles,          vc1
  1713663150 cycles,      vc1test
  1697335145 cycles,         vivo
  1594652123 cycles,          vmd
  2881705742 cycles,       vobsub
  2448696469 cycles,          voc
  1488891656 cycles,          vpk
 15458122870 cycles,      vplayer
  1875426898 cycles,          vqf
  1736408027 cycles,          w64
  1556581121 cycles,          wav
  1559664542 cycles,     wc3movie
  2718309078 cycles,       webvtt
  2054066995 cycles,        wsaud
  1596128715 cycles,          wsd
  1560575382 cycles,        wsvqa
  2489983302 cycles,          wtv
  2666389992 cycles,          wve
  1801011668 cycles,           wv
  1692224323 cycles,           xa
  2017237422 cycles,         xbin
  1864517924 cycles,          xmv
  2387538962 cycles,         xvag
  1935027570 cycles,         xwma
  1501248394 cycles,          yop
  2389021238 cycles, yuv4mpegpipe
  1473129814 cycles,     bmp_pipe
  1815162516 cycles,     dds_pipe
  1730860644 cycles,     dpx_pipe
  1557607480 cycles,     exr_pipe
  1597011227 cycles,     j2k_pipe
  1550024144 cycles,    jpeg_pipe
  1466714468 cycles,  jpegls_pipe
  1696590915 cycles,     pam_pipe
  1614645142 cycles,     pbm_pipe
  1507544252 cycles,     pcx_pipe
  1770918401 cycles,  pgmyuv_pipe
  1576701750 cycles,     pgm_pipe
  1651009366 cycles,  pictor_pipe
  1498297364 cycles,     png_pipe
  1604007053 cycles,     ppm_pipe
  1456912936 cycles,     psd_pipe
  1528209484 cycles,   qdraw_pipe
  1597650224 cycles,     sgi_pipe
  1975846380 cycles,     svg_pipe
  1627941519 cycles, sunrast_pipe
  1716338310 cycles,    tiff_pipe
  1529806064 cycles,    webp_pipe
  1398728534 cycles,     xpm_pipe
  2717378296 cycles,   libmodplug
 60610962842 cycles,   libopenmpt

  1478316166           libopenmpt (per module file format)

Overall increase in CPU cycles is about 6% on this system.


Intel Core i7-6700, Debian 9, GCC 6.3.0, libopenmpt 0.3.5-pre.0 (r9426):
Also around 5% increase.


The ~10% vs. ~5% are probably due to the ancient GCC version on that
system, however I have no way to verify that right now (I have neither
2 different CPUs with same GCC version available nor different GCCs on
the same CPU). Also note that GCC 4.8 is the oldest GCC that can even
build libopenmpt 0.3 at all.



The following results are all hot page cache, best of 3 runs, on the
idle AMD FX-8320E system.

probeperf.c:

    #include <inttypes.h>
    #include <stdio.h>
    #include <stdint.h>
    #include <sys/time.h>
    #include "libavformat/avformat.h"
    #define BUFSIZEMAX 65536
    static char buf[BUFSIZEMAX + AVPROBE_PADDING_SIZE];
    static uint64_t now(void) {
        struct timeval tv;
        gettimeofday(&tv, NULL);
        return (uint64_t)tv.tv_sec * 1000000 + tv.tv_usec;
    }
    int main(int argc, const char * * argv) {
        uint64_t iotime = 0;
        uint64_t probetime = 0;
        av_log_set_level(AV_LOG_WARNING);
        av_register_all();
        for (int bufsize = 1; bufsize <= BUFSIZEMAX; bufsize *= 2)
        {
            fprintf(stdout, "bufsize %d\n", bufsize);
            for (int i = 1; i < argc; ++i) {
                AVProbeData pd;
                FILE * f;
                size_t size;
                memset(&pd, 0, sizeof(AVProbeData));
                pd.filename = argv[i];
                memset(buf, 0, bufsize + AVPROBE_PADDING_SIZE);
                iotime -= now();
                f = fopen(pd.filename, "rb");
                size = fread(buf, 1, bufsize, f);
                fclose(f);
                iotime += now();
                pd.buf_size = size;
                probetime -= now();
                av_probe_input_format(&pd, 1);
                probetime += now();
            }
        }
        fprintf(stdout, "io    [us]: %"PRIu64"\n", iotime);
        fprintf(stdout, "probe [us]: %"PRIu64"\n", probetime);
        return 0;
    }



result with old file-extension based probing for libopenmpt:

$ find ../samples/ -type f | xargs --no-run-if-empty ./probeperf 
bufsize 1
bufsize 2
bufsize 4
bufsize 8
bufsize 16
bufsize 32
bufsize 64
bufsize 128
bufsize 256
bufsize 512
bufsize 1024
bufsize 2048
bufsize 4096
bufsize 8192
bufsize 16384
bufsize 32768
bufsize 65536
io    [us]: 260836
probe [us]: 10846094


results with new libopenmpt-based file probing:

$ find ../samples/ -type f | xargs --no-run-if-empty ./probeperf 
bufsize 1
bufsize 2
bufsize 4
bufsize 8
bufsize 16
bufsize 32
bufsize 64
bufsize 128
bufsize 256
bufsize 512
bufsize 1024
bufsize 2048
bufsize 4096
bufsize 8192
bufsize 16384
bufsize 32768
bufsize 65536
io    [us]: 264617
probe [us]: 11043671

Compared to the random data that tools/probetest generates, these more
real-world examples of going through av_probe_input_format() on all
files in FATE show an even smaller relative increase:
11043671 / 10846094 - 1 = 0.018 ~= 2%. Note that in actual use, the to
be probed data will likely not be in page cache.



Jörn Heusipp (4):
  avformat/libopenmpt: Fix mixed code and declarations
  avformat/libopenmpt: Update to libopenmpt 0.3 API
  avformat/libopenmpt: Update file extensions list for libopenmpt 0.3
  avformat/libopenmpt: Probe file format from file data if possible

 libavformat/libopenmpt.c | 99 +++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 93 insertions(+), 6 deletions(-)

-- 
1.9.1



More information about the ffmpeg-devel mailing list