SuperREP 3.2: huge-dictionary LZ77 preprocessor

Homepage: http://freearc.org/research/SREP.aspx

Description

SuperREP is the first LZ77 compressor that supports dictionaries larger than RAM available. Default settings (-l512) allow to process files that are 20x larger than RAM size. Memory requirements are proportional to 1/L, so by increasing -l value it's possible to process even larger files. Compression speed is about 100 mb/s and decompression runs at 250 mb/s on i3-2100.

Changelog

SREP 3.2 (April 6, 2013):

  • -m2 -lN now is the same as -m3 -lN -cN: compression ratio is average between -m1 and -m3, while speed is the same as in old versions
  • -a0: the same compresssion ratio as -a1, memory usage is smaller by 5-10%, but 1.5-2x slower
  • -a32/-a64: sometimes faster than -a16 (only with large pages), but needs even more memory
  • -slp[+/-/]: force/disable/try(default) large pages support

    • Minor changes:

  • -v[0..2]: verbosity level
  • -pcMAX_OFFSET: print performance counters for matches closer than MAX_OFFSET
  • -l64k/-c1mb syntax support (k/m/kb/mb suffixes for kilobytes/megabytes)
  • Both 32-bit and 64-bit default executables are compiled with GCC 4.7
  • 32/64-bit dynamic/static linux builds

    • SREP 3.1 (February 23, 2013):

  • -m1f -a4 now is default compression mode, for quick and dirty compression. Use -m3f -a1 for maximum compression
  • 32-bit version became 1.5x faster than in SREP 3.0
  • gcc64 version: srep64g.exe. srep32i/srep64i still are the fastest executables
  • bugfix: it was impossible to allocate more than 4 gb for bitarr[]
  • bugfix: 32-bit version now uses only 1.5 gb for future-lz decompression by default (instead of 4gb-1 that's too much)

    • Changes in the printed stats:

  • cpu time displayed now is the sum of times spent in *ALL* threads
  • cpu/global times and speeds are printed, with speeds measured in MiB/s
  • on future-lz decompression, I/O and VM stats are printed only when they are non-zero
  • -pc option displays performance counters

    • Unlike 32-bit SREP 3.01 that used one 32-bit hash resulting in degraded speed on huge files and in -m3 mode, now 32-bit version uses two 32-bit hashes, making compression both fast and accurate. 64-bit version still employs single 64-bit hash, making it up to 1.5x faster than 32-bit code.


    SREP 3.01 (March 4, 2012):

  • speed up the 32-bit executable by calculating 32-bit hash (instead of 64-bit one); it may hurt speed/compression on very large files