zpaq is a free and open source (GPL v3) incremental, journaling command-line archiver for Windows and Linux. Incremental means that when you back up your hard drive, for example:
zpaq add e:backup.zpaq c:\*then only those files whose last-modified date has changed since the previous backup are added. For 100 GB of files, this typically takes about a minute, vs. an hour to create the first version. Journaling means that the archive is append-only. When you add files or directories to the archive, both the old and new versions are saved. You can recover old versions by specifying the date or version number, for example:
zpaq extract e:backup.zpaq c:\Users\Bob -to tmp -until 20131030will extract all the files and directories in c:\Users\Bob as of the last backup on or before Oct. 30, 2013 and put them in a directory named tmp.
zpaq uses deduplication, multi-threading, and a range of compression algorithms (LZ77, BWT, and context mixing) for high performance. The following comparison is from the 10 GB benchmark (10 GB in 83,437 files and directories) using default settings on a Core i7 M620, 2.66 GHz, 4 GB backup to an external USB hard drive in Ubuntu Linux.
Archiver Size Compress Extract License -------- ------- -------- ------- ------- zip 4844 MB 1206 sec 229 sec Free, open source with specification rar 4493 MB 1423 sec 239 sec Commercial, unrar is free and open source 7zip 3892 MB 6593 sec 472 sec Free, open source freearc 3711 MB 1448 sec 422 sec Free, open source zpaq 3825 MB 379 sec 249 sec Free, open source with specification
zpaq is unique among archivers in that it uses a self-describing compression format to maintain both backward and forward compatibility between versions. When a new version improves the compression algorithm, older versions can still read the archive. The format is precisely described by a specification and reference decoder. The format, algorithm, and code are not protected by any patents or pending patents as far as I know. I have purposely published all past versions (below) to establish prior art so that no patents can be filed.
zpaq supports archive encryption with AES-256 using Scrypt for password strengthening.
libzpaq (included in the download) is a public domain API in C++ that provides streaming compression and decompression services to and from files, strings, or arrays. The API is documented in libzpaq.h. libzpaq also includes encryption code for SHA1, SHA256, AES (encryption only), and Scrypt.
libzpaq supports both built-in and custom compression algorithms. Custom algorithms are described in a language called ZPAQL, which is documented in libzpaq.h and in the specification. The zpaqd development tool allows you to run, test, and debug ZPAQL code. zpaqd and some sample ZPAQL config files and preprocessors, and sample libzpaq applications can be found on the ZPAQ utilities page.
calgarytest2.zpaq is a test case which should decode to the Calgary corpus. It tests all of the ZPAQL instructions and components and various compression modes.
past source code releases
on Github, maintained by Evan Nemerson.
WinZPAQ GUI interface (discussion) for benchmark testing, by Sportman.
Zpaq Explorer GUI for browsing archives (list and extract only) by Surfersat.
Github fork of zpaq v6.20 maintained by Yonggang Luo.
PeaZip by Giorgio Tani, supports ZPAQ streaming format, among many others. Does not support journaling format.
Debian packages, very old version.
PAKKA GUI versioned extractor.
ZPAQ development began on Feb. 15, 2009 with a series of mutually incompatible experimental programs (v0.01 through v0.09, now obsolete). Archives created with these versions could only be read by the same version. The level 1 standard became fixed on Mar. 12, 2009 with version 1.00. All versions from 1.00 onward are level 1 compliant and can extract older archives. With two exceptions, all programs can also read archives produced by newer versions. This forward compatibility was broken twice:
Other major developments:
The SHA-1 code used in these older versions (prior to libzpaq 1.00) is derived from RFC 3174, which is copyright (C) 2001, The Internet Society. Please see this document for the full license.
Open source (C++) and Win32 executables, Feb. 15, 2009.
zpaq v0.02 adds E8E9 transform. Fully supports post-processing. Not compatible with v0.01. Feb. 19, 2009.
zpaq v0.03 modifies MIX, MIX2, IMIX to fix poor compression on large files. Not compatible with v0.02. Feb. 19, 2009.
zpaq v0.04 modifies train() and squash() for improved compression. Not compatible with v0.03. Feb. 21, 2009.
zpaq v0.05 modifies probability representation and mixer weights to prevent mixer overflow and to improve compression for highly redundant data. Not compatible with v0.04. Feb. 26, 2009.
zpaq v0.06 adds SHA1 checksums, replaces IMIX2 with ISSE. Not compatible with v0.05. Feb. 27, 2009.
zpaq v0.07 improves ISSE and bit-history state table. Not compatible with v0.06. Feb. 28, 2009.
zpaq v0.08 adds LZP transform and minor improvements. Not compatible with v0.07. Mar. 8, 2009.
zpaq v0.09 removes counters from ISSE and ICM to improve speed. Not compatible with v0.09. Mar. 9, 2009.
zpaq v1.00 (first level 1 compliant version) includes unzpaq1 candidate reference decoder. Simplified bit history tables. Not compatible with earlier versions. Mar. 12, 2009.
fast.cfg written Apr. 26, 2010. Now part of libzpaq distribution.
unzpaq 1.01 updates reference decoder comments and help message and fixes some VS2005 compiler issues. Compatible with 1.00. Apr. 27, 2009.
unzpaq 1.02 and zpaq 1.02 closes extracted files immediately after decompression instead of when program exits. Fixes g++ 4.4 warnings. Compatible with 1.00 and 1.01. June 14, 2009.
unzpaq 1.03 and zpaq 1.03 has a default compression mode (mid.cfg), supports compressing files in segments to separate blocks and extracting them as suggested in part 7 of the spec. Does not store paths by default. Does not extract to absolute paths by default. Some minor improvements. Sept. 7, 2009 (added zpaq.exe Sept. 8, 2009).
zpaq103b adds zpaqsfx 1.03, a stub for creating self extracting archives. No changes to zpaq or unzpaq. Sept. 14, 2009.
zpaq104 can list and extract from self extracting archives without running them. Added progress meter. zpaqsfx.exe stub is slightly smaller. unzpaq unchanged. Sept. 18, 2009.
zpaq105 removes built in x and p preprocessors and makes them separate programs called from config files with compile time postprocessor testing. Adds if-else-endif and do-while to ZPAQL. Many small changes. Sept. 28, 2009.
zpaq106 adds "ta" to append locater tags to allow ZPAQ streams to be detected when embedded in arbitrary data. zpaq1.pdf revision 1 adds this recommendation. unzpaq106.cpp implements it. Sept. 29, 2009.
zpaqsfx 1.06 self extracting archive stub is now separate from the ZPAQ distribution. Updated Sept. 29, 2009, posted Oct. 26, 2009. (Replaced by zpsfx in libzpaq 2.01)
zpipe v1.00, a simple streaming compressor, Sept. 30, 2009. Linux patch added Jan. 18, 2010.
zpaq107 adds config file parameters and fixes some bugs. From now on the specification and reference decoder are not included unless they change. Oct. 2, 2009.
bwt_j2 is a config file (by Jan Ondrus) and preprocessor for BWT compression. Posted Oct. 7, 2009.
bwt_j3 is a bug fix for bwt_j2 to accept multiple files. Jan Ondrus, Oct 7, 2009.
exe_j1 is a config file and preprocessor for .exe and .dll files. It extends the E8E9 transform in exe.cfg to conditional jumps. Jan Ondrus, Oct. 7, 2009.
unzpaq108.cpp removes undefined behavior of ZPAQL shifts larger than 31 bits on non x86 hardware. Oct. 14, 2009.
zpaq108 generates optimized code that runs about twice as fast on systems with a C++ compiler installed. Oct. 14, 2009.
bmp_j4, configuration for .bmp files by Jan Ondrus, Oct. 14, 2009.
bwt_slowmode1 BWT compression based on BBB slow mode. Jan Ondrus, Oct. 15, 2009.
jpg_test2 JPEG config by Jan Ondrus, Oct. 20, 2009, posted Oct. 26, 2009.
zpaq109 Linux port and some cosmetic bug fixes. Oct. 21, 2009.
zpaq110 bug fix for Linux/g++ 4.4.1, Dec. 28, 2009.
zp v1.00 simple ZPAQ compatible archiver with 3 optimized compression levels, Apr. 26, 2010.
Added a license file to zpaq 1.10, zpipe 1.00, zpaqsfx 1.06, and zp 1.00 distributions on May 23, 2010. No software changes.
libzpaq 0.01, Sept. 27, 2010.
libzpaq 0.02, Sept. 28, 2010.
zpipe 2.00 updated to use libzpaq 0.02, Sept. 28, 2010.
libzpaq 1.00, Sept. 29, 2010. Package includes libzpaq, ZPAQ specification, reference decoder, zp, zpipe, and fast, mid, max config files.
libzpaq 1.01, Oct. 14, 2010. Updates libzpaq interface to use inheritance instead of templates, requiring changes to zp and zpipe. Now compiles faster.
libzpaq 1.02, Oct. 20, 2010. Adds zpsfx self extracting archive stub. Separates optimized models from libzpaq.cpp to libzpaqo.cpp.
libzpaq 2.00, Oct. 30, 2010. Ports zpaq to libzpaq, replacing zp.
libzpaq 2.01, Nov. 5, 2010. Added optimized self extracting archives. Simplified installation.
libzpaq 2.02, Nov. 13, 2010. zpaq shows compression component statistics. Libzpaq support added.
zpaq 2.03, Dec. 23, 2010, adds Linux support. The remaining code is split into libzpaq 2.02, zpipe 2.01, zpsfx 1.00, and configuration files min, fast, mid, and max.
zpaq 2.04, Dec.29, 2010, adds support for Visual C++, Borland, and Mars compilers in addition to g++. A Windows install script is added.
zpaq 2.05, Jan. 5, 2011. Fixed a bug in which zpaq crashed when decompressing an unnamed file (as created with zpipe or zpaq nc) without renaming. Separated zpaq.1.pod. (Updated corrupted install.sh on Jan. 13, 2011).
libzpaq 2.02a, Jan. 6, 2011. Updates the documentation.
pzpaq 0.01 parallel file compressor, Jan. 21, 2011.
pzpaq 0.02, Jan. 26, 2011, adds large file support (over 2 GB) to Windows.
pzpaq 0.03, Feb. 2, 2011, optimizes decompression for nonstandard compression levels by recompiling itself with g++ (like "zpaq ox").
pzpaq 0.04, Feb. 4, 2011, Windows version uses native threads and no longer requires pthreads-win32.
pzpaq 0.05, Feb. 10, 2011 removes -s option, puts temporary files in $TMPDIR or %TEMP%.
bwt v1, Mar. 16, 2011. 4 BWT based configurations.
unzp 1.00, May 10, 2011, a block level parallel decompresser optimized for fast, mid, max, bwtrle1, bwt2 models with source level JIT for other models.
zp 1.01, May 12, 2011, a block level parallel compressor with 4 levels (bwtrle1, bwt2, mid, max). With unzp replaces pzpaq.
zp 1.02, May 16, 2011. Fixed -t option.
May 18, 2011. Undated zp.102.zip and unzp.100.zip with static x86-64 Linux binaries.
zp 1.03, May 26, 2011. Merges the compressor and decompresser unzp into one program.
wbpe 1.00, June 12, 2011. Dictionary preprocessor for text files.
wbpe 1.10, June 21, 2011.
zpaq 3.00, July 16, 2011. Combines features of zpaq v2.05 and zp v1.03. zp support is discontinued. Windows only.
zpaq 3.01, July 21, 2011. Adds 64 bit Linux support. Includes libzpaq 3.00.
bmp_j4a, July 21, 2011. Updated bmp_j4 .bmp configuration for zpaq v3.01.
libzpaq 3.00, July 28, 2011, from zpaq v3.01 but as a separate download.
libzpaq 4.00, Nov. 13, 2011. libzpaq.cpp, libzpaq.h, libzpaq.3.pod. Replaces source-level JIT with internal JIT for x86-32 and x86-64.
zpaq 4.00, Nov. 13, 2011. zpaq.cpp, zpaq.1.pod for use with libzpaq 4.00. Removes source generation, b and e commands and -j option.
calgarytest.zpaq, Nov. 13, 2011. Test case for ZPAQ compliance.
zpipe v2.01, Nov. 13, 2011. zpipe.exe linked to libzpaq v4.00. Source unchanged.
zpaq v4.01, Nov. 26, 2011. Source code adds incremental update and extraction.
zpaq v4.02, Nov. 28, 2011. Source code adds commands c, x output/, list hcomp/pcomp. Updated pi.cfg for this version.
libzpaq v4.01, Dec. 20, 2011. Fix for Mac OS (MAP_ANONYMOUS -> MAP_ANON).
zpaq v4.03, Dec. 21, 2011. Adds -n, -r, and -f options. Fixed bug in u (did not save filenames with no args).
lz1.zip, Dec. 29, 2011. LZ77 model.
ZPAQ level 2 standard, unzpaq200.cpp reference decoder, libzpaq 5.00 support, and calgarytest2.zpaq test case, Feb. 1, 2012. Level 2 allows the COMP section to be empty to store uncompressed (but possibly preprocessed) data to support faster compression models.
libzpaq 5.01, Feb. 2, 2012. Removed debugging code from libzpaq.cpp.
tiny_unzpaq.cpp v1.0, Mar. 21, 2012.
zpaq v4.04, Mar. 26, 2012. Fixed bug in r command that truncated output file.
zpsfx v1.01, Apr. 4, 2012. Self extractor modified by Klaus Post to create directories as needed.
zpaq v5.00, Aug. 27, 2012. Candidate release, primarily a small development tool. Updates libzpaq to v6.00a to include ZPAQL compiler.
zpaq v6.00, Sept.26, 2012. Candidate release. Adds journaling, incremental update, and deduplication to support large backups. Includes 4 compression levels (fast and slow LZ77, BWT, mid) plus all v5.00 features.
zpaq201.pdf, Sept. 28, 2012. Updated specification to describe streaming and journaling archive formats.
zpaq v6.01, Sept. 28, 2012. Adds -method 0, -list -force, improves -list -detailed, and bug fixes.
zpaq v6.02, Sept. 29, 2012. Speed and compression improvements. Adds -quiet option.
zpaq v6.03, Sept. 30, 2012. Saves and restores file attributes. Cleans up -list.
zpaq v6.04, Oct. 1, 2012. Compression and speed improvements by sorting by filename extension and storing uncompressible data. Adds -list -quiet.
bmp_j4b.zip, Oct. 1, 2012. Updated bmp_j4 .bmp model to work with zpaq v6.xx
zpaq v6.05, Oct. 2, 2012. Adds -list -history -summary, Linux port, bug fixes, and improved docs.
zpaq v6.06, Oct. 4, 2012. Simplifies -list and adds -compare.
zpaq v6.07, Oct. 7, 2012. Fixes porting issues with Mac OS/X and Visual C++.
lazy v1.00, Oct. 10, 2012. A fast LZ77 compressor/preprocessor and and config.
lazy2 v1.00, Oct. 31, 2012. lazy with an E8E9 filter (and 1 GB file size limit).
zpaq v6.16, Nov. 5, 2012. Better compression using lazy (-method 1) + e8e9 (all methods). Adds -test and -post.
zpaq v6.17, Dec. 13, 2012. Fixed display of international characters. libzpaq v6.17 has slightly faster SHA1. Has bugs. Do not use.
zpaq v6.18, Dec. 14, 2012. Bug fix.
zpaq v6.19, Jan. 23, 2013. Splits into zpaq (journaling archiver) and zpaqd (development tool). Adds methods 5-9. libzpaq v6.19 adds single pass compression checksums.
bmp_j4c, Jan. 24, 2013. Updated .bmp config file to work with new zpaq/zpaqd syntax.
zpaq v6.20, Feb. 1, 2013. Improved compression for methods 5 through 9. zpaq64.exe added Feb. 4, 2013.
zpaq v6.21, Feb. 6, 2013. Extract directories restores timestamps and attributes. Adds -until date. Lists alphabetically. Fixed docs. zpaq621-64.exe added Feb. 8, 2013.
zpaq v6.22, Feb. 13, 2013. -method supports custom algorithms. zpaqd and libzpaq fixes for Win64. Command line accepts international characters.
zpaq v6.23, Mar. 25, 2013. -method supports config files without preprocessors. zpaqd 6.23 speed improvements for g++ 4.7.0 and "ds" command. libzpaq 6.23 faster initialization.
zpaq v6.24, Apr. 12, 2013. Adds d (delete) command. Works with wildcards. zpaqd adds built-in configs 1..3.
zpaq v6.24a, Apr. 14, 2013. Recompile zpaq.exe, zpaqd.exe to get around compiler bug in 64 bit version of MinGW causing 32 bit zpaq to crash in WinXP.
zpaq v6.25, Apr. 19, 2013. libzpaq optimizations (3-5% faster) and bug fix for WinXP. No changes to zpaq or zpaqd except version number.
zpaq v6.26, May 9, 2013. Optimizations: zpaq improves grouping of incompressible files into blocks, faster StringBuffer. libzpaq JIT optimizes consecutive ZPAQL increments. zpaqd fixes compiler warning.
zpaq v6.27, May 13, 2013. Adds -all and -test options. Improved recovery of damaged archives. zpaqd updated to verify checksums when listing journaling archives.
zpaq v6.28, May 22, 2013. Changed zpaq -test to a command. Improved handling of damaged archives.
zpaq202.pdf, June 3, 2013. Level 2 revision 2 of spec adds a fragmentation recommendation for deduplication compatibility.
zpaq v6.29, June 6, 2013. Improved compression. Extended method 1 and 2 LZ77 parameters. Test command implements new 2.02 spec.
zpaq v6.30, June 7, 2013. Fixes bug in extracting read-only files. Adds -attr option.
zpaq v6.31, June 7, 2013. Changed -attr default to select all files.
zpaqd v6.27, June 11, 2013. From zpaq 6.27 but now a separate distribution.
zpaqd v6.32, June 19, 2013. Faster I/O when linked with libzpaq v6.32 (included).
zpaqd v6.33, June 20, 2013. libzpaq 6.33 bug fix and recompile to fix list command. No change to zpaqd 6.32 source.
zpaq v6.33, June 21, 2013. Improved compression, supports block sizes, streaming mode, -fragile option and compress to empty archive. Removed -attributes, -above. -method is 0..7
zpaq v6.34, June 25, 2013. Supports long LZ77 offsets. -method is 0..6. Default block size increased to 64 MB for 2..6.
zpaq v6.35, June 28, 2013. LZ77 look-ahead and other improvements. Better handling of nonexistent input files.
zpaq v6.36, July 5, 2013. LZ77 compression improvements. Memory options for special methods.
zpaq v6.37, July 15, 2013. Adds purge command.
zpaq v6.38, July 18, 2013. Fixes extraction bug in v6.28-6.37. Adds compare command.
zpaq v6.39, July 25, 2013. List command shows compression ratios. Fixes -method 0 compression in DEBUG mode.
zpaq v6.40, July 28, 2013. Adds -noattributes option. Windows version does not add reparse points.
zpaq v6.41, Aug. 2, 2013. Adds restore command. Fixed wildcard handling and extract -fragile.
zpaq v6.42, Sept. 26, 2013. Adds list -duplicates, faster updates, minor bug fixes.
zpaq v6.43, Dec. 20, 2013. Adds -key (encryption), show, sha1, sha256 commands. Updates libzpaq to v6.43 (adds AES, SHA256, Scrypt)
zpaq v6.44, Jan. 9, 2014. Adds encrypt command, removes restore, show, sha1, sha256, changes extract to skip existing files instead of error, changes purge syntax, prompt for passwords without echo, faster -method 5, some minor bug fixes.
zpaq v6.45, Jan. 12, 2014. Improves compression by sorting files by case insensitive extension, then by decreasing size rounded to 16K. Fixed VC++ compile error in 6.44.
zpaq203.pdf, Jan. 16, 2014. Level 2 revision 3 of the specification adds encryption.
zpaq v6.46, Jan. 17, 2014. Improved compare, added -fragment option. Fixed extracting streaming encrypted archives.
zpaq v6.47, Jan. 21, 2014. Adds snip command to support remote backups. Extends -since to extract and compare. Increased compression buffers for better core utilization.
zpaq v6.48, Jan. 23, 2014. Adds join command. Renames snip to split. Optimized decoder in libzpaq.cpp 6.48.
zpaq v6.49, Jan. 31, 2014. Adds progress indicator and other UI improvements. test can take filename args. libzpaq.cpp 6.49 and Makefile to fix Mac OS/X compiler warnings.
Discussion about ZPAQ updates.
ZPAQ is intended to replace PAQ and its variants (PAQ8, PAQ9A, LPAQ, LPQ1, etc) with similar or better compression in a portable, standard format. Current versions of PAQ break archive compatibility with each compression improvement. ZPAQ is intended to fix that. I no longer maintain the older PAQ code.
ZPAQ was written by Matt Mahoney, mattmahoneyfl (at) gmail (dot) com