dwarfs-0.7.0-RC6

Features

• Support delayed loading of WinFsp DLL for universal binary. This makes the mkdwarfs, dwarfsck and dwarfsextract tools of the universal binary usable without the WinFsp DLL.

Performance

• Optimized the offset cache to improve random read latency as well as sequential read latency. This gave a 100x higher throughput for a case where DwarFS was used to compress raw file system images. Fixes github #142.

Bugfixes

• Fixed building with make instead of ninja. Also fix builing in Debug mode. Fixes github #146.
• Fixed ninja clean.
• Fixed symlink creation for mount.dwarfs/mount.dwarfs2.

Other

• Added CI pipeline.
• Don't write versioning files to source tree.

dwarfs-0.7.0-RC5

Features

• Windows support. All tools can now be built and run on Windows, including the FUSE driver, which makes use of WinFsp. Also fixes github #85.
• Build a "universal" binary that combines mkdwarfs, dwarfsck, dwarfsextract and dwarfs in a single binary. This binary can be used either through symbolic links with the proper names of the tool, or by passing --tool=<name> as the first argument on the command line.
• Bypass the block cache for uncompressed blocks. This saves copying block data to memory unnecessarily and allows us to keep all uncompressed blocks accessible directly through the memory mapping. Partially addresses github #139.
• Show throughput in the scanning and segmenting phases in mkdwarfs.
• Show how much of a file has been consumed in the segmenting phase. Useful primarily for large files.
• dwarfs and dwarfsextract now have options to enable performance monitoring. This can give insight into the latency of various file system operations.
• Added inode offset cache, which improves read() latency for very fragmented files.

Bugfixes

• Use folly::hardware_concurrency(). Fixes github #130.
• Handle ARCHIVE_FAILED status from libarchive, which could be triggered by trying to write long path names to old archive formats.
• Properly handle unicode path truncation.

Documentation

• Update file system format documentation to cover headers and section indices.

Testing

• Lots of new tools tests.
• Remove dependency on tar and diff binaries.

Other

• Switch to C++20.

dwarfs-0.7.0-RC4

Features

• Add --compress-niceness option to mkdwarfs.

dwarfs-0.7.0-RC3

Bugfixes

• Fix heap-use-after-free in dwarfsextract.

• Fix dwarfs benchmark binary.

Features

• Add --stdout-progress option to dwarfsextract. Fixes github #117.

Other

• Reduce amount of test data to speed up compiles and avoid timeouts on travis.

dwarfs-0.7.0-RC2

Bugfixes

• Fix linking against compression libs. Fixes github #112.

• Default FUSE driver debuglevel to warn in background mode. Fixes github #113.

Features

• Add --chmod option. Fixes github #7.

• Add unreadable files as empty files. Fixes github #40.

Documentation

• Document how to produce bit-identical images

• Update internal operation section of mkdwarfs manpage

• Add more documentation details for --file-hash option

Other

• Test image reproducibility for path and similarity ordering

dwarfs-0.7.0-RC1

Bugfixes

• Fixed extract_block.py, which was incorrectly using printf instead of print.

• Support LZ4 compression levels above 9.

Features

• Added --filter option to support simple (rsync-like) filter rules. This was driven by a discussion on github #6.

• Added --input-list option to support reading a list of input files from a file or stdin. At least partially fixes github #6.

• The compression code has been made more modular. This should make it much easier to add support for more compression algorithms in the future.

• Added support for Brotli compression. This is generally much slower at compression than ZSTD or LZMA, but faster than LZMA, while offering a compression ratio better than ZSTD. Fixes github #76.

• Added support for choosing the file hashing algorithm using the --file-hash option. This allows you to pick a secure hash instead of the default XXH3. Also fixes github #92.

• Improved de-duplication algorithm to only hash files with the same size. File hashing is delayed until at least one more file with the same size is discovered. This happens automatically and should improve scanning speed, especially on slow file systems.

• Added --max-similarity-size option to prevent similarity hashing of huge files. This saves scanning time, especially on slow file systems, while it shouldn't affect compression ratio too much.

• Honour user locale when formatting numbers.

• Added --num-scanner-workers option.

• Added support for extracting corrupted file systems with dwarfsextract. This is enabled using the --continue-on-error and, if really needed, --disable-integrity-check options. Fixes github #51.

Other

• Added unit tests for progress class.

• Lots of internal cleanups.

dwarfs-0.6.2

Bugfixes

• Fix #91: image creation reproducibility. Add --no-create-timestamp option, produce deterministic inode numbers and fix fsst bug that causes symbol tables to be non-deterministic. Images built while omitting create timestamps will now be bit-identical.

• Fix #93: only overwrite existing output file when --force option given on command line.

• Fix #104: extracting large files was causing dwarfsextract to OOM. This was fixed by extracting large files in chunks rather than all at once.

• Fix #105: handle strrchr() return NULL.

• Fix out-of-bounds access (PR #106).

• Fix swapped-out cached block detection (PR #107).

• Fix data race in cached block that was triggered by statistics collection and could cause the process to crash.

• Fix heap-use-after-free when writing section index.

dwarfs-0.6.1

Bugfixes

• Fix binary installation. This caused the 0.6.0 binary release to contain test binaries as well as duplicate binaries.

• The fuse2 driver (dwarfs2) was also missing in the 0.6.0 binary release.

dwarfs-0.6.0

Features

• Add support for cache tidying, which releases cache memory when the mounted file system is unused.

• Section index support for speeding up mount times (fixes #48).

Bugfixes

• Fix and simplify static builds as much as possible. Document how to set up a static build environment. This also fixes #75 and #54. Huge shoutout to Maxim Samsonov (@maxirmx) for implementing most of this!

• Fix #71: driver hangs when unmounting

• Fix #67: dwarfs I/O hangs if call to to fuse_reply_iov fails

• Fix #86: block size bits config issues

• Various build fixes.

dwarfs-0.5.6

Bugfixes

• Build fixes for gcc-11 (fixes #52)

• Use REALPATH in version.cmake to fix building in symbolically linked repositories (fixes #47).