Optionally deduplicate segments in compact mode

[gatesn]

Within the compact strategy, we should try to deduplicate segments by checksum.

Recommended by @kszucs

See https://huggingface.co/blog/parquet-cdc

[kszucs]

Hi @gatesn!

I think content-defined chunking is a different concept. The deduplication doesn't happen in the file format itself but rather at the storage side. You can read more about it in the FastCDC paper.

What we have done in parquet is that we apply a similar chunking algorithm to consistently split data pages according to its content, minimizing the number of changed data pages when there are changes in the data (e.g. when comparing two revisions of the same file). Then the content addressable storage receiving the serialized file can efficiently deduplicate the data. Alternatively you can imagine having a KV-store (which could be a blob store as well) and during transfer you negotiate which data pages are already existing at the storage side and which are missing, then only transfer the missing pages (similarly when downloading).

My explanation may not be straightforward, so let me know if you need more context :)

Couple of references:

• https://joshleeb.com/posts/content-defined-chunking.html
• https://github.com/huggingface/dataset-dedupe-estimator
• libparquet implementation: chunker_internal.h and chunker_internal.cc