IPFS as local CAS and local storage savings?

[KristoferHansson]

If I understand correctly IPFS is chunk based CAS, and nerdctl has support for IPFS, has anyone done any evaluation of it's positive effect on local disc storage if utilizing only local IPFS? Composefs storage support in podman has the ability to offer file level deduplication support within/across layers (not just only layer based deduplication), would IPFS be able to offer an additional level of deduplication? Or is it like zstd:chunked, only has a positive effect during pull (and based on my current experience, mostly only on file level)?

[AkihiroSuda]

stargz-snapshotter supports lazy pulling eStargz over IPFS: https://github.com/containerd/stargz-snapshotter/blob/main/docs/ipfs.md
Does this work for you ?

cc @ktock

[KristoferHansson]

@AkihiroSuda : Thanks for the pointer. I will take a look at it and see how it acts to get a better understanding.

Altho it's not really the lazy pulling(for speeding up container start) part I'm interested in, it's more of the deduplication, 1: not having to pull everything if I already from an earlier pull, in local cache, have a large portion of the chunks in the image about to be pulled, 2: local cache deduplication, chunk based CAS

In my experiments with zstd:chunked push and pull using podman I was unable to get any deduplication of "parts of file" chunks to work, only file level deduplication, so I started thinking about IPFS, knowing from before nerdctl had support for it.

Will also take a look at overlaybd and nydus and se how well they could fulfill my goals.