Thoughts on how Icechunk would handle remote to local replication

[alexander0042]

Hi,

I'm the developer behind Pirate Weather, and I've been looking for a while at a better solution to keeping track of the constantly updating Zarr files that underpin the whole project. In short, as it stands every hour an ingest container runs, converts some forecast GRIB files into a new Zarr, and I overwrite what's on disk via a copy to and from S3. This works, but really isn't elegant, so I've been looking for a better solution for a while.

In a lot of ways IceChunk is perfect, and I'm so happy to have stumbled on it! I've done some tests and it's wonderfully drop-in, keeping all the Zarr ideas, which I really appreciate!

One thing I was wondering is how the format would react to being replicated from remote (s3) to local storage. If I'm writing to a s3 store in one container, RCloneing the bucket to local in another, and reading that local store in a third, would you expect the format to handle that gracefully? As it stands, raw Zarr files are surprisingly resilient to this sort of setup, but not sure about IceChunk. While reads directly off S3 do work, they're considerably slower (~0.05 seconds) and less consistent compared to almost instant reads off local disks, so would love to get a bit more performance here.

[paraseba]

Hi @alexander0042. Glad you are enjoying Icechunk! It is ideal for your use case with frequent updates. A few thoughts:

• The local filesystem store is currently intended mostly for tests and exploration. We don't recommend it for production. It's not optimized, and it doesn't offer some of the consistency guarantees the object store offer.
• That said, replication (S3 -> local) should work out of the box, and if you are only doing reads on the local side, that shouldn't have any issues.

So, I'd say go ahead and give it a try. Please report back any issues you find.

Mandatory disclaimer: Icechunk is really optimized for object store, if you bring the compute close to your bucket it should be very fast. But of course, you cannot compete in latency against local disk. Particularly if your chunks are small.

[paraseba]

You also have other options if you want to trade memory for latency. For example, Icechunk has chunk (and metadata) caching you could enable. These are probably useful in your case, where people are normally querying a small subset of the data.

If you want, we can set up a call to discuss your options. Would love to understand your use case better.

[alexander0042]

Thank you both so much for the lightning fast replies- as someone who is shamefully slow at replying to GitHub issues, I really appreciate it!

Really good perspective here on the overall methodology of Icechunk- hearing that makes me think sticking with S3 is probably the best plan, especially with some caching layered on top (it's less helpful than you'd imagine, since weather models are so high resolution that people are often querying different cells, but does help with repeated temporal updates).

I love to talk about large updating array storage approaches (I've tested pretty well all of them over the years), so feel free to ping me sometime at api@pirateweather.net to set something up!

[rabernat]

I would note that Icechunk is explicitly designed to enable safe and robust caching. In contrast to vanilla Zarr (but inspired by Apache Iceberg) all of the actual files in an icechunk store are immutable, so they can safely be cached forever. Right now we only have in-memory caching, but #960 discusses extensions (e.g. local file cache, distributed cache, etc.) Immutable files in object storage + multiple layers of caching is basically the architecture that powers data warehouses like Snowflake and BigQuery.