Easy rebasing in case of simultaneous (conceptually non-conflicting) append_dim writes?

[jonaslb]

If there are multiple writers that write with append_dim, then causes a conflict today. This conflict, at least conceptually, seems simple enough to resolve (just move the conflicting blobs to where they would be appended instead). But afaik, this isn't really practical to resolve today. Maybe I'm wrong, in that case I'd love to hear it, but otherwise this could serve as a feature request :)

[rabernat]

Does the order of appending matter here?

[jonaslb]

I would say the order is not important since the xarray dataset to my understanding can always just be reordered along the "append coordinate" (usually a time axis) for slicing purposes anyway. It is important that there are no duplicates though (and a conflict error in that case would be totally appropriate).

[rabernat]

I'm curious about the workflow you're using here which generates these out-of-order updates.

If you're trying to backfill a historical dataset, you don't really want to use the append pattern at all. Instead, you want to pre-allocate the full dataset and then fill in each piece in parallel, as in this example pipeline.

If you're trying to stream new updates e.g. of a weather forecast dataset, it's a good idea to process the new updates sequentially, e.g. using a queue, to ensure they happen in order.

So while there are some tools we could build to help this work more smoothly, there may be other ways to solve your problem.

[jonaslb]

What I'm trying to do is use icechunk with a workflow coordinator that doesn't necessarily think it should do things in order (dagster) nor necessarily one at a time (although that might be possible to set up). It does support combining contiguous runs of time indices though.

So, expanding + writing as a two-step process can work to some degree, but it won't completely lift the challenge. Also I should say I'm not dealing with very large amounts of data at a time here. (A follow up on this: when we're used to xarray/dask, I would expect icechunk.xarray.to_icechunk to already be parallelised if the xarray dataset is chunked?)

Anyway, what can happen is that due to data gaps, multiple load jobs are started (for each contiguous part), possibly running at the same time. Or, maybe one time, for whatever reason, the job failed, and someone restarts it manually, without coordination with the "regular" updates. Each job would expand the time axis and insert the relevant data (or as I have my WIP now, using append_dim). And then whichever one calls commit last would fail.