Is it possible to use the Icechunk Storage to set and get data outside the Repository?

[josephnowak]

I would like to know if I can use the Icechunk storage to write some parquet files outside the repository, something similar to what can be done if I use Fsspec but with the performance improvement of a package done in Rust like Icechunk (this would also avoid me the creation or use of fsspec in conjunction with Icechunk on the same project).

Context:
I have one process that runs a set of N tasks, and every task can generate an array of different shapes on different random dates (you can see it as Ai,j represents an array of unknown length for the date "i" for the task "j"), due to the unknown shape of the data and that sometimes it can weight more than a couple of GBs, I decided to write every array on a parquet file and only preserve on memory some metadata that allows me to create a dataset with all the coords and chunks in a deterministic way, and use that information to read the parquet files in a lazy way to write it to Icechunk.

[dcherian]

No. Icechunk is a backend for Zarr, not a replacement for fsspec. Look at obstore for that: https://github.com/developmentseed/obstore

[TomNicholas]

something similar to what can be done if I use Fsspec

At this point in time, Icechunk isn't intended to be used as a general filesystem abstraction, but only for reading and writing zarr chunks. (If you want rust-powered speedup of general reads and writes to object storage you might be interested in obstore.)

Context:
I have one process that runs a set of N tasks, and every task can generate an array of different shapes on different random dates (you can see it as Ai,j represents an array of unknown length for the date "i" for the task "j"), due to the unknown shape of the data and that sometimes it can weight more than a couple of GBs, I decided to write every array on a parquet file and only preserve on memory some metadata that allows me to create a dataset with all the coords and chunks in a deterministic way, and use that information to read the parquet files in a lazy way to write it to Icechunk.

This sounds quite convoluted - either you have a very unusual use case or you're doing something odd! 😁 Can you tell us more about your use case?

[josephnowak]

Thanks for the prompt response, I will use obstore for sure, but I wanted to check if Icechunk already provided the same capabilities to avoid adding additional dependencies to my project.

Related to my use case, I think it's unusual, I'm developing/improving a backtester for financial indexes (portfolio backtester is also a common term for it). The process consists of selecting a set of instruments every X days during a range of dates and assigning a weight to all of them (the weight indicates the percentage of the money that you want to invest in every instrument), the way to select and weight the instruments depends a lot of every index but the most simple ones take the best ranked based on their Market Capitalization. The market capitalization changes constantly/daily which makes the instruments also change making the determination of the final shape a little bit hard (sometimes 50K+ instruments sometimes less than 10 it varies a lot).