Streaming or mounting STAC assets from Blob Container

[KennSmithDS]

My team is trying to demonstrate a workflow for training a semantic segmentation model (U-Net architecture specifically) on a Planetary Computer instance, but the pipeline of fetching all source and label .tif assets for a dataset into a stack of xarray is proving to be difficult.

1. PC instance has limited temporary storage provisioned and won't fit the entire dataset "locally" from download
2. Making repeated HTTP requests with STAC label Items to fetch source Items will take hours, and likely not be able to be cached on PC instance memory

Is there a way in Planetary Computer to mount a Blob Container directly to an instance for "local" access to the dataset? Does anyone have experience with this kind of problem, and/or able to solve it with streaming data out of an archive stored on a Blob Container?

[TomAugspurger]

1. PC instance has limited temporary storage provisioned and won't fit the entire dataset "locally" from download

The home directory is fairly limited (15G) but you have a bit more room in outside of it, in e.g. /tmp/ (https://planetarycomputer.microsoft.com/docs/overview/environment/#understanding-the-file-system). For medium size datasets this might be an option. But this is not persisted between notebook server sessions.

Making repeated HTTP requests with STAC label Items to fetch source Items will take hours, and likely not be able to be cached on PC instance memory

Can you expand on this? Where exactly are things slow: the search, the asset downloading, something else? And what do you mean by "cached on PC instance memory"? The STAC metadata or the assets?

https://nbviewer.org/github/microsoft/PlanetaryComputerExamples/blob/main/tutorials/mosaiks.ipynb#Extract-features-from-the-imagery-around-each-point has an example of structuring and parallelizing queries for a large number of points.

Is there a way in Planetary Computer to mount a Blob Container directly to an instance for "local" access to the dataset?

That's possible for the Hub admin (e.g. https://github.com/microsoft/planetary-computer-hub/blob/main/helm/values.yaml#L183-L189), but not as a user of the hub. You can use the Python SDK or adlfs to work with the remote files. Either way, the files are still in the Azure Storage Service, so performance of accessing data from that container will be comparable to accessing Planetary Computer data. So if it's downloading the assets that's the problem, you'll still be bound by the network.

[KennSmithDS]

Hey Tom,

Thanks for the thorough and rapid response as always!

The home directory is fairly limited (15G)

This is the first problem we encountered, trying to download LandCoverNet, for example which is 80+ GB, definitely won't fit on the home directory. Datasets not persisting between notebook server sessions using /tmp also won't work too well if we're handling large datasets for ML training.

Making repeated HTTP requests with STAC label Items to fetch source Items will take hours, and likely not be able to be cached on PC instance memory

Expanding on this, there are two approaches we are working on in parallel right now for comparison:

1. Download the label archive to PC notebook server instance, read as label Collection (which is self referential when downloaded from MLHub) and update the LabelRelType.SOURCE Link hrefs to point to the source Item endpoints via our API. Then HTTP requests are made and the JSON responses are loaded as source Item STAC Objects in memory using the Item.from_dict() call. This is an O(n) bottleneck, as we scale up the sample size of labels we want to process, so does the number of HTTP requests required. Fetching even a subset of all source Items (4 from each chip, instead of all 72) is estimated to take upward of 20 hours.

2. Use the pystac-client API to construct a paginated Catalog in memory so that labels can reference the source Items and Assets directly by making less HTTP requests. However the CollectionClient object (inherited from the regular STAC Collection) method to get_item() performs a walk on the entire Collection for each Item. So even though it's able to handle the paginated results and fetch 100 source Items for each request, this becomes an O(n^2) bottleneck because it walks the Collection for each source Item linked to a label Item (technically O(nm) if n total source items and m known label items).

The entire catalog of JSON files by itself is 1.8GB, which I think should fit in the memory of the notebook server instance, but getting to that point is the issue. Also using Dask delayed to load the source chips into stackstac and train the model hasn't been a problem so far when working with a very small sample.

https://nbviewer.org/github/microsoft/PlanetaryComputerExamples/blob/main/tutorials/mosaiks.ipynb#Extract-features-from-the-imagery-around-each-point has an example of structuring and parallelizing queries for a large number of points.

This is a great tutorial and good demonstration of using Dask to parallelize searching a collection with known spatial and temporal parameters. I think our goal is to train a model with the entire LandCoverNet dataset, so we don't have a dataframe of designated points to parse/parallelize the querying of Items/Assets. Perhaps we need to look at futures to parallelize the task of querying the API for source Items.