Using cog3pio to determine byte ranges in COG files?

[TomNicholas]

Basically same question as gauteh/hidefix#38 (comment) but for this library 😁

I'm building VirtualiZarr, an evolution of kerchunk, that allows you to determine byte ranges of chunks in netCDF files, but then concatenate the virtual representation of those chunks using xarray's API.

This works by creating a ChunkManifest object in-memory (one per netCDF Variable per file initially), then defining ways to merge those manifests.

What I'm wondering is if cog3pio's code could be useful to me as a way to generate the ChunkManifest for a netCDF file without using kerchunk/fsspec (see this issue). In other words I use cog3pio only to determine the byte ranges, not for actually reading the data. (I plan to actually read the bytes later as if it were zarr using the rust object-store crate, see zarr-developers/zarr-python#1661).

Q's:

• Is this idea dumb?
• Does cog3pio expose the byte range information currently?
• Can cog3pio read over S3?

cc @norlandrhagen

[weiji14]

Oh hi Tom, I was just chatting to @norlandrhagen at the Pangeo weekly meeting 😆. I will say that yes, it should be possible to work out the byte ranges from just the GeoTIFF's header, and we can expose that via an API function somehow. I just need to figure out how GDAL does this, and re-implement it here (easier said than done).

Note that I'm already using object-store in cog3pio (#5), and passing a HTTP url to a GeoTIFF should already work. Reading from s3 (or azure, gcp, etc) will work too if I enable the feature flag here and recompile:

cog3pio/Cargo.toml

Line 17 in 33285da

object_store = { version = "0.9.0", features = ["http"] }

I'm aware that the Zarr v3 implementation is using object-store, and keeping an eye on progress at https://github.com/roeap/object-store-python. Would definitely be keen to standardize on object-store as the 'fsspec-for-Rust'.

[TomNicholas]

Awesome! Thanks @weiji14

Would definitely be keen to standardize on object-store as the 'fsspec-for-Rust'.

Yeah this would be great, and I like the way you've described the aim there.

[martindurant]

I would add, that there's no reason that kerchunk needs to use fsspec - it's really a package of ways to make reference files. Therefore, if you come up with a way to get COG offsets, it can easily live there together with the other reference makers. Does TIFFFile do the required work too?

[print-sid8]

Hey guys, I have an implementation of what GDAL does in Python here, it may not be neat but i got it working pretty well for my rasteret library.

This can understand the headers of a COG, then decide which byte ranges to read for a given geometry AOI.

maybe its useful for you to port to rust? async-tiff by Kyle Barron already has most of what I made, he released his rust codebase after I made mine, I would prefer to offload the high concurrent reading bits from my httpx implementation to his rust implementation.

Anyway im leaving it here for you guys to checkout

https://github.com/terrafloww/rasteret/blob/main/src/rasteret/stac/parser.py
https://github.com/terrafloww/rasteret/blob/main/src/rasteret/fetch/cog.py

[weiji14]

Thanks for chiming in @print-sid8! It's on the roadmap to port the backend to async-tiff, though it's lower on my priority list for 2025 right now 🙈

This can understand the headers of a COG, then decide which byte ranges to read for a given geometry AOI.

maybe its useful for you to port to rust?

Yes, I think it would be useful to port the logic in your compute_tile_indices function (+some other ones) to Rust. Just need to find the time for it 😭

async-tiff by Kyle Barron already has most of what I made, he released his rust codebase after I made mine, I would prefer to offload the high concurrent reading bits from my httpx implementation to his rust implementation.

I see you've switched to aiohttp already in terrafloww/rasteret#8 😄 My hunch is that you've probably got most of the speedup gain already by switching the network call from a synchronous to asynchronous I/O.

[print-sid8]

I see you've switched to aiohttp already in terrafloww/rasteret#8 😄 My hunch is that you've probably got most of the speedup gain already by switching the network call from a synchronous to asynchronous I/O.

Oh yes, jeff's benchmarks obviously made me move to aiohttp, and it has had very good improvements on certain parts of my library, like the bootstrapping of the COG headers into stac-geoparquet. So similar to you, i don't think ill spend to much time trying to refactor my codebase again to add async-tiff immediately to rasteret.

I would rather take time to understand efforts like cog3pio and contribute what I can.

Yes, I think it would be useful to port the logic in your compute_tile_indices function (+some other ones) to Rust. Just need to find the time for it 😭

I'm no rust sensei but I can try. 😄 Let me know your ideas and philosophy behind where such similar functions might help in cog3pio, I'll try and understand and give it a shot.

[weiji14]

I would rather take time to understand efforts like cog3pio and contribute what I can.

Yes, I think it would be useful to port the logic in your compute_tile_indices function (+some other ones) to Rust. Just need to find the time for it 😭

I'm no rust sensei but I can try. 😄 Let me know your ideas and philosophy behind where such similar functions might help in cog3pio, I'll try and understand and give it a shot.

Cool, would be great to get you into Rust 😄 My long-term idea is to have the following dependency chain:

async-tiff (Rust 🦀 core, low-level) -> geotiff (Rust 🦀 'geo' 🌐 lib, medium-level) -> cog3pio (Rust+Python 🦀🐍 high-level bindings)

Right now, the picture is a little complicated, as there isn't a clear separation of pure-TIFF/GeoTIFF and low-level/high-level API interfaces. Again, limiting factor is just having the time + people to do all this!

If you just want to focus on porting the 'compute_tile_indices' function, I'd recommend checking out this issue - georust/geotiff#23, and we can continue the discussion there.

[TomNicholas]

dependency chain

Interesting, thanks for describing this plan @weiji14 !

So for a VirtualiZarr (geo)tiff parser, which of those layers is needed? All 3? (For a parser written in python)

[weiji14]

So for a VirtualiZarr (geo)tiff parser, which of those layers is needed? All 3? (For a parser written in python)

Yeah, about that... at the moment, async-tiff has some geo-bits, and virtual-tiff is relying on it (for good or for bad), which is in turn used by Virtualizarr. It depends on what Virtualizarr is after, if you want to virtualize any TIFFs (including non-geo ones), then this current async-tiff -> virtual-tiff -> Virtualizarr setup probably works fine. If Virtualizarr just wants to focus on GeoTIFFs, then I'd say async-tiff -> geotiff -> virtual-(geo)tiff -> Virtualizarr? The latter case will slowly become feasible once developmentseed/async-tiff#100 is implemented.

Which is to say, I could probably close this issue since I don't anticipate cog3pio having an API for determining byte ranges anymore. The ecosystem has changed significantly since this thread was opened in Apr 2024, most notably async-tiff arrived on the scene, and I should point you to threads like developmentseed/async-tiff#6 if you're interested in smart range request handling.

[TomNicholas]

It depends on what Virtualizarr is after, if you want to virtualize any TIFFs (including non-geo ones)

Yes I want to be able to virtualize non-geo TIFF data too! But I am not sure whether that's what @MaxJones is going for.

async-tiff has some geo-bits

Really? Why? (We should presumably discuss this elsewhere)