Follow HTTP redirect

[rabernat]

Follows #1366.

Users have requested the ability to follow HTTP redirects in Icechunk. The specific use case is to share a generic stable url, e.g.

https://data-provider.org/my-icechunk-repo

And have this redirect to the actual storage location, e.g.

https://backend-server.cloud-provider.com/ABC/123

This would allow the data provider to move the underlying storage location (e.g. to a different cloud provider) without disrupting user workflows. (N.B. Arraylake, Earthmover's hosted Icechunk data platform, already provides this capability today.)

To move this forward, we need to decide which status codes to follow.

[paraseba]

One concern is consistency, Icechunk 2 is frequently requesting the root object for the repository, and unexpected things could happen if this thing changes. I propose the redirect is only respected on first contact with the HTTP storage for the repo.

[paraseba]

I propose we accept redirection status codes: 302, 303, 307. Anything else generates an Icechunk error.

Accepting permanent redirects would be mistake, since it gives the client the right to cache the destination forever, which is not the intended purpose.

[rabernat]

For icechunk 2.0, which is the exact file that would be requested for the URL https://data-provider.org/my-icechunk-repo? https://data-provider.org/my-icechunk-repo/repo?

[paraseba]

For icechunk 2.0, which is the exact file that would be requested for the URL https://data-provider.org/my-icechunk-repo? https://data-provider.org/my-icechunk-repo/repo?

Yes, or whatever we change the name to be, I don't love repo

[aldenks]

I propose the redirect is only respected on first contact with the HTTP storage for the repo.

This seems very reasonable and would cover the case we are interested in.