Trying to create an Icechunk repo on OSN

[rsignell]

I'm trying to create an icechunk repo on OSN, and when I try this:

import icechunk
storage = icechunk.s3_storage(bucket="esip", endpoint_url='https://usgs.osn.mghpcc.org',
                              prefix="rsignell/necofs/icechunk2", 
                              access_key_id=os.environ['OSN_ACCESS_KEY_ID'],
                              secret_access_key=os.environ['OSN_SECRET_ACCESS_KEY'])
repo = icechunk.Repository.create(storage)

I get back:

---------------------------------------------------------------------------
IcechunkError                             Traceback (most recent call last)
Cell In[13], line 6
      1 import icechunk
      2 storage = icechunk.s3_storage(bucket="esip", endpoint_url='https://usgs.osn.mghpcc.org',
      3                               prefix="rsignell/necofs/icechunk2", 
      4                               access_key_id=os.environ['OSN_ACCESS_KEY_ID'],
      5                               secret_access_key=os.environ['OSN_SECRET_ACCESS_KEY'])
----> 6 repo = icechunk.Repository.create(storage)

File ~/miniforge3/envs/icechunk/lib/python3.12/site-packages/icechunk/repository.py:51, in Repository.create(cls, storage, config, virtual_chunk_credentials)
     25 @classmethod
     26 def create(
     27     cls,
   (...)
     30     virtual_chunk_credentials: dict[str, AnyCredential] | None = None,
     31 ) -> Self:
     32     """
     33     Create a new Icechunk repository.
     34     If one already exists at the given store location, an error will be raised.
   (...)
     48         An instance of the Repository class.
     49     """
     50     return cls(
---> 51         PyRepository.create(
     52             storage,
     53             config=config,
     54             virtual_chunk_credentials=virtual_chunk_credentials,
     55         )
     56     )

IcechunkError:   x error listing objects in object store dispatch failure
  | 
  | context:
  |    0: icechunk::repository::create
  |              at icechunk/src/repository.rs:138
  | 
  |-> error listing objects in object store dispatch failure
  |-> dispatch failure
  |-> io error
  |-> error trying to connect: dns error: failed to lookup address information: Name or service not known
  |-> dns error: failed to lookup address information: Name or service not known
  `-> failed to lookup address information: Name or service not known

I'm a novice here, so guessing/hoping it's a simple user error?

[dcherian]

As the error message says, it's a dns error.

[rsignell]

That's strange because everything with fsspec seems to work -- I tried listing and uploading a local file.
For example, this works:

import fsspec
fs = fsspec.filesystem('s3', anon=True, endpoint_url='https://usgs.osn.mghpcc.org/')
fs.ls('esip/rsignell/era5')

producing

['esip/rsignell/era5/combined_usa.parq']

[rsignell]

So should I raise an issue?

[rabernat]

It's not necessary to open a new issue. We are aware of this now.

I wonder if this is because it's missing a region parameter. Look how we have to do it for minio: https://icechunk.io/en/latest/icechunk-python/storage/#minio

icechunk.s3_storage(
    bucket="icechunk-test",
    prefix="quickstart-demo-1",
    region='us-east-1',
    access_key_id='minio',
    secret_access_key='minio123',
    endpoint_url='http://localhost:9000',
    allow_http=True,
)

Can you try adding region='us-east-1' to your configuration?

[rsignell]

I tried adding region='us-east-1' (even though there are no "regions" with OSN), but the error is the same. I'm happy to share the OSN credentials with you guys if you want to try.

[rsignell]

Oh, I just tried adding allow_http=True and THAT worked! (no region specification necessary!)

[rabernat]

So I guess there is a redirect happening from https to http by the server? Which means your credentials are being passed over the open internet unencrypted? Sounds sketchy. 🤔