Creating an icechunk repo on Pangeo-EOSC s3-compatible MinIO storage

[rsignell]

I'm trying to create an icechunk repo on the Pangeo-EOSC s3-compatible MinIO storage.

I've got AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set as environment variables, and and when I use:

endpoint = 'https://pangeo-eosc-minioapi.vm.fedcloud.eu'
storage = icechunk.s3_storage(bucket="rsignell4-protocoast", prefix="icechunk", from_env=True, endpoint_url=endpoint,
                                allow_http=True, force_path_style=True, region='us-east-1')
repo = icechunk.Repository.create(storage)

I get back:

IcechunkError:   x error writing object to object store service error
  | 
  | context:
  |    0: icechunk::storage::s3::write_snapshot
  |            with id=1CECHNKREP0F1RSTCMT0
  |              at icechunk/src/storage/s3.rs:653
  |    1: icechunk::asset_manager::write_snapshot
  |              at icechunk/src/asset_manager.rs:201
  |    2: icechunk::repository::create
  |              at icechunk/src/repository.rs:146
  | 
  |-> error writing object to object store service error
  |-> service error
  |-> unhandled error (AccessDenied)
  `-> Error { code: "AccessDenied", message: "There were headers present in the request which were not signed", s3_extended_request_id:
      "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8", aws_request_id: "18566CF4F5F556CB" }

What should I try next?

[paraseba]

Hello @rsignell, thank you for reporting this.

We do most of our testing using MinIO and we have never seen this. Do you have any proxies between you and MinIO?

[paraseba]

pangeo-data/pangeo-eosc#114

[rsignell]

Although I'm able to create storage on the Pangeo-EOSC storage, I'm still struggling to create an icechunk repo.

I'm close, but my code here:

import icechunk
import os

def open_or_create_icechunk_repo(
    bucket_name: str,
    store_name: str,
    endpoint_url: str,
    region: str = "us-west-2"
):
    """
    Opens or creates an icechunk repository on an S3-compatible storage endpoint.

    This function assumes that AWS credentials (AWS_ACCESS_KEY_ID and
    AWS_SECRET_ACCESS_KEY) are available as environment variables.

    Args:
        bucket_name (str): The name of the S3 bucket.
        store_name (str): A unique name for your data store within the bucket.
                          This will be used as a prefix.
        endpoint_url (str): The full URL of the S3-compatible endpoint.
        region (str, optional): The AWS region. Defaults to "us-west-2".

    Returns:
        icechunk.Repository: An initialized repository object, or None if an
                             error occurs.
    """
    print(f"Attempting to open/create repo '{store_name}' in bucket '{bucket_name}'...")
    print(f"Endpoint: {endpoint_url}, Region: {region}")

    try:
        # 1. Configure the main storage backend for repository metadata.
        #    We pass the endpoint_url here and specify that credentials
        #    should be sourced from the environment.
        storage = icechunk.s3_storage(
            bucket=bucket_name,
            prefix=f"icechunk/{store_name}",
            from_env=True,  # Use AWS credentials from environment variables
            endpoint_url=endpoint_url,
            region=region
        )

        # 2. Set up the general repository configuration.
        config = icechunk.RepositoryConfig.default()

        # 3. Configure the virtual chunk container for the actual data chunks.
        #    It's crucial to also provide the endpoint_url here so the repository
        #    knows where to read/write the data chunks.
        s3_chunk_store = icechunk.s3_store(
            region=region,
            endpoint_url=endpoint_url
        )
        
        config.set_virtual_chunk_container(
            icechunk.VirtualChunkContainer(
                f"s3://{bucket_name}/", # The protocol and host it handles
                s3_chunk_store
            )
        )

        # 4. Assemble all configuration options for creation.
        repo_config = {
            "storage": storage,
            "config": config,
        }

        # 5. Initialize the repository. This is the correct method for creation.
        #    Note: This will likely raise an error if the repository already exists.
        #    A more robust implementation would be to try opening first, and only
        #    initialize if that fails.
        print("Attempting to initialize repository...")
        repo = icechunk.Repository.create(**repo_config)
        print("Successfully initialized icechunk repository.")
        return repo

    except Exception as e:
        # A common error here might be that the repository already exists.
        # We can try to open it in that case.
        if "already exists" in str(e): # This is a guess at the error message
            try:
                print("Repository seems to exist. Attempting to open...")
                # If initialization fails because it exists, we open it.
                # Note: We only need the 'storage' object to open.
                repo = icechunk.Repository.open(storage)
                print("Successfully opened existing repository.")
                return repo
            except Exception as open_e:
                print(f"Failed to open existing repository: {open_e}")
                return None
        else:
            print(f"An error occurred while creating the repository: {e}")
            return None


# --- Example Usage ---
# Before running, make sure to set your environment variables, for example:
# export AWS_ACCESS_KEY_ID='YOUR_ACCESS_KEY'
# export AWS_SECRET_ACCESS_KEY='YOUR_SECRET_KEY'

if __name__ == "__main__":
    # Replace with your actual bucket, store name, and endpoint URL
    MY_BUCKET = "rsignell4-protocoast"
    MY_STORE_NAME = "scientific-dataset-1"
    MY_ENDPOINT_URL = "https://pangeo-eosc-minioapi.vm.fedcloud.eu"

    # Check if credentials are set (basic check)
    if not os.getenv("AWS_ACCESS_KEY_ID") or not os.getenv("AWS_SECRET_ACCESS_KEY"):
        print("Error: AWS credentials are not set in the environment.")
    else:
        icechunk_repo = open_or_create_icechunk_repo(
            bucket_name=MY_BUCKET,
            store_name=MY_STORE_NAME,
            endpoint_url=MY_ENDPOINT_URL
        )

        if icechunk_repo:
            # You can now work with the repository object
            print("Repository is ready to use.")
            # For example, you can check its state:
            # print(f"Repository state: {icechunk_repo.state()}")
            

is returning:

Attempting to open/create repo 'scientific-dataset-1' in bucket 'rsignell4-protocoast'...
Endpoint: https://pangeo-eosc-minioapi.vm.fedcloud.eu/, Region: us-west-2
Attempting to initialize repository...
An error occurred while creating the repository:   x error listing objects in object store dispatch failure
  | 
  | context:
  |    0: icechunk::repository::create
  |              at icechunk/src/repository.rs:168
  | 
  |-> error listing objects in object store dispatch failure
  |-> dispatch failure
  |-> io error
  |-> error trying to connect: dns error: failed to lookup address information: Name or service not known
  |-> dns error: failed to lookup address information: Name or service not known
  `-> failed to lookup address information: Name or service not known

@paraseba do you see my issue?

[paraseba]

Two thoughts:

• verify the machine where you are running this can hit https://pangeo-eosc-minioapi.vm.fedcloud.eu/ without issues.
• Try passing force_path_style=True to s3_storage

[rabernat]

Sometimes it's necessary to pass region as a parameter, right? Could that be relevant here?

[paraseba]

Yes, but it looks like @rsignell is passing the region

[rsignell]

Woohoo @paraseba , that was it!

import icechunk
import os

def open_or_create_icechunk_repo(
    bucket_name: str,
    store_name: str,
    endpoint_url: str,
    region: str = "us-west-2"
):
    """
    Opens or creates an icechunk repository on an S3-compatible storage endpoint.

    This function assumes that AWS credentials (AWS_ACCESS_KEY_ID and
    AWS_SECRET_ACCESS_KEY) are available as environment variables.

    Args:
        bucket_name (str): The name of the S3 bucket.
        store_name (str): A unique name for your data store within the bucket.
                          This will be used as a prefix.
        endpoint_url (str): The full URL of the S3-compatible endpoint.
        region (str, optional): The AWS region. Defaults to "us-west-2".

    Returns:
        icechunk.Repository: An initialized repository object, or None if an
                             error occurs.
    """
    print(f"Attempting to open/create repo '{store_name}' in bucket '{bucket_name}'...")
    print(f"Endpoint: {endpoint_url}, Region: {region}")

    try:
        # 1. Configure the main storage backend for repository metadata.
        #    We pass the endpoint_url here and specify that credentials
        #    should be sourced from the environment.
        storage = icechunk.s3_storage(
            bucket=bucket_name,
            prefix=f"icechunk/{store_name}",
            from_env=True,  # Use AWS credentials from environment variables
            endpoint_url=endpoint_url,
            force_path_style=True,
            region=region
        )

        # 2. Set up the general repository configuration.
        config = icechunk.RepositoryConfig.default()

        # 3. Configure the virtual chunk container for the actual data chunks.
        #    It's crucial to also provide the endpoint_url here so the repository
        #    knows where to read/write the data chunks.
        s3_chunk_store = icechunk.s3_store(
            region=region,force_path_style=True,
            endpoint_url=endpoint_url
        )
        
        config.set_virtual_chunk_container(
            icechunk.VirtualChunkContainer(
                f"s3://{bucket_name}/", # The protocol and host it handles
                s3_chunk_store
            )
        )

        # 4. Assemble all configuration options for creation.
        repo_config = {
            "storage": storage,
            "config": config,
        }

        # 5. Initialize the repository. This is the correct method for creation.
        #    Note: This will likely raise an error if the repository already exists.
        #    A more robust implementation would be to try opening first, and only
        #    initialize if that fails.
        print("Attempting to initialize repository...")
        repo = icechunk.Repository.create(**repo_config)
        print("Successfully initialized icechunk repository.")
        return repo

    except Exception as e:
        # A common error here might be that the repository already exists.
        # We can try to open it in that case.
        if "already exists" in str(e): # This is a guess at the error message
            try:
                print("Repository seems to exist. Attempting to open...")
                # If initialization fails because it exists, we open it.
                # Note: We only need the 'storage' object to open.
                repo = icechunk.Repository.open(storage)
                print("Successfully opened existing repository.")
                return repo
            except Exception as open_e:
                print(f"Failed to open existing repository: {open_e}")
                return None
        else:
            print(f"An error occurred while creating the repository: {e}")
            return None


# --- Example Usage ---
# Before running, make sure to set your environment variables, for example:
# export AWS_ACCESS_KEY_ID='YOUR_ACCESS_KEY'
# export AWS_SECRET_ACCESS_KEY='YOUR_SECRET_KEY'

if __name__ == "__main__":
    # Replace with your actual bucket, store name, and endpoint URL
    MY_BUCKET = "rsignell4-protocoast"
    MY_STORE_NAME = "scientific-dataset-1"
    MY_ENDPOINT_URL = "https://pangeo-eosc-minioapi.vm.fedcloud.eu"

    # Check if credentials are set (basic check)
    if not os.getenv("AWS_ACCESS_KEY_ID") or not os.getenv("AWS_SECRET_ACCESS_KEY"):
        print("Error: AWS credentials are not set in the environment.")
    else:
        icechunk_repo = open_or_create_icechunk_repo(
            bucket_name=MY_BUCKET,
            store_name=MY_STORE_NAME,
            endpoint_url=MY_ENDPOINT_URL
        )

        if icechunk_repo:
            # You can now work with the repository object
            print("Repository is ready to use.")
            # For example, you can check its state:
            # print(f"Repository state: {icechunk_repo.state()}")

produces:

Attempting to open/create repo 'scientific-dataset-1' in bucket 'rsignell4-protocoast'...
Endpoint: https://pangeo-eosc-minioapi.vm.fedcloud.eu/, Region: us-west-2
Attempting to initialize repository...
Successfully initialized icechunk repository.
Repository is ready to use.

[TomNicholas]

   # 5. Initialize the repository. This is the correct method for creation.
   #    Note: This will likely raise an error if the repository already exists.
   #    A more robust implementation would be to try opening first, and only
   #    initialize if that fails.

I missed all the context but @rsignell why can't you just use Repository.open_or_create() here?

[rsignell]

Sorry, that comment was left over from original Gemini code. The code above does in fact use Repository.create and Repository.open. Here's the code again with more accurate comments:

import icechunk
import os

def open_or_create_icechunk_repo(
    bucket_name: str,
    store_name: str,
    endpoint_url: str,
    region: str = "us-west-2"  # not-used since we are writing to a non-AWS endpoint, but required by icechunk
):
    """
    Opens or creates an icechunk repository on an S3-compatible storage endpoint.

    This function assumes that AWS credentials (AWS_ACCESS_KEY_ID and
    AWS_SECRET_ACCESS_KEY) are available as environment variables.

    Args:
        bucket_name (str): The name of the S3 bucket.
        store_name (str): A unique name for your data store within the bucket.
                          This will be used as a prefix.
        endpoint_url (str): The full URL of the S3-compatible endpoint.
        region (str, optional): The AWS region. Defaults to "us-west-2".

    Returns:
        icechunk.Repository: An initialized repository object, or None if an
                             error occurs.
    """
    print(f"Attempting to open/create repo '{store_name}' in bucket '{bucket_name}'...")
    print(f"Endpoint: {endpoint_url}, Region: {region}")

    try:
        # 1. Configure the main storage backend for repository metadata.
        #    We pass the endpoint_url here and specify that credentials
        #    should be sourced from the environment.
        storage = icechunk.s3_storage(
            bucket=bucket_name,
            prefix=f"icechunk/{store_name}",
            from_env=True,  # Use AWS credentials from environment variables
            endpoint_url=endpoint_url,
            force_path_style=True,
            region=region
        )

        # 2. Set up the general repository configuration.
        config = icechunk.RepositoryConfig.default()

        # 3. Configure the virtual chunk container for the actual data chunks.
        #    It's crucial to also provide the endpoint_url here so the repository
        #    knows where to read/write the data chunks.
        s3_chunk_store = icechunk.s3_store(
            region=region,force_path_style=True,
            endpoint_url=endpoint_url
        )
        
        config.set_virtual_chunk_container(
            icechunk.VirtualChunkContainer(
                f"s3://{bucket_name}/", # The protocol and host it handles
                s3_chunk_store
            )
        )

        # 4. Assemble all configuration options for creation.
        repo_config = {
            "storage": storage,
            "config": config,
        }

        # 5. Create the repository. 
        #    Note: This will likely raise an error if the repository already exists.
        print("Attempting to create the repository...")
        repo = icechunk.Repository.create(**repo_config)
        print("Successfully created icechunk repository.")
        return repo

    except Exception as e:
        # The repository might already exist. 
        # We can try to open it in that case.
        if "repositories can only be created in clean prefixes" in str(e): 
            try:
                print("Repository seems to exist. Attempting to open...")
                # If initialization fails because it exists, we open it.
                # Note: We only need the 'storage' object to open.
                repo = icechunk.Repository.open(storage)
                print("Successfully opened existing repository.")
                return repo
            except Exception as open_e:
                print(f"Failed to open existing repository: {open_e}")
                return None
        else:
            print(f"An error occurred while creating the repository: {e}")
            return None


# --- Example Usage ---
# Before running, make sure to set your environment variables, for example:
# export AWS_ACCESS_KEY_ID='YOUR_ACCESS_KEY'
# export AWS_SECRET_ACCESS_KEY='YOUR_SECRET_KEY'

if __name__ == "__main__":
    # Replace with your actual bucket, store name, and endpoint URL
    MY_BUCKET = "rsignell4-protocoast"
    MY_STORE_NAME = "scientific-dataset-1"
    MY_ENDPOINT_URL = "https://pangeo-eosc-minioapi.vm.fedcloud.eu"

    # Check if credentials are set (basic check)
    if not os.getenv("AWS_ACCESS_KEY_ID") or not os.getenv("AWS_SECRET_ACCESS_KEY"):
        print("Error: AWS credentials are not set in the environment.")
    else:
        icechunk_repo = open_or_create_icechunk_repo(
            bucket_name=MY_BUCKET,
            store_name=MY_STORE_NAME,
            endpoint_url=MY_ENDPOINT_URL
        )

        if icechunk_repo:
            # You can now work with the repository object
            print("Repository is ready to use.")
            # For example, you can check its state:
            # print(f"Repository state: {icechunk_repo.state()}")

[TomNicholas]

The code above does in fact use Repository.create and Repository.open.

I didn't mean "or" in the sense of "use this method or that method". There is a 3rd method literally called open_or_create. Isn't that what you want here instead of your try...except?

[rsignell]

@TomNicholas -- oh man, why is there no "egg-on-face" emoticon? 😄
I see now. And yes, of course that's what I should use. Even Gemini thinks so! 🙄

[TomNicholas]

While we're at it - the young-uns tell me that copy-pasting code into ChatGPT / Claude /Gemini is old-hat now - it's much easier to run Claude Code / Gemini CLI / Cursor locally as they essentially do that step for you.

[rsignell]

Here's the working code I ended up with that creates icechunk storage on an s3-compatible MinIO endpoint (Pangeo-EOSC) storing references for a virtual dataset that accesses data chunks from NetCDF3 stored in s3-compatible Ceph storage (Open Storage Network):
https://nbviewer.org/gist/rsignell/673d031bd27a73a600348bbd6a431d04
💦