Integration with Apache Ozone for s3-compatible data store

[the-Fitz]

Does anyone have experience configuring lakeFS to use Apache Ozone for it's block storage solution?

I am currently attempting this configuration myself and I'm running into AuthorizationHeaderMalformed errors.

I have lakeFS and Ozone operating independently of one another, and I can interact with the ozone API endpoint to control buckets and objects:

> aws s3 ls s3://ozone-bucket --endpoint https://ozone.mydomain.com --profile lakefs-user
2025-04-29 16:12:47       1453 test.txt

I am using the following configuration to enable the two apps to speak:

    stats:
      enabled: false
    database:
      type: postgres
    blockstore:
      type: s3
      s3:
        region: us-east-1
        force_path_style: true
        endpoint: https://ozone.mydomain.com
        skip_verify_certificate_test_only: true
        credentials:
          access_key_id: XXXXX-XXXXXX-XXXXXXX
          secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    gateways:
      s3:
        domain_name: lakefs-s3.mydomain.com
        region: us-east-1

Note: I don't currently have authorization enabled in ozone as this is just a PoC. So my understanding is that the credentials I set don't seem to matter and my initial CLI tests seem to corroborate that.

I have successfully connected MinIO to my lakeFS deployment using a similar configuration block.

This is an example of the errors that I see in the lakeFS pod logs:

time="2025-04-29T20:54:20Z" level=trace msg="dispatched execute result" func="pkg/batch.(*Executor).Run.func2" file="build/pkg/batch/executor.go:170" host=lakefs.mydomain.com key="GetRepository:test" method=POST operation_id=CreateRepository path=/api/v1/repositories request_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx service_name=rest_api user=XXXXX waiters=1
time="2025-04-29T20:54:20Z" level=debug msg="performing API action" func="pkg/api.(*Controller).LogAction" file="build/pkg/api/controller.go:5280" class=api_server client=lakefs-webui/1.28.0 host=lakefs.mydomain.com method=POST name=create_repo operation_id=CreateRepository path=/api/v1/repositories ref="" repository=test request_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx service=api_gateway service_name=rest_api source_ip="XXX.XXX.XXX.XXX:57880" source_ref="" user=XXXXX user_id=XXXXX
time="2025-04-29T20:54:20Z" level=debug msg="performing API action" func="pkg/api.(*Controller).LogAction" file="build/pkg/api/controller.go:5280" class=api_server client=lakefs-webui/1.28.0 host=lakefs.mydomain.com method=POST name=repo_sample_data operation_id=CreateRepository path=/api/v1/repositories ref="" repository=test request_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx service=api_gateway service_name=rest_api source_ip="XXX.XXX.XXX.XXX:57880" source_ref="" user=XXXXX user_id=XXXXX
time="2025-04-29T20:54:20Z" level=debug msg="creating client for region" func="pkg/block/s3.(*ClientCache).Get" file="build/pkg/block/s3/client_cache.go:103" host=lakefs.mydomain.com method=POST operation_id=CreateRepository path=/api/v1/repositories region="" request_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx service_name=rest_api user=XXXXX
time="2025-04-29T20:54:20Z" level=error msg="failed to get S3 object bucket wordcount key dummy" func="pkg/logging.(*logrusEntryWrapper).Errorf" file="build/pkg/logging/logger.go:338" error="operation error S3: GetObject, https response error StatusCode: 400, RequestID: 7bdd4cf4-9fbf-4020-a137-60cb3d10665c, HostID: R0Fnpio5hmW, api error AuthorizationHeaderMalformed: The authorization header you provided is invalid." host=lakefs.mydomain.com method=POST operation=GetObject operation_id=CreateRepository path=/api/v1/repositories request_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx service_name=rest_api user=XXXXX
time="2025-04-29T20:54:20Z" level=warning msg="Could not access storage namespace" func="pkg/api.(*Controller).CreateRepository" file="build/pkg/api/controller.go:2004" error="operation error S3: GetObject, https response error StatusCode: 400, RequestID: 7bdd4cf4-9fbf-4020-a137-60cb3d10665c, HostID: R0Fnpio5hmW, api error AuthorizationHeaderMalformed: The authorization header you provided is invalid." reason=unknown service=api_gateway storage_namespace="s3://wordcount/"
time="2025-04-29T20:54:20Z" level=debug msg="HTTP call ended" func=net/http.HandlerFunc.ServeHTTP file="usr/local/go/src/net/http/server.go:2136" client=lakefs-webui/1.28.0 host=lakefs.mydomain.com log_audit=true method=POST operation_id=CreateRepository path=/api/v1/repositories request_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx sent_bytes=68 service_name=rest_api source_ip="XXX.XXX.XXX.XXX:57880" status_code=400 took=57616767 took_str=57.616767ms user=XXXXX

[the-Fitz]

I was able to resolve this issue by enabling virtual-host URLs in my Ozone deployment.

Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. http://bucketname.host:9878/) and the path-style addresses (eg. http://host:9878/bucketname)

By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your ozone-site.xml:

<property>
 <name>ozone.s3g.domain.name</name>
 <value>s3g.internal</value>
</property>

To come to this conclusion, I set blockstore.s3.client_log_request: true to gather additional logs and discovered that my lakeFS deployment was configured to use virtual-host style URLs (see documentation here).

In essence, lakeFS was targeting my ozone endpoint as bucket.ozone.mydomain.com/key when my Ozone application was expecting (by default) to be found at it's path-based url: ozone.mydomain.com/bucket/key