S3Proxy auth and service path configuration works locally but fails in Kubernetes

[ferjm]

I am trying to run multiple instances of S3Proxy, each configured to handle requests on distinct paths such as https://s3-proxy.myservice.com/s3proxy-N. To achieve this, I am using the S3PROXY_SERVICE_PATH environment variable.

When running S3Proxy locally using Docker with the same auth credentials (but no service path as it all S3Proxy instances run on the same host, but different port), everything works as expected. However, when deploying S3Proxy remotely in a Kubernetes environment, with the different path approach, the instances fail to function as intended, consistently returning 403 AccessDenied responses. If I disable authentication, all works fine.

I get this output:

[s3proxy] I 11-18 17:14:36.991 main o.g.s.CrossOriginResourceSharing:113 |::] CORS allowed origins: [*]
[s3proxy] I 11-18 17:14:36.994 main o.g.s.CrossOriginResourceSharing:114 |::] CORS allowed methods: [PUT, POST]
[s3proxy] I 11-18 17:14:36.994 main o.g.s.CrossOriginResourceSharing:115 |::] CORS allowed headers: [*]
[s3proxy] I 11-18 17:14:36.994 main o.g.s.CrossOriginResourceSharing:116 |::] CORS exposed headers: []
[s3proxy] I 11-18 17:14:36.994 main o.g.s.CrossOriginResourceSharing:117 |::] CORS allow credentials:
[s3proxy] I 11-18 17:14:37.509 main o.g.s.o.e.jetty.server.Server:384 |::] jetty-11.0.22; built: 2024-06-27T16:27:26.756Z; git: e711d4c7040cb1e61aa68cb248fa7280b734a3bb; jvm 17.0.12+7
[s3proxy] I 11-18 17:14:37.615 main o.g.s.o.e.j.s.AbstractConnector:376 |::] Started ServerConnector@748fe51d{HTTP/1.1, (http/1.1)}{0.0.0.0:4449}
[s3proxy] I 11-18 17:14:37.707 main o.g.s.o.e.jetty.server.Server:439 |::] Started Server@eb507b9{STARTING}[11.0.22,sto=0] @5939ms
[s3proxy] D 11-18 17:14:46.707 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:300 |::] request: Request(GET http://10.1.251.89:4449/)@1dcf62c3
[s3proxy] T 11-18 17:14:46.709 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: Accept: */*
[s3proxy] T 11-18 17:14:46.709 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: User-Agent: kube-probe/1.29+
[s3proxy] T 11-18 17:14:46.709 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: Connection: close
[s3proxy] T 11-18 17:14:46.709 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: Host: 10.1.251.89:4449
[s3proxy] D 11-18 17:14:46.711 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:2980 |::] sendSimpleErrorResponse: 403 AccessDenied Forbidden {}

I am setting these env vars:

S3PROXY_ENDPOINT=http://0.0.0.0:4449
S3PROXY_IDENTITY=studioazuretests
S3PROXY_CREDENTIAL=***REDACTED***
S3PROXY_AUTHORIZATION=aws-v2-or-v4
S3PROXY_SERVICE_PATH=/s3proxy-N
S3PROXY_CORS_ALLOW_HEADERS=*
S3PROXY_CORS_ALLOW_METHODS=PUT POST
S3PROXY_CORS_ALLOW_ORIGINS=*
S3PROXY_IGNORE_UNKNOWN_HEADERS=true
JCLOUDS_PROVIDER=azureblob
JCLOUDS_ENDPOINT=https://studioazuretests.blob.core.windows.net
JCLOUDS_IDENTITY=studioazuretests
JCLOUDS_CREDENTIAL=***REDACTED***

[ferjm]

Some progress here.

I figured the GET http://10.1.251.89:4449 request was k8s health check, which lacks the required authorization header. Moving to a port check fixed that part.

Now I am stuck on the service path part. I found out that if I serve S3Proxy without the service path (i.e. https://s3-proxy.myservice.com), authorization works. However, if I serve S3Proxy on paths like https://s3-proxy.myservice.com/s3proxy-N, authorization does not work, and I get a signature mismatch error.

[s3proxy] D 11-19 16:56:46.290 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:300 |::] request: Request(GET http://s3-proxy.myservice.com/)@40d62122
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: X-Amz-Date: 20241119T165646Z
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: X-Request-ID: e71502f6dff24078bc904157569cf354
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: X-Forwarded-Host: s3-proxy.myservice.com
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: X-Forwarded-Proto: https
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: User-Agent: aws-cli/2.18.2 md/awscrt#0.21.2 ua/2.0 os/macos#23.6.0 md/arch#arm64 lang/python#3.12.7 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#off md/command#s3.ls
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: Host: s3-proxy.myservice.com
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: Accept-Encoding: identity
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: X-Forwarded-Port: 443
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: X-Forwarded-For: 10.1.240.39
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: X-Real-IP: 10.1.240.39
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: X-Forwarded-Scheme: https
[s3proxy] T 11-19 16:56:46.291 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:325 |::] header: X-Scheme: https
[s3proxy] D 11-19 16:56:46.292 S3Proxy-Jetty-20 o.gaul.s3proxy.S3ProxyHandler:2980 |::] sendSimpleErrorResponse: 403 SignatureDoesNotMatch Forbidden {}

Note that from these logs it seems that the GET request is done to http://s3-proxy.myservice.com/ while I would expect it to be done to http://s3-proxy.myservice.com/s3proxy-7, considering that I have this env var set S3PROXY_SERVICE_PATH : /s3proxy-7.

Am I misinterpreting the use of service path? @gaul has service path been tested with authorization enabled? Thanks!

[immusk]

@gaul do we have any update on this, we are even facing the same issue.
trying to expose via kubernetes and getting the signature mismatch.

error: Failed to bind to xxxxxx/xx.xx.xx.xx:443
when setting directly the endpoint

[re1709]

Working fix

#815