feat: S3 Express support #1906

jbelkins · 2025-03-14T22:08:54Z

Description of changes

Provides customizations needed to utilize the S3 Express feature.

Companion S3 PR:
Includes integration tests:

Test that S3 Express client config features select the correct signing mode
Test of GetObject & PutObject presigned S3 Express URLs.
Test of several S3 operations on S3 Express-enabled buckets.

New/existing dependencies impact assessment, if applicable

No new dependencies were added to this change.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

...ore/AWSSDKCommon/Sources/AWSSDKCommon/FileBasedConfiguration/CRTFileBasedConfiguration.swift

jbelkins · 2025-03-14T22:15:46Z

Sources/Core/AWSSDKHTTPAuth/Sources/AWSSDKHTTPAuth/AWSSigV4Signer.swift

@@ -49,7 +50,7 @@ public class AWSSigV4Signer: SmithyHTTPAuthAPI.Signer {
            )
        }

-        guard let identity = identity as? AWSCredentialIdentity else {
+        guard let identity = (identity as? AWSCredentialIdentity) ?? (identity as? S3ExpressIdentity)?.awsCredentialIdentity else {


When the CRT signer signs in S3 Express mode, it intakes the S3 Express credentials using the same Credentials type as with any other signing operation. So, when an S3ExpressIdentity is supplied, convert it to an AWS credential identity to pass into CRT (see private helper method below.)

jbelkins · 2025-03-14T22:17:05Z

Sources/Core/AWSSDKHTTPAuth/Sources/AWSSDKHTTPAuth/SigV4S3ExpressAuthScheme.swift

+import protocol SmithyHTTPAuthAPI.Signer
+import struct Smithy.Attributes
+
+public struct SigV4S3ExpressAuthScheme: AuthScheme {


This AuthScheme for S3 Express is basically the sigv4 auth scheme, but with a different scheme ID & signing algorithm.

jbelkins · 2025-03-14T22:18:38Z

Sources/Core/AWSSDKIdentity/Sources/AWSSDKIdentity/AWSIdentityPropertyKeys.swift

+
+import struct Smithy.AttributeKey
+
+public enum AWSIdentityPropertyKeys {


These keys are used to pass data & components into identity resolvers in a type-safe manner.

jbelkins · 2025-03-14T22:23:17Z

Sources/Core/AWSSDKIdentity/Sources/AWSSDKIdentity/S3Express/S3ExpressCreateSessionClient.swift

+        clientConfig: DefaultClientConfiguration,
+        bucket: String
+    ) async throws -> S3ExpressIdentity
+}


This protocol provides an interface for the S3 Express identity resolver to call S3 CreateSession to get credentials, without depending on the code-generated S3 client.

A SwiftIntegration on the S3 service provides conformance with this protocol.

jbelkins · 2025-03-14T22:24:05Z

Sources/Core/AWSSDKIdentity/Sources/AWSSDKIdentity/S3Express/S3ExpressIdentity.swift

+
+import protocol SmithyIdentityAPI.Identity
+import struct Foundation.Date
+


A type to hold a S3 Express identity.

Though this type is more or less the same as AWSCredentialIdentity (minus only the account ID), a separate type is used for S3 Express credentials because the two credential types are not interchangeable, and separate types helps make clear which credentials are being provided.

jbelkins · 2025-03-14T22:25:24Z

Sources/Core/AWSSDKIdentity/Sources/AWSSDKIdentity/S3Express/S3ExpressIdentityResolver.swift

+import struct Smithy.Attributes
+import protocol SmithyIdentityAPI.IdentityResolver
+
+public protocol S3ExpressIdentityResolver: IdentityResolver where IdentityT == S3ExpressIdentity {}


A convenience declaration for the type of an S3 Express-specialized identity resolver.

jbelkins · 2025-03-14T22:27:47Z

Sources/Services/AWSS3/Sources/AWSS3/AuthSchemeResolver.swift

+                    authOption.signingProperties.set(key: SmithyHTTPAuthAPI.SigningPropertyKeys.signingRegion, value: param.signingRegion)
+                    authOption.identityProperties.set(key: AWSSDKIdentity.AWSIdentityPropertyKeys.bucket, value: serviceParams.bucket)
+                    authOption.identityProperties.set(key: AWSSDKIdentity.AWSIdentityPropertyKeys.s3ExpressClient, value: S3ExpressCreateSessionClient())
+                    validAuthOptions.append(authOption)


[Note: this is generated code]

An auth option is added for S3 Express. The bucket name and a S3ExpressCreateSessionClient are added to identity properties for use by the S3 Express identity resolver.

S3ExpressCreateSessionClient, which is added into identity properties above, is a wrapper around the S3 service client that allows runtime code to perform the S3 CreateSession operation for the purpose of obtaining S3 Express credentials, while preventing runtime from having to depend on the S3 client directly.

jbelkins · 2025-03-14T22:34:16Z

Sources/Services/AWSS3/Sources/AWSS3/Models.swift

@@ -22522,6 +22523,7 @@ extension GetObjectInput {
                      .withExpiration(value: expiration)
                      .withIdentityResolver(value: config.awsCredentialIdentityResolver, schemeID: "aws.auth#sigv4")
                      .withIdentityResolver(value: config.awsCredentialIdentityResolver, schemeID: "aws.auth#sigv4a")
+                      .withIdentityResolver(value: config.s3ExpressIdentityResolver, schemeID: "aws.auth#sigv4-s3express")


[Note: this is generated code]

For this & several other presigned URLs:

The client config is added to context so it can be used for the S3 CreateSession operation to get credentials later on.

The S3 Express identity resolver is added to the operation.

jbelkins · 2025-03-14T22:35:28Z

Sources/Services/AWSS3/Sources/AWSS3/Plugins.swift

@@ -44,7 +45,7 @@ public class DefaultAWSAuthSchemePlugin: ClientRuntime.Plugin {
    public func configureClient(clientConfiguration: ClientRuntime.ClientConfiguration) throws {
        if let config = clientConfiguration as? S3Client.S3ClientConfiguration {
            config.authSchemeResolver = DefaultS3AuthSchemeResolver()
-            config.authSchemes = [AWSSDKHTTPAuth.SigV4AuthScheme(), AWSSDKHTTPAuth.SigV4AAuthScheme()]
+            config.authSchemes = [AWSSDKHTTPAuth.SigV4AuthScheme(), AWSSDKHTTPAuth.SigV4AAuthScheme(), AWSSDKHTTPAuth.SigV4S3ExpressAuthScheme()]


[Note: this is generated code]

SigV4S3ExpressAuthScheme is added to the service's auth schemes

jbelkins · 2025-03-14T22:36:52Z

...y-aws-swift-codegen/src/main/kotlin/software/amazon/smithy/aws/swift/codegen/AWSAuthUtils.kt

-
+        if (ctx.service.isS3) {
+            updatedAuthSchemeList += writer.format("\$N()", AWSSDKHTTPAuthTypes.SigV4S3ExpressAuthScheme)
+        }


SigV4S3ExpressAuthScheme is added, but only to the S3 service client.

jbelkins · 2025-03-14T22:37:34Z

...en/src/main/kotlin/software/amazon/smithy/aws/swift/codegen/AWSHTTPProtocolCustomizations.kt

@@ -36,6 +37,9 @@ abstract class AWSHTTPProtocolCustomizations : DefaultHTTPProtocolCustomizations
        }
        writer.write("  .withIdentityResolver(value: config.awsCredentialIdentityResolver, schemeID: \$S)", "aws.auth#sigv4")
        writer.write("  .withIdentityResolver(value: config.awsCredentialIdentityResolver, schemeID: \$S)", "aws.auth#sigv4a")
+        if (ctx.service.isS3) {
+            writer.write("  .withIdentityResolver(value: config.s3ExpressIdentityResolver, schemeID: \$S)", "aws.auth#sigv4-s3express")
+        }


S3 Express identity resolver is added to the context for an operation, but only to the S3 service client.

jbelkins · 2025-03-14T22:39:01Z

...tware/amazon/smithy/aws/swift/codegen/customization/RulesBasedAuthSchemeResolverGenerator.kt

+                            )
+                            write("validAuthOptions.append(authOption)")
+                            dedent()
+                        }


In the S3 rules-based auth scheme resolver only, the S3 Express auth option is added.

(See the generated code above for more discussion of the content here.)

jbelkins · 2025-03-14T22:41:43Z

...ain/kotlin/software/amazon/smithy/aws/swift/codegen/customization/s3/S3ExpressIntegration.kt

+        ctx: SwiftCodegenContext,
+        protocolGenerationContext: ProtocolGenerator.GenerationContext,
+        delegator: SwiftDelegator
+    ) {


This method generates a file that creates a concrete implementation of the S3ExpressCreateSessionClient protocol.

An instance of this class is passed into the S3 credential provider, and allows that credential provider to call S3 without actually depending on S3.

See the generated file above:
https://github.com/awslabs/aws-sdk-swift/pull/1906/files/fec058fa0f5518470dcf1479b5b38365db683302#diff-9eaf9732200d9284b72d8a77b8d86ee2076509f03417f0ba54ad5434b3258d15

jbelkins · 2025-06-13T21:41:55Z

Sources/Services/AWSS3/Package.swift.txt

+        .package(
+            id: "aws-sdk-swift.AWSSDKIdentity",
+            exact: "0.0.1"
+        ),


Here & below, the generated AWSS3 service client is shown. There are no changes in any other AWS service client.

jbelkins · 2025-06-13T21:53:24Z

Sources/Services/AWSS3/Sources/AWSS3/S3Client+S3Express.swift

+import protocol ClientRuntime.DefaultClientConfiguration
+import struct AWSSDKIdentity.S3ExpressIdentity
+
+


This is a code-generated, concrete implementation of the AWSSDKIdentity.S3ExpressCreateSessionClient protocol.

This serves as a "dependency inversion" wrapper to allow AWSSDKIdentity to call the S3 client without depending on it.

The method below takes a DefaultClientConfiguration instead of the concrete S3 configuration type, then it casts it to S3 config. Because this wrapper is only ever generated for S3, the cast to S3Client.Config should never fail.

jbelkins · 2025-06-13T21:56:47Z

...tware/amazon/smithy/aws/swift/codegen/customization/RulesBasedAuthSchemeResolverGenerator.kt

@@ -125,6 +128,34 @@ class RulesBasedAuthSchemeResolverGenerator {
                        )
                        write("validAuthOptions.append(sigV4Option)")
                        dedent()


When S3 Express is used as the auth scheme, all the identity properties below need to be set so the S3 Express identity resolver has what it needs to get S3 Express credentials.

sichanyoo

Some Qs

...gen/src/main/kotlin/software/amazon/smithy/aws/swift/codegen/AWSHttpProtocolServiceClient.kt

...ity/Sources/AWSSDKIdentity/S3Express/IdentityResolver/DefaultS3ExpressIdentityResolver.swift

sichanyoo · 2025-06-17T17:52:33Z

...WSSDKIdentity/Sources/AWSSDKIdentity/S3Express/IdentityResolver/S3ExpressCachedElement.swift

+    private func scheduleRefreshTask() {
+        // Cancel any previous refresh task
+        self.refreshTask?.cancel()
+
+        // Task captures self weakly, to prevent refreshing a cache that has been purged.
+        // self is then bound strongly both before & after the credential refresh wait.
+        self.refreshTask = Task { [weak self] in
+            guard let selfBeforeWait = self else { return }
+            let expiration: Date
+            do {
+                expiration = try await selfBeforeWait.retrieveTask.value.expiration ?? Date.distantPast
+            } catch {
+                // If there was an error retrieving credentials, throw it to the caller
+                // and remove this cached element.
+                // A subsequent access to credentials will create a new cached element.
+                await selfBeforeWait.removeSelfFromCache()
+                throw error
+            }
+
+            // Asynchronously wait until the credentials are within buffer time of expiring.
+            // Use the modern `ContinuousClock` time measurement if available.
+            let now = Date()
+            if #available(macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0, *) {
+                let interval = expiration.timeIntervalSince(now) - Self.buffer
+                let duration = Duration(secondsComponent: Int64(interval), attosecondsComponent: 0)
+                let clock = ContinuousClock.now.advanced(by: duration)
+                let tolerance = Duration(secondsComponent: 1, attosecondsComponent: 0)
+                try await Task.sleep(until: clock, tolerance: tolerance, clock: .continuous)
+            } else {
+                let interval = expiration.timeIntervalSince(now) - Self.buffer
+                try await Task.sleep(nanoseconds: UInt64(interval) * 1_000_000_000)
+            }
+
+            // wait has now passed.  Perform the credential refresh if credentials have been accessed,
+            // else this cached element will be purged for not being used.
+            guard !Task.isCancelled, let selfAfterWait = self else { return }
+            await selfAfterWait.performRefreshIfNeeded()
+        }
+    }
+
+    private func performRefreshIfNeeded() async {
+        switch status {
+        case .initial:
+            // This is the very first access to these credentials.
+            // If there isn't another access before refresh, these credentials will get purged.
+            status = .unaccessed
+        case .unaccessed:
+            // Credential hasn't been accessed since initial access.
+            // Purge this cached element from the cache.
+            await removeSelfFromCache()
+        case .accessed:
+            // Credential has been accessed since last refresh
+            // Refresh again.
+            self.status = .unaccessed
+            retrieveTask = Self.newRetrieveTask(identityProperties: identityProperties)
+            scheduleRefreshTask()
+        }


Q: What was the reason a simpler approach (where credentials themselves are directly cached & refreshed each time it's accessed if it's within refresh buffer window) was not taken?

Answered offline; because specifications require us to refresh credentials automatically in the background to avoid having to refresh credential at request time. Doing so reduces duration of API call.

As discussed verbally, S3 Express credentials are refreshed on a timer, prior to expiration, so that fresh credentials are always available, preventing a S3 Express request from having to wait for credential refresh before it can proceed.

jbelkins added 2 commits March 14, 2025 17:02

Add S3 codegen

76b7286

Runtime & codegen changes

fec058f

jbelkins commented Mar 14, 2025

View reviewed changes

...ore/AWSSDKCommon/Sources/AWSSDKCommon/FileBasedConfiguration/CRTFileBasedConfiguration.swift Outdated Show resolved Hide resolved

jbelkins commented Mar 14, 2025

View reviewed changes

jbelkins added 14 commits March 14, 2025 17:58

Fix codegen tests

9bf1e9f

Fix ktlint

6171042

fix Swiftlint

91d5131

Merge branch 'main' into jbe/s3_express

1c23aeb

Merge branch 'main' into jbe/s3_express

75d558a

Add env var & profile config for disableS3ExpressSessionAuth

f1c87c1

Merge branch 'main' into jbe/s3_express

a451df5

Merge branch 'main' into jbe/s3_express

460a79a

Merge branch 'main' into jbe/s3_express

5454f00

Merge branch 'main' into jbe/s3_express

6e7e78e

Merge remote-tracking branch 'origin/main' into jbe/s3_express

d575d33

Merge branch 'main' into jbe/s3_express

47c4149

Merge branch 'main' into jbe/s3_express

50cff85

Merge branch 'main' into jbe/s3_express

d988d00

jbelkins and others added 14 commits June 5, 2025 13:10

Fix lint, provide error types

b629602

Add S3 codegen changes

837860e

Merge branch 'main' into jbe/s3_express

3ba7b6e

Fix ktlint & codegen tests

fa00696

Merge remote-tracking branch 'origin/main' into jbe/s3_express

39ba60e

Merge remote-tracking branch 'origin/jbe/s3_express' into jbe/s3_express

4159fa0

Merge branch 'main' into jbe/s3_express

d947a97

Added integration tests

c03e6b2

Merge branch 'main' into jbe/s3_express

75a2d96

Add comments, fix Linux tests

477782a

Merge branch 'main' into jbe/s3_express

1495354

Fix Swiftlint

9fa6047

Fix integration tests on Linux/Swift 5.9

16d2404

Unwrap client config before use

0a0f9f9

jbelkins commented Jun 13, 2025

View reviewed changes

jbelkins added 2 commits June 16, 2025 10:35

Revert CRTFileBasedConfiguration change

243c5dd

Merge branch 'main' into jbe/s3_express

5bb1704

jbelkins marked this pull request as ready for review June 16, 2025 15:51

jbelkins requested review from dayaffe and sichanyoo June 16, 2025 15:51

sichanyoo reviewed Jun 17, 2025

View reviewed changes

jbelkins added 6 commits June 18, 2025 15:19

Merge branch 'main' into jbe/s3_express

50fef3a

Add import to fix build

dd17b7d

Add needed dependency to AWSSDKHTTPAuth

20d60c9

Merge remote-tracking branch 'origin/main' into jbe/s3_express

cee061e

Merge branch 'main' into jbe/s3_express

41d61db

Delete unused UUID on S3 credential cache key type

561894c

sichanyoo approved these changes Jun 20, 2025

View reviewed changes


		import struct Smithy.AttributeKey

		public enum AWSIdentityPropertyKeys {


		import protocol SmithyIdentityAPI.Identity
		import struct Foundation.Date

		import protocol ClientRuntime.DefaultClientConfiguration
		import struct AWSSDKIdentity.S3ExpressIdentity

feat: S3 Express support #1906

Are you sure you want to change the base?

feat: S3 Express support #1906

Uh oh!

Conversation

jbelkins commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

New/existing dependencies impact assessment, if applicable

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sichanyoo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jbelkins commented Mar 14, 2025 •

edited

Loading