Credentials rotation causes `BackupBucket` issues

[timuthy]

How to categorize this issue?

/area control-plane
/kind bug
/platform azure

What happened:
For some of our seed clusters we experienced failing BackupBuckets after the Azure credentials were rolled.

The reconciliation failed with the following error:

{"log":{"controller":"backupbucket","error":"failed to ensure the resource group and storage account: PUT https://management.azure.com/subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.Storage/storageAccounts/REDACTED\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: NoKeyCreationTime\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"NoKeyCreationTime\",\n    \"message\": \"No KeyCreationTime for key: key1, please regenerate that key\"\n  }\n}\n--------------------------------------------------------------------------------\n","level":"error","msg":"Reconciler error","name":"REDACTED","namespace":"","reconcileID":"449c0292-aed7-439e-8041-a58118448d50","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249","ts":"2

Only after manually rotating the storage account secrets in the Azure portal were the reconciliation errors resolved.

What you expected to happen:
The reconciliation to succeed.

Anything else we need to know?:
The credential rotation happened for multiple seeds whereas only a handful was affected by the described issue. Hence, the exact root cause is still unclear.

Environment:

• Gardener version (if relevant):
• Extension version: v1.52.0
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others:

[kon-angelo]

I think that this is a documentation issue more than anything else. The issue that occurred in our production seeds affected only relatively old seeds.

The reason is that originally Azure did not keep the creation time of keys and eventually enabled it at some point. This created an inconsistency that some keys do have a creation date and some are not. While the credential rotation works regardless of the creation timestamp, enabling the credential expiration date does not work and it is what was validated against by the Azure API here. In our production, I believe we enabled them both at the same time.

We can either add a safeguard, but because we cannot test it anymore reliably (you cannot create a storage account without key timestamps anymore), it is probably better to make a comment in the documentation that to enable the credential expiration only if the keys have been rotated at least once.