PodidentityAssociation ready state is false

[ashishvaishno]

Hello,

I have upgrade my crossplane deployment to xpkg.upbound.io/upbound/provider-aws-eks:v1.0.0. I have started to use Podidentityassociation.

Crossplane was successfully able to create the association and synced, but the ready state is false 😕 Below is the error of the managed resource. I dont have any previous managed resource / already existing association with an association id a-stubassocid123456

async create failed: resource creation call returned error diags: creating Amazon EKS (Elastic Kubernetes) Pod Identity Association ("a-stubassocid123456"): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: f9cb61ba-2e55-4c0e-a8f7-962bfd8302bc, ResourceInUseException: Association already exists: a-dt1bedfjc72dlwp5n: operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: f9cb61ba-2e55-4c0e-a8f7-962bfd8302bc, ResourceInUseException: Association already exists: a-dt1bedfjc72dlwp5n

[candonov]

Hi @ashishvaishno, can you provide the Crossplane manifest(s) you used so I reproduce this?

[ashishvaishno]

@candonov Sure. But the issue persists only on few PIA. I tried to clean them up and re-deploy them as well. But its still the same 😢
Composition

# XRD to create the s3 bucket and the associated irsa
---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xs3withpodidentitys.crossplane.xyz.com
spec:
  claimNames:
    kind: S3WithPodIdentity
    plural: s3withpodidentitys
  group: crossplane.xyz.com
  names:
    kind: XS3WithPodIdentity
    plural: xs3withpodidentitys
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          properties:
            spec:
              description: S3WithPodIdentitySpec defines the desired state of S3WithPodIdentity
              properties:
                resourceConfig:
                  description: ResourceConfig defines general properties of this AWS resource.
                  properties:
                    deletionPolicy:
                      description: Defaults to Delete
                      enum:
                      - Orphan
                      type: string
                      default: "Orphan"
                    bucketName:
                      description: Name of the bucket
                      type: string
                    serviceAccountName:
                      description: Name of the service account which will also be used to generate the k8s sa
                      type: string
                    serviceNamespace:
                      description: Namespace of the service where its running
                      type: string
                    bucketVersioningStatus:
                      description: Bucket versioning status
                      type: string
                      default: "Disabled"
                      enum:
                      - Disabled
                      - Suspended
                      - Enabled
                    bucketNonCurrentVersionExpirationDays:
                      description: Cleanup Old Objects older than x days
                      type: number
                      default: 90                     
                    bucketCleanupOldObjectStatus:
                      description: Cleanup Old Objects
                      type: string
                      default: "Disabled"
                      enum:
                      - Disabled
                      - Enabled
                    bucketCleanupOldObjectExpirationDays:
                      description: Cleanup Old Objects older than x days
                      type: number
                      default: 180
                    awsBackup:
                      description: Setup awsbackup to exclude for s3 buckets
                      type: string
                      default: "exclude"
                    tags:
                      items:
                        properties:
                          key:
                            type: string
                          value:
                            type: string
                        required:
                        - key
                        - value
                        type: object
                      type: array
                  required:
                  - bucketName
                  - serviceAccountName
                  - serviceNamespace
                  type: object
              required:
              - resourceConfig
              type: object
            status:
              description: S3WithPodIdentityStatus defines the observed state of S3WithPodIdentity
              properties:
                bucketArn:
                  type: string
                bucketVersioningStatus:
                  type: string
                bucketNonCurrentVersionExpirationDays:
                  type: number
                bucketCleanupOldObjectStatus:
                  type: string
                bucketCleanupOldObjectExpirationDays:
                  type: number
                roleArn:
                  type: string
                awsAccountID:
                  type: string
                policyArn:
                  type: string
              type: object
          type: object

XRD :

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: dev.xs3withpodidentity.crossplane.xyz.com
  labels:
    environment: dev
spec:
  compositeTypeRef:
    apiVersion: crossplane.xyz.com/v1alpha1
    kind: XS3WithPodIdentity
  patchSets:
    - name: deletion-policy-field
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.deletionPolicy
          toFieldPath: spec.deletionPolicy
    - name: common-fields
      patches:
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
            strategy: string
            string:
              fmt: "aws-provider-config-%s"
          toFieldPath: spec.providerConfigRef.name
    - name: tags
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
          toFieldPath: spec.forProvider.tags.Environment
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceAccountName
          toFieldPath: spec.forProvider.tags.serviceAccountName
    - name: region
      patches:          
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
          toFieldPath: spec.forProvider.region
          transforms:
            - type: map
              map:
                dev: eu-central-1
                stage: eu-west-1
                prod: eu-west-1
    - name: s3-extra-tag
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.awsBackup
          toFieldPath: spec.forProvider.tags.AWSBackup
  resources:
    - name: s3-bucket
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: Bucket
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: deletion-policy-field
        - type: PatchSet
          patchSetName: tags
        - type: PatchSet
          patchSetName: region
        - type: PatchSet
          patchSetName: s3-extra-tag
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketName
          toFieldPath: metadata.annotations[crossplane.io/external-name]
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketName
          toFieldPath: spec.forProvider.tags.Name
        - type: ToCompositeFieldPath
          fromFieldPath: status.atProvider.arn
          toFieldPath: status.bucketArn
    - name: s3-bucketpublicaccessblock
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: BucketPublicAccessBlock
        spec:
          forProvider:
            blockPublicAcls: true
            blockPublicPolicy: true
            ignorePublicAcls: true
            restrictPublicBuckets: true
            bucketSelector:
              matchControllerRef: true
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: deletion-policy-field
        - type: PatchSet
          patchSetName: region
    - name: s3-bucketserversideencryptionconfiguration
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: BucketServerSideEncryptionConfiguration
        spec:
          forProvider:
            bucketSelector:
              matchControllerRef: true
            rule:
              - applyServerSideEncryptionByDefault:
                  - sseAlgorithm: AES256
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: deletion-policy-field
        - type: PatchSet
          patchSetName: region
    - name: s3-bucketversioning
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: BucketVersioning
        spec:
          forProvider:
            bucketSelector:
              matchControllerRef: true
            versioningConfiguration:
              - status: ""
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: deletion-policy-field
        - type: PatchSet
          patchSetName: region
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketVersioningStatus
          toFieldPath: spec.forProvider.versioningConfiguration[0].status
        - type: ToCompositeFieldPath
          fromFieldPath: status.atProvider.versioningConfiguration[0].status
          toFieldPath: status.bucketVersioningStatus
    - name: s3-bucketlifecycleconfiguration
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: BucketLifecycleConfiguration
        spec:
          forProvider:
            bucketSelector:
              matchControllerRef: true
            rule:
              - id: "Cleanup of Non Current Versions"
                noncurrentVersionExpiration:
                  - noncurrentDays: 
                status: ""
              - id: "Cleanup of Old Objects"
                expiration:
                  - days: 
                status: ""
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: deletion-policy-field
        - type: PatchSet
          patchSetName: region
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketVersioningStatus
          toFieldPath: spec.forProvider.rule[0].status
          transforms:
            - type: map
              map:
                Disabled: Disabled
                Suspended: Enabled
                Enabled: Enabled
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketNonCurrentVersionExpirationDays
          toFieldPath: spec.forProvider.rule[0].noncurrentVersionExpiration[0].noncurrentDays
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketCleanupOldObjectStatus
          toFieldPath: spec.forProvider.rule[1].status
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketCleanupOldObjectExpirationDays
          toFieldPath: spec.forProvider.rule[1].expiration[0].days
        - type: ToCompositeFieldPath
          fromFieldPath: spec.forProvider.rule[0].status
          toFieldPath: status.bucketVersioningStatus
        - type: ToCompositeFieldPath
          fromFieldPath: spec.forProvider.rule[0].noncurrentVersionExpiration[0].noncurrentDays
          toFieldPath: status.bucketNonCurrentVersionExpirationDays
        - type: ToCompositeFieldPath
          fromFieldPath: spec.forProvider.rule[1].status
          toFieldPath: status.bucketCleanupOldObjectStatus
        - type: ToCompositeFieldPath
          fromFieldPath: spec.forProvider.rule[1].expiration[0].days
          toFieldPath: status.bucketCleanupOldObjectExpirationDays
    - name: iamrole
      base:
        apiVersion: iam.aws.upbound.io/v1beta1
        kind: Role
        spec:
          deletionPolicy: Delete
          forProvider:
            assumeRolePolicy: |
              {
                "Version": "2012-10-17",
                "Statement": [
                  {
                    "Effect": "Allow",
                    "Principal": {
                      "Service": "pods.eks.amazonaws.com"
                    },
                    "Action": [
                      "sts:TagSession",
                      "sts:AssumeRole"
                    ]
                  }
                ]
              }
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: tags
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
            - fromFieldPath: spec.resourceConfig.serviceAccountName
            strategy: string
            string:
              fmt: "k8s-%s-%s"
          toFieldPath: metadata.annotations[crossplane.io/external-name]
        - type: ToCompositeFieldPath
          fromFieldPath: status.atProvider.arn
          toFieldPath: status.roleArn
    - name: iam-policy
      base: 
        apiVersion: iam.aws.upbound.io/v1beta1
        kind: Policy
        spec:
          deletionPolicy: Delete
          forProvider:
            description: ""
            policy: ""
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: tags
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
            - fromFieldPath: spec.resourceConfig.serviceAccountName
            strategy: string
            string:
              fmt: "k8s-%s-%s"
          toFieldPath: metadata.annotations[crossplane.io/external-name]
        - type: ToCompositeFieldPath
          fromFieldPath: status.atProvider.arn
          toFieldPath: status.policyArn
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: spec.resourceConfig.bucketName
            strategy: string
            string:
              fmt: "IAM Policy for bucket %s created by crossplane"
          toFieldPath: spec.forProvider.description
        - type: CombineFromComposite
          toFieldPath: spec.forProvider.policy
          policy:
            fromFieldPath: Required
          combine:
            variables:
            - fromFieldPath: spec.resourceConfig.bucketName
            - fromFieldPath: spec.resourceConfig.serviceAccountName
            - fromFieldPath: spec.resourceConfig.bucketName
            - fromFieldPath: spec.resourceConfig.serviceAccountName
            strategy: string
            string:
              fmt: |
                { 
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Action": "s3:ListBucket",
                      "Effect": "Allow",
                      "Resource": [
                          "arn:aws:s3:::%s"
                      ],
                      "Condition": {
                        "StringLike": {
                          "aws:PrincipalTag/kubernetes-pod-name": "%s*"
                        }
                      }
                    },
                    {
                      "Action": [
                        "s3:ListObjects",
                        "s3:PutObject",
                        "s3:GetObject",
                        "s3:DeleteObject"
                      ],
                      "Effect": "Allow",
                      "Resource": [
                          "arn:aws:s3:::%s/*"
                      ],
                      "Condition": {
                        "StringLike": {
                          "aws:PrincipalTag/kubernetes-pod-name": "%s*"
                        }
                      }
                    }
                  ]
                }
    - name: policy-attachment-1
      base:
        apiVersion: iam.aws.upbound.io/v1beta1
        kind: RolePolicyAttachment
        spec:
          deletionPolicy: Delete
          forProvider:
            policyArnSelector:
              matchControllerRef: true
            roleSelector:
              matchControllerRef: true
      patches:
        - type: PatchSet
          patchSetName: common-fields
    - name: pod-identity-association
      base:
        apiVersion: eks.aws.upbound.io/v1beta1
        kind: PodIdentityAssociation
        spec:
          deletionPolicy: Delete
          forProvider:
            clusterName: "go-main-dev"
            region: "eu-central-1"
            namespace: ""
            serviceAccount: ""
            roleArnSelector:
              matchControllerRef: true
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceNamespace
          toFieldPath: spec.forProvider.namespace
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceAccountName
          toFieldPath: spec.forProvider.serviceAccount
    - name: service-account
      base:
        apiVersion: kubernetes.crossplane.io/v1alpha1
        kind: Object
        spec:
          deletionPolicy: Delete
          forProvider:
            manifest:
              apiVersion: v1
              kind: ServiceAccount
              metadata:
                name: ""
                namespace: ""
              imagePullSecrets:
                - name: registrypullsecret
      patches:
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
            strategy: string
            string:
              fmt: "kubernetes-provider-config-%s"
          toFieldPath: spec.providerConfigRef.name
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceNamespace
          toFieldPath: spec.forProvider.manifest.metadata.namespace
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceAccountName
          toFieldPath: spec.forProvider.manifest.metadata.name
    - name: remote-configmap
      base:
        apiVersion: kubernetes.crossplane.io/v1alpha1
        kind: Object
        spec:
          deletionPolicy: Delete
          forProvider:
            manifest:
              apiVersion: v1
              kind: ConfigMap
              metadata:
                name: ""
                namespace: ""
              data:
                roleArn: ""
                policyArn: ""
                bucketArn: ""
                serviceAccountName: ""
                region: ""
                bucketVersioningStatus: ""
      patches:
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
            strategy: string
            string:
              fmt: "kubernetes-provider-config-%s"
          toFieldPath: spec.providerConfigRef.name
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceNamespace
          toFieldPath: spec.forProvider.manifest.metadata.namespace
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceAccountName
          toFieldPath: spec.forProvider.manifest.metadata.name
        - type: FromCompositeFieldPath
          fromFieldPath: status.roleArn
          toFieldPath: spec.forProvider.manifest.data.roleArn
        - type: FromCompositeFieldPath
          fromFieldPath: status.policyArn
          toFieldPath: spec.forProvider.manifest.data.policyArn
        - type: FromCompositeFieldPath
          fromFieldPath: status.bucketArn
          toFieldPath: spec.forProvider.manifest.data.bucketArn
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceAccountName
          toFieldPath: spec.forProvider.manifest.data.serviceAccountName
        - type: FromCompositeFieldPath
          fromFieldPath: status.bucketVersioningStatus
          toFieldPath: spec.forProvider.manifest.data.bucketVersioningStatus
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
          toFieldPath: spec.forProvider.manifest.data.region
          transforms:
            - type: map
              map:
                dev: eu-central-1
                stage: eu-west-1
                prod: eu-west-1

Claim :

---
apiVersion: crossplane.xyz.com/v1alpha1
kind: S3WithPodIdentity
metadata:
  name: test-s3with-podidentity
  namespace: dev
spec:
  compositionSelector:
    matchLabels:
      environment: dev
  resourceConfig:
    bucketName: test-s3with-podidentity
    serviceAccountName:service-account-name
    serviceNamespace: default

Error :

    Message:               async create failed: resource creation call returned error diags: creating Amazon EKS (Elastic Kubernetes) Pod Identity Association ("a-dzwb0z8xhym98oebr"): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 428d429c-75f8-4d31-bf88-cdfa1f756370, ResourceInUseException: Association already exists: a-dzwb0z8xhym98oebr: operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 428d429c-75f8-4d31-bf88-cdfa1f756370, ResourceInUseException: Association already exists: a-dzwb0z8xhym98oebr

On AWS is do see the association a-dzwb0z8xhym98oebr created by crossplane. Not sure where is the drift 🤷‍♂️

[ashishvaishno]

@candonov Were you able to test this out?

[candonov]

Hi @ashishvaishno , I tried testing but the provider names are different than the ones in this repository. I know it's been a while, did you get this working or should we dig in?

[ashishvaishno]

@candonov Sorry for the late reply. This is still an issue. It also complains for already existing resources

async create failed: resource creation call returned error diags: creating Amazon EKS (Elastic Kubernetes) Pod Identity Association ("a-wlcylmg69wzd65sp9"): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 558cbce6-04a6-4e15-bf01-a047e2c3ffb3, ResourceInUseException: Association already exists: a-wlcylmg69wzd65sp9: operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 558cbce6-04a6-4e15-bf01-a047e2c3ffb3, ResourceInUseException: Association already exists: a-wlcylmg69wzd65sp9

• Can I force crossplane reconciliation if the PIA already exists somehow?