Support access to S3 storage using Service account role

[tarangini-shetty]

Currently, while configuring the S3 as only storage for the Pulp in eks, the only way to provide access for pods to s3 is through access keys specified in a k8 secret.
I did give a try with giving necessary permissions to a service account role to access s3, and attaching that service account role arn to the pods using spec.sa_annotations option in Pulp CR . But the pulp cr always expects spec.object_storage_s3_secret. Thus wasnt able to use only the service account option (without a s3 secret) with pulpcr. Could this support be added to the Pulp CR, so that access key as secret can be avoided. Especially when the pulp deployment is taken care using git, the storing of the access keys in git is slightly challenging, and cant be fully automated.

Raising this issue, based on the discussion in https://discourse.pulpproject.org/t/how-to-install-pulp-on-aws-eks-cluster-with-s3-storage/893/34

[catalinpan]

I've tested that with v1.0.1 pulp-operator and pulp-minimal 3.76.1. It seems to work with IAM role attached to the service account.

Here is the pulp config used.

Pulp config

apiVersion: repo-manager.pulpproject.org/v1
kind: Pulp
metadata:
  name: pulp
  namespace: pulp
spec:
  object_storage_s3_secret: pulp-s3-config
  sa_annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::000000000000:role/pulp-sa-0000000000
............

Kubernetes secret

kind: Secret
apiVersion: v1
metadata:
  name: pulp-s3-config
type: Opaque
data:
  s3-bucket-name: .........
  s3-region: ........

Below is the SA created by the pulp-operator

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pulp
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::000000000000:role/pulp-sa-0000000000