Sync AWS Secrets manager with Kubernetes secrets on local not working when using minikube and localstack

[deepaktripathi1997]

I am trying to configure AWS secrets with my kubernetes cluster in such a way that I can expose secrets as environment variable in my deployment file

OS: Mac
I am using minikube for local testing
For AWS I am using localstack

secrets manager I have created a secret

I have created this yaml file

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: airflow-deployment-aws-secrets-dev
spec:
  provider: aws
  secretObjects:       # [OPTIONAL] SecretObject defines the desired state of synced K8s secret objects
  - data:
    - key: AIRFLOW__CORE__EXECUTOR                           # data field to populate
      objectName: AIRFLOW__CORE__EXECUTOR                        # name of the mounted content to sync. this could be the object 
    secretName: airflow-environment                    # name of the Kubernetes Secret object
    type: Opaque    

  parameters:
    objects: |
        - objectName: "AirflowSecretDev"
          objectType: "secretsmanager"
    region: "us-west-1"
    endpoint-url: "http://host.docker.internal:4566"

In my deployment file I have used

apiVersion: apps/v1
kind: Deployment
metadata:
  name: airflow-scheduler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: airflow-scheduler
  template:
    metadata:
      labels:
        app: airflow-scheduler
    spec:
      containers:
        - name: airflow-scheduler
          image: airflow-testing-image-1:latest
          imagePullPolicy: Never 
          command: ["/bin/bash", "-c",  "airflow db check-migrations -t 0 || airflow db upgrade || true; airflow scheduler"]
          env:
            - name: AIRFLOW__CORE__EXECUTOR
              valueFrom:
                secretKeyRef:
                  name: airflow-environment
                  key: AIRFLOW__CORE__EXECUTOR

But this whole is not working and getting following error

kubernetes logs for csi drivers

Please let me know what am I doing wrong in this .

[manojtr]

Hello all, I have the same issue. Any help here is greatly appreciated. Thank you

[stecullum]

did you install with this feature syncSecret.enabled=true in your helm install

[swt-yoromero]

Hola a todos!!!. Me paso lo mismo, tengo EKS 1.30, y lo solucione agregandole permisos al rol ya que faltaban permisos para los recursos secrets, este era el error que me daba encontrado en los logs del pod:
kubectl logs -n kube-system secrets-store-csi-driver-45w8a

secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
I0613 20:17:17.679272       1 reflector.go:424] "pkg/mod/k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
E0613 20:17:17.679334       1 reflector.go:140] "pkg/mod/k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
I0613 20:18:00.577364       1 reflector.go:424] "pkg/mod/k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"

luego de agregar estos permisos [list, watch, get] me dio este error:

I0613 20:46:34.691739       1 secretproviderclasspodstatus_controller.go:342] "The secret operation failed with forbidden error. If you installed the CSI driver using helm, ensure syncSecret.enabled=true is set.\n"
E0613 20:46:34.694549       1 secretproviderclasspodstatus_controller.go:338] "failed to create Kubernetes secret" err="secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot create resource \"secrets\" in API group \"\" in the namespace \"fincuo\"" spc="fincuo/api-funded-natives-secrets" pod="fincuo/api-funded-natives-5df4c46dc7-zgm74" secret="fincuo/api-funded-natives-secrets" spcps="fincuo/api-funded-natives-5df4c46dc7-zgm74-fincuo-api-funded-natives-secrets"

y lo que agregue al ClusterRole quedo fue:

- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

quedando asi el recurso:

# Source: secrets-store-csi-driver/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: secretproviderclasses-role
 labels:
   app.kubernetes.io/instance: "secrets-store-csi-driver"
   app.kubernetes.io/managed-by: "Helm"
   app.kubernetes.io/name: "secrets-store-csi-driver"
   app.kubernetes.io/version: "1.4.3"
   app: secrets-store-csi-driver
   helm.sh/chart: "secrets-store-csi-driver-1.4.3"
rules:
- apiGroups:
 - ""
 resources:
 - events
 verbs:
 - create
 - patch
- apiGroups:
 - ""
 resources:
 - pods
 verbs:
 - get
 - list
 - watch
- apiGroups:
 - secrets-store.csi.x-k8s.io
 resources:
 - secretproviderclasses
 verbs:
 - get
 - list
 - watch
- apiGroups:
 - secrets-store.csi.x-k8s.io
 resources:
 - secretproviderclasspodstatuses
 verbs:
 - create
 - delete
 - get
 - list
 - patch
 - update
 - watch
- apiGroups:
 - secrets-store.csi.x-k8s.io
 resources:
 - secretproviderclasspodstatuses/status
 verbs:
 - get
 - patch
 - update
- apiGroups:
 - storage.k8s.io
 resourceNames:
 - secrets-store.csi.k8s.io
 resources:
 - csidrivers
 verbs:
 - get
 - list
 - watch
- apiGroups:
 - ""
 resources:
 - secrets
 verbs:
 - create
 - delete
 - get
 - list
 - patch
 - update
 - watch