Fuse Sidecar container is stuck in PodInitialization for pods mounting two volumes backed gcs bucket

[rkhir]

Describe the issue

We are using gcs buckets with fuse driver to mount file system on k8s workloads. I have had issues getting two workloads (deployment/statefulset) to initialize while mounting two pv backed by one gcs bucket. pods simply are stuck in the PodInitialization state, and none of the sidecar containers start, including the fuse sidecar.
This has worked on one pod that is mounting one pv backed by gcs bucket.

As mentioned in GCP documentation, I tried manually injecting the sidecar container, and that didn't help to start the pod or any of the init-containers on it.

System & Version (please complete the following information):

• OS: Container-Optimized OS from Google (k8s node)
• Platform GKE, GCS
• Versions:
  -- fuse driver version: v1.5.0
  -- GKE version: v1.30.5-gke.1713000
• Node:
  -- containerRuntimeVersion: containerd://1.7.23
  -- kernelVersion: 6.1.112+
  -- kubeProxyVersion: v1.30.5-gke.1713000
  -- kubeletVersion: v1.30.5-gke.1713000
  -- operatingSystem: linux
  -- osImage: Container-Optimized OS from Google
• sidecar image: gcs-fuse-csi-driver-sidecar-mounter:v1.5.0-gke.3@sha256:ce8d3905b165220299dffda672cae32eb94cdb5d8872314bc40aeffdba5ecd76

Steps to reproduce the behavior with the following information:

• PV yaml has all mount options:

apiVersion: v1
kind: PersistentVolume
metadata:
 annotations:
   pv.kubernetes.io/bound-by-controller: "yes"
 finalizers:
 - kubernetes.io/pv-protection
 name: xxxx
spec:
 accessModes:
 - ReadWriteMany
 capacity:
   storage: 5Gi
 claimRef:
   apiVersion: v1
   kind: PersistentVolumeClaim
   name: xxxx-claim
   namespace: yyyyyy
 csi:
   driver: gcsfuse.csi.storage.gke.io
   volumeAttributes:
     disableMetrics: "true"
   volumeHandle: gcsbucket-929a14e
 mountOptions:
 - only-dir=/dags
 - implicit-dirs
 - gid=0
 - uid=50000
 - file-mode=777
 - dir-mode=777

**The Fuse driver daemonset pod logs **

These logs are collected from the daemonset pod that is running on the k8s node, which is responsible for attaching the fuse sidecar container

I thought it could be helpful

bernetes.io~csi/xxx-airflow-logs/mount" volume_capability:{mount:{mount_flags:"only-dir=/logs" mount_flags:"implicit-dirs" mount_flags:"gid=0" mount_flags:"uid=50000" mount_flags:"file-mode=777" mount_flags:"dir-mode=777" volume_mount_gr
oup:"0"} access_mode:{mode:MULTI_NODE_MULTI_WRITER}} volume_context:{key:"csi.storage.k8s.io/ephemeral" value:"false"} volume_context:{key:"csi.storage.k8s.io/pod.name" value:"airflow-webserver-cfd8b5d58-7rjhc"} volume_context:{key:"csi.storage.
k8s.io/pod.namespace" value:"xxxxx"} volume_context:{key:"csi.storage.k8s.io/pod.uid" value:"8716a2a6-6e7b-4c8c-868d-642e80f16b1e"} volume_context:{key:"csi.storage.k8s.io/serviceAccount.name" value:"xxx-airflow-59aed630"} volume_c
ontext:{key:"csi.storage.k8s.io/serviceAccount.tokens" value:"***stripped***"} volume_context:{key:"disableMetrics" value:"true"}
gcs-fuse-csi-driver I0131 16:30:10.977207       1 node.go:190] NodePublishVolume succeeded on volume "gcsbucket-929a14e" to target path "/var/lib/kubelet/pods/8716a2a6-6e7b-4c8c-868d-642e80f16b1e/volumes/kubernetes.io~csi/xxxx/mount", mount already exists.

What I have Tried

• When I modify one of the mounted volumes to point to emotyDir instead of the pvc, the fuse container works, but we need the two volumes to be mounted
• creating two different buckets, one for each volume and each has its own PVC, and mounting them on the pod that didn't work
• creating two volumes for the same bucket and two different pvc and mounting them that also didn't work
• the daemon s

Additional context
FWIW, the stuck pods are Airflow pods (Triggerer and Scheduler). However, I tried mounting the two volumes on another test deployment I got the same results.

SLO:
We strive to respond to all bug reports within 24 business hours provided the information mentioned above is included.

[rkhir]

@ashmeenkaur @gargnitingoogle, I see your names are assigned to other issues. Can you help me with this issue, please?

[ashmeenkaur]

Hello @rkhir, apologies for the delay in response here.

I see only one PV in the post, can you provide

• Pod specs (both, if there are 2)
• PV spec (both, if there are 2)
• PVC spec (both, if there are 2)

We suspect that you are trying to mount the same GCS bucket to different directories which is not supported.

Could you instead try using Kubernetes subPath? This would allow you to mount the same GCS bucket in different directories.

Alternatively, if you need different mount options, you could also try mounting GCS buckets as ephemeral volumes: https://cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-ephemeral. This does support the use-case that you are trying to do.

[rkhir]

Hello @ashmeenkaur, thank you for your response.

We tried to take advantage of the mountOptions only-dir and mount each PV on a different path on the bucket, but that didn't work (the pods were stuck). Below is the second PV the yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
  finalizers:
  - kubernetes.io/pv-protection
  name: xxx-airflow-dags
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: xxx-airflow-dags
    namespace:xxx
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeAttributes:
      disableMetrics: "true"
    volumeHandle: xxx-bucket-49386d2
  mountOptions:
  - only-dir=/dags
  - implicit-dirs
  - gid=xxx
  - uid=xxx
  - file-mode=xxx
  - dir-mode=xx
  persistentVolumeReclaimPolicy: Retain
  storageClassName: some-storageclass
  volumeMode: Filesystem

The k8s subPath didn't work either with one (PV and one PVC).

We finally went with two buckets, each PV/PVC backed by its own

[gargnitingoogle]

Thanks @rkhir for confirming that ony-dir and k8s subPath did not work for you. I am glad that you were able to unblock yourself with splitting your workload between two separate files.

For helping us fix this issue for the future, could you share more details of what problems/errors you faced with only-dir and subPath ?

[rkhir]

sure @gargnitingoogle,

here is the log I am getting from the daemonset pod gcsfusecsi-node-xxxx that runs nodePublish Volume followed by the pod yaml

Note: Since the pod is stuck the nodePublish volume keeps trying to mount the PV to the bucke,t but its already mounted

  gcs-fuse-csi-driver I0220 15:58:44.243090       1 utils.go:120] /csi.v1.Node/NodePublishVolume succeeded.
  gcs-fuse-csi-driver I0220 15:58:44.242934       1 csi_mounter.go:179] [Pod 97d1320c-5d1c-4e56-9ede-196cc0d2a03f, Volume xxxx-airflow-logs, Bucket xxxx-airflow-bucket-0f2b6ee] start to accept connections to the listener.
....
....
....
...
gcs-fuse-csi-driver I0220 16:14:31.675812       1 utils.go:114] /csi.v1.Node/NodePublishVolume called with request: volume_id:"example-airflow-bucket-abc123" target_path:"/var/lib/kubelet/pods/example-pod-uuid/volumes/kubernetes.io~csi/example-airflow-logs/mount" volume_capability:{mount:{mount_flags:"only-dir=/logs" mount_flags:"implicit-dirs" mount_flags:"gid=0" mount_flags:"uid=50000" mount_flags:"file-mode=777" mount_flags:"dir-mode=777" volume_mount_group:"0"} access_mode:{mode:MULTI_NODE_MULTI_WRITER}} volume_context:{key:"csi.storage.k8s.io/ephemeral" value:"false"} volume_context:{key:"csi.storage.k8s.io/pod.name" value:"airflow-webserver-example-pod"} volume_context:{key:"csi.storage.k8s.io/pod.namespace" value:"example-namespace"} volume_context:{key:"csi.storage.k8s.io/pod.uid" value:"example-pod-uuid"} volume_context:{key:"csi.storage.k8s.io/serviceAccount.name" value:"example-airflow-sa"} volume_context:{key:"csi.storage.k8s.io/serviceAccount.tokens" value:"***stripped***"} volume_context:{key:"disableMetrics" value:"true"}

gcs-fuse-csi-driver I0220 16:14:31.676577       1 node.go:190] NodePublishVolume succeeded on volume "example-airflow-bucket-abc123" to target path "/var/lib/kubelet/pods/example-pod-uuid/volumes/kubernetes.io~csi/example-airflow-logs/mount", mount already exists.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
  labels:
    argocd.argoproj.io/instance: example-airflow
    chart: airflow-1.14.0
    component: scheduler
    executor: KubernetesExecutor
    heritage: Helm
    release: airflow
    tier: airflow
  name: airflow-scheduler
  namespace: example-namespace
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      component: scheduler
      release: airflow
      tier: airflow
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        component: scheduler
        release: airflow
        tier: airflow
    spec:
      affinity: {}
      containers:
        - args:
            - bash
            - -c
            - exec airflow scheduler
          image: example.registry/org/apache/airflow:latest
          imagePullPolicy: IfNotPresent
          livenessProbe:
            exec:
              command:
                - sh
                - -c
                - |
                  CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
                  airflow jobs check --job-type SchedulerJob --local
            failureThreshold: 5
            initialDelaySeconds: 10
            periodSeconds: 60
            successThreshold: 1
            timeoutSeconds: 20
          name: scheduler
          resources:
            limits:
              cpu: "1"
              memory: 2Gi
            requests:
              cpu: 500m
              memory: 1Gi
          volumeMounts:
            - mountPath: /opt/airflow/pod_templates/pod_template_file.yaml
              name: config
              readOnly: true
              subPath: pod_template_file.yaml
            - mountPath: /opt/airflow/logs
              name: logs
              subPath: logs/
            - mountPath: /opt/airflow/airflow.cfg
              name: config
              readOnly: true
              subPath: airflow.cfg
            - mountPath: /opt/airflow/config/airflow_local_settings.py
              name: config
              readOnly: true
              subPath: airflow_local_settings.py
            - mountPath: /opt/airflow/webserver_config.py
              name: webserver-config
              readOnly: true
              subPath: webserver_config.py
            - mountPath: /opt/airflow/dags
              name: dags
              readOnly: true
              subPath: dags/
        - image: example.registry/org/git-sync:latest
          imagePullPolicy: IfNotPresent
          name: git-sync
          resources: {}
          securityContext:
            runAsUser: 65533
          volumeMounts:
            - mountPath: /git
              name: dags
              subPath: dags/
        - args:
            - bash
            - /clean-logs
          env:
            - name: AIRFLOW__LOG_RETENTION_DAYS
              value: "15"
            - name: AIRFLOW_HOME
              value: /opt/airflow
          image: example.registry/org/airflow:latest
          imagePullPolicy: IfNotPresent
          name: scheduler-log-groomer
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          volumeMounts:
            - mountPath: /opt/airflow/logs
              name: logs
              subPath: logs/
            - mountPath: /opt/airflow/webserver_config.py
              name: webserver-config
              readOnly: true
              subPath: webserver_config.py
      dnsPolicy: ClusterFirst
      imagePullSecrets:
        - name: regcred
      initContainers:
        - args:
            - airflow
            - db
            - check-migrations
            - --migration-wait-timeout=60
          image: example.registry/org/apache/airflow:latest
          imagePullPolicy: IfNotPresent
          name: wait-for-airflow-migrations
          resources:
            limits:
              cpu: "1"
              memory: 2Gi
            requests:
              cpu: 500m
              memory: 1Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          volumeMounts:
            - mountPath: /opt/airflow/airflow.cfg
              name: config
              readOnly: true
              subPath: airflow.cfg
            - mountPath: /opt/airflow/config/airflow_local_settings.py
              name: config
              readOnly: true
              subPath: airflow_local_settings.py
            - mountPath: /opt/airflow/webserver_config.py
              name: webserver-config
              readOnly: true
              subPath: webserver_config.py
        - image: example.registry/org/git-sync:latest
          imagePullPolicy: IfNotPresent
          name: git-sync-init
          securityContext:
            runAsUser: 65533
          volumeMounts:
            - mountPath: /git
              name: dags
              subPath: dags/
      nodeSelector:
        topology.kubernetes.io/zone: us-west1-b
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 0
        runAsUser: 50000
      serviceAccount: example-airflow-sa
      serviceAccountName: example-airflow-sa
      terminationGracePeriodSeconds: 10
      tolerations:
        - effect: NoSchedule
          key: kubernetes.io/arch
          operator: Equal
          value: amd64
      volumes:
        - configMap:
            defaultMode: 420
            name: airflow-config
          name: config
        - configMap:
            defaultMode: 420
            name: airflow-webserver-config
          name: webserver-config
        - name: dags
          persistentVolumeClaim:
            claimName: xxxx-airflow-logs
        - name: logs
          persistentVolumeClaim:
            claimName: xxxx-airflow-logs

[gargnitingoogle]

thanks @rkhir for the details in #487 (comment). For now, we've handed it over to the GCSfuse CSI driver team as this issue is more in their expertise area.

[choryuidentify]

any progess?