-
|
I created an image with iscsi tools 0.2.0 and updated my node: talosctl upgrade -e 192.168.2.210 --nodes 192.168.2.211 --talosconfig ./talosconfig --image=factory.talos.dev/metal-installer/c9078f9419961640c712a8bf2bb9174933dfcf1da383fd8ea2b7dc21493f8bac:v1.10.2after the node reboots, I can see the iscsi extensions are running: talosctl -e 192.168.2.210 -n 192.168.2.211 --talosconfig ./talosconfig get extensions
NODE NAMESPACE TYPE ID VERSION NAME VERSION
192.168.2.211 runtime ExtensionStatus 0 1 iscsi-tools v0.2.0
192.168.2.211 runtime ExtensionStatus 1 1 schematic c9078f9419961640c712a8bf2bb9174933dfcf1da383fd8ea2b7dc21493f8bactalosctl -e 192.168.2.210 -n 192.168.2.211 --talosconfig ./talosconfig services
NODE SERVICE STATE HEALTH LAST CHANGE LAST EVENT
192.168.2.211 apid Running OK 17m56s ago Health check successful
192.168.2.211 auditd Running OK 18m8s ago Health check successful
192.168.2.211 containerd Running OK 18m8s ago Health check successful
192.168.2.211 cri Running OK 17m56s ago Health check successful
192.168.2.211 dashboard Running ? 17m57s ago Process Process(["/sbin/dashboard"]) started with PID 2135
192.168.2.211 ext-iscsid Running ? 17m56s ago Started task ext-iscsid (PID 2287) for container ext-iscsid
192.168.2.211 kubelet Running OK 17m54s ago Health check successful
192.168.2.211 machined Running OK 18m8s ago Health check successful
192.168.2.211 syslogd Running OK 18m7s ago Health check successful
192.168.2.211 udevd Running OK 17m59s ago Health check successfulI launch the democratic-csi chart (0.15.0) with the following values: node:
hostPID: true
driver:
extraEnv:
- name: ISCSIADM_HOST_STRATEGY
value: nsenter
- name: ISCSIADM_HOST_PATH
value: /usr/local/sbin/iscsiadm
iscsiDirHostPath: /usr/local/etc/iscsi
iscsiDirHostPathType: ""When the daemonset launches all the pods have the same error: Here's the pod spec: apiVersion: v1
kind: Pod
metadata:
annotations:
checksum/configmap: 6582a8e71381c8634d8861616cd3eca084bf6e64315f5f8b62ef10b875646dbf
checksum/secret: 3322ec8c61972c022ea128d7a30f786d51ce4c3193a3758cd884671fb7451052
creationTimestamp: '2025-05-22T19:15:01Z'
generateName: truenas-iscsi-democratic-csi-node-
generation: 1
labels:
app.kubernetes.io/component: node-linux
app.kubernetes.io/csi-role: node
app.kubernetes.io/instance: truenas-iscsi
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: democratic-csi
controller-revision-hash: 7cc69c7d77
pod-template-generation: '2'
name: truenas-iscsi-democratic-csi-node-pbjrw
namespace: democratic-csi
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: DaemonSet
name: truenas-iscsi-democratic-csi-node
uid: d532d856-ede8-4d0e-bd43-4c029c0d66b7
resourceVersion: '140712'
uid: 200ea888-b49e-45de-a7ed-976c14821c03
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchFields:
- key: metadata.name
operator: In
values:
- qa-node-1
containers:
- args:
- '--csi-version=1.5.0'
- '--csi-name=org.democratic-csi.freenas-api-iscsi'
- '--driver-config-file=/config/driver-config-file.yaml'
- '--log-level=info'
- '--csi-mode=node'
- '--server-socket=/csi-data/csi.sock.internal'
env:
- name: CSI_NODE_ID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: NODE_EXTRA_CA_CERTS
value: /tmp/certs/extra-ca-certs.crt
- name: ISCSIADM_HOST_STRATEGY
value: nsenter
- name: ISCSIADM_HOST_PATH
value: /usr/local/sbin/iscsiadm
image: docker.io/democraticcsi/democratic-csi:latest
imagePullPolicy: Always
livenessProbe:
exec:
command:
- bin/liveness-probe
- '--csi-version=1.5.0'
- '--csi-address=/csi-data/csi.sock.internal'
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 60
successThreshold: 1
timeoutSeconds: 15
name: csi-driver
resources: {}
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- SYS_ADMIN
privileged: true
terminationMessagePath: /tmp/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /csi-data
name: socket-dir
- mountPath: /var/lib/kubelet
mountPropagation: Bidirectional
name: kubelet-dir
- mountPath: /usr/local/etc/iscsi
mountPropagation: Bidirectional
name: iscsi-dir
- mountPath: /var/lib/iscsi
mountPropagation: Bidirectional
name: iscsi-info
- mountPath: /lib/modules
name: modules-dir
readOnly: true
- mountPath: /etc/localtime
name: localtime
readOnly: true
- mountPath: /run/udev
name: udev-data
- mountPath: /host
mountPropagation: Bidirectional
name: host-dir
- mountPath: /sys
name: sys-dir
- mountPath: /dev
name: dev-dir
- mountPath: /config
name: config
- mountPath: /tmp/certs
name: extra-ca-certs
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-c42h9
readOnly: true
- env:
- name: BIND_TO
value: unix:///csi-data/csi.sock
- name: PROXY_TO
value: unix:///csi-data/csi.sock.internal
image: docker.io/democraticcsi/csi-grpc-proxy:v0.5.6
imagePullPolicy: IfNotPresent
name: csi-proxy
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /csi-data
name: socket-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-c42h9
readOnly: true
- args:
- '--v=5'
- '--csi-address=/csi-data/csi.sock'
- >-
--kubelet-registration-path=/var/lib/kubelet/plugins/org.democratic-csi.freenas-api-iscsi/csi.sock
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- /csi-node-driver-registrar
- >-
--kubelet-registration-path=/var/lib/kubelet/plugins/org.democratic-csi.freenas-api-iscsi/csi.sock
- '--mode=kubelet-registration-probe'
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: driver-registrar
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /csi-data
name: socket-dir
- mountPath: /registration
name: registration-dir
- mountPath: /var/lib/kubelet
name: kubelet-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-c42h9
readOnly: true
- args:
- while true; do sleep 2; done;
command:
- /bin/sh
- '-c'
- '--'
image: docker.io/busybox:1.37.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/sh
- '-c'
- >-
rm -rf /plugins/org.democratic-csi.freenas-api-iscsi
/registration/org.democratic-csi.freenas-api-iscsi-reg.sock
name: cleanup
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /plugins
name: plugins-dir
- mountPath: /registration
name: registration-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-c42h9
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
enableServiceLinks: true
hostIPC: true
hostNetwork: true
hostPID: true
nodeName: qa-node-1
nodeSelector:
kubernetes.io/os: linux
preemptionPolicy: PreemptLowerPriority
priority: 2000001000
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: truenas-iscsi-democratic-csi-node-sa
serviceAccountName: truenas-iscsi-democratic-csi-node-sa
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: qalab/function
operator: Equal
value: gpu
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/disk-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/pid-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/unschedulable
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/network-unavailable
operator: Exists
volumes:
- hostPath:
path: /var/lib/kubelet/plugins/org.democratic-csi.freenas-api-iscsi
type: DirectoryOrCreate
name: socket-dir
- hostPath:
path: /var/lib/kubelet/plugins
type: Directory
name: plugins-dir
- hostPath:
path: /var/lib/kubelet/plugins_registry
type: Directory
name: registration-dir
- hostPath:
path: /var/lib/kubelet
type: Directory
name: kubelet-dir
- hostPath:
path: /usr/local/etc/iscsi
type: ''
name: iscsi-dir
- hostPath:
path: /var/lib/iscsi
type: ''
name: iscsi-info
- hostPath:
path: /dev
type: Directory
name: dev-dir
- hostPath:
path: /lib/modules
type: ''
name: modules-dir
- hostPath:
path: /etc/localtime
type: ''
name: localtime
- hostPath:
path: /run/udev
type: ''
name: udev-data
- hostPath:
path: /sys
type: Directory
name: sys-dir
- hostPath:
path: /
type: Directory
name: host-dir
- name: config
secret:
defaultMode: 420
secretName: truenas-iscsi-democratic-csi-driver-config
- configMap:
defaultMode: 420
items:
- key: extra-ca-certs
path: extra-ca-certs.crt
name: truenas-iscsi-democratic-csi
name: extra-ca-certs
- name: kube-api-access-c42h9
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: '2025-05-22T19:15:03Z'
status: 'True'
type: PodReadyToStartContainers
- lastProbeTime: null
lastTransitionTime: '2025-05-22T19:15:01Z'
status: 'True'
type: Initialized
- lastProbeTime: null
lastTransitionTime: '2025-05-22T19:15:01Z'
message: 'containers with unready status: [csi-driver csi-proxy driver-registrar]'
reason: ContainersNotReady
status: 'False'
type: Ready
- lastProbeTime: null
lastTransitionTime: '2025-05-22T19:15:01Z'
message: 'containers with unready status: [csi-driver csi-proxy driver-registrar]'
reason: ContainersNotReady
status: 'False'
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: '2025-05-22T19:15:01Z'
status: 'True'
type: PodScheduled
containerStatuses:
- containerID: >-
containerd://a73caa8da341592c273dd8367d1ee738d3490d328cf4a8e6d187596686ad9a39
image: docker.io/library/busybox:1.37.0
imageID: >-
docker.io/library/busybox@sha256:37f7b378a29ceb4c551b1b5582e27747b855bbfaa73fa11914fe0df028dc581f
lastState: {}
name: cleanup
ready: true
resources: {}
restartCount: 0
started: true
state:
running:
startedAt: '2025-05-22T19:15:02Z'
user:
linux:
gid: 0
supplementalGroups:
- 0
- 10
uid: 0
volumeMounts:
- mountPath: /plugins
name: plugins-dir
- mountPath: /registration
name: registration-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-c42h9
readOnly: true
recursiveReadOnly: Disabled
- image: docker.io/democraticcsi/democratic-csi:latest
imageID: ''
lastState: {}
name: csi-driver
ready: false
restartCount: 0
started: false
state:
waiting:
message: >-
failed to generate container
"3c5f67e709882273fe9775b9a5536487c092b3c3ca2e1f2cbe1527c6d3a229b9"
spec: failed to apply OCI options: failed to mkdir
"/usr/local/etc/iscsi": mkdir /usr/local/etc/iscsi: read-only file
system
reason: CreateContainerError
volumeMounts:
- mountPath: /csi-data
name: socket-dir
- mountPath: /var/lib/kubelet
name: kubelet-dir
- mountPath: /usr/local/etc/iscsi
name: iscsi-dir
- mountPath: /var/lib/iscsi
name: iscsi-info
- mountPath: /lib/modules
name: modules-dir
readOnly: true
recursiveReadOnly: Disabled
- mountPath: /etc/localtime
name: localtime
readOnly: true
recursiveReadOnly: Disabled
- mountPath: /run/udev
name: udev-data
- mountPath: /host
name: host-dir
- mountPath: /sys
name: sys-dir
- mountPath: /dev
name: dev-dir
- mountPath: /config
name: config
- mountPath: /tmp/certs
name: extra-ca-certs
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-c42h9
readOnly: true
recursiveReadOnly: Disabled
- containerID: >-
containerd://7cd98a114ca27087a9fbf4457daadc22aca734a4e8d471cc49d0b7169ef8c4a6
image: docker.io/democraticcsi/csi-grpc-proxy:v0.5.6
imageID: >-
docker.io/democraticcsi/csi-grpc-proxy@sha256:fc5fc1fe9c682463fe8b219db070930e77067d76449749ef5ba99cedd30fa437
lastState:
terminated:
containerID: >-
containerd://7cd98a114ca27087a9fbf4457daadc22aca734a4e8d471cc49d0b7169ef8c4a6
exitCode: 2
finishedAt: '2025-05-22T19:23:48Z'
reason: Error
startedAt: '2025-05-22T19:22:48Z'
name: csi-proxy
ready: false
resources: {}
restartCount: 5
started: false
state:
waiting:
message: >-
back-off 2m40s restarting failed container=csi-proxy
pod=truenas-iscsi-democratic-csi-node-pbjrw_democratic-csi(200ea888-b49e-45de-a7ed-976c14821c03)
reason: CrashLoopBackOff
user:
linux:
gid: 0
supplementalGroups:
- 0
- 1
- 2
- 3
- 4
- 6
- 10
- 11
- 20
- 26
- 27
uid: 0
volumeMounts:
- mountPath: /csi-data
name: socket-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-c42h9
readOnly: true
recursiveReadOnly: Disabled
- containerID: >-
containerd://48ad4305834c688df9979ae144cea52f83e5f39f84baff7316a83ddbfd404bb4
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0
imageID: >-
registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:cd21e19cd8bbd5bc56f1b4f1398a436e7897da2995d6d036c9729be3f4e456e6
lastState:
terminated:
containerID: >-
containerd://48ad4305834c688df9979ae144cea52f83e5f39f84baff7316a83ddbfd404bb4
exitCode: 1
finishedAt: '2025-05-22T19:24:10Z'
reason: Error
startedAt: '2025-05-22T19:23:40Z'
name: driver-registrar
ready: false
resources: {}
restartCount: 6
started: false
state:
waiting:
message: >-
back-off 5m0s restarting failed container=driver-registrar
pod=truenas-iscsi-democratic-csi-node-pbjrw_democratic-csi(200ea888-b49e-45de-a7ed-976c14821c03)
reason: CrashLoopBackOff
user:
linux:
gid: 0
supplementalGroups:
- 0
uid: 0
volumeMounts:
- mountPath: /csi-data
name: socket-dir
- mountPath: /registration
name: registration-dir
- mountPath: /var/lib/kubelet
name: kubelet-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-c42h9
readOnly: true
recursiveReadOnly: Disabled
hostIP: 192.168.2.211
hostIPs:
- ip: 192.168.2.211
phase: Pending
podIP: 192.168.2.211
podIPs:
- ip: 192.168.2.211
qosClass: BestEffort
startTime: '2025-05-22T19:15:01Z'
Google only brought up democratic-csi/democratic-csi#461, which I remember running into before. I had the same issue with my 1.9.1 cluster and downgrading my iscsi-tools from 0.2.0 to 0.1.6. That doesn't appear to be an option in 1.10.2 though? Is the issue with talos or with democratic-csi? Thanks |
Beta Was this translation helpful? Give feedback.
Answered by
frezbo
May 23, 2025
Replies: 1 comment
-
|
|
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
mlbiam
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
/usr/local/etc/iscsiis not a supported path for iscsi extension anymore, see siderolabs/extensions#688 (comment)