Milvus的pod一直crashbackoff

[Aragorn1026]

问题：milvus pod存在crashloopbackoff现象

背景

• 部署milvus运行几天后，发现milvus-operator、datanode、querynode存在crash现象。刚部署的前几天，未出现此问题，能正常访问和读写少量数据。
• 尝试扩容limit后无效，日志有报错信息。
• 6月17日在三个collection中分别插入11000/6000/6000条测试数据，使用的是官网提供的Python脚本插入hello_milvus的数据量规模，并进行了同minio的backup和restore测试，测试功能正常。6月18日上午发现有crash现象的。

版本

k8s: 1.28.0(ARM)
milvus-operator: 1.2.6
milvus: 2.5.12

请帮忙看下原因，谢谢~

---

相关信息如下：

1、milvus的pod crash

[root@kylin-milvus-1 crashbackloop]# kubectl get pod -owide
NAME                              READY   STATUS             RESTARTS        AGE   IP               NODE             NOMINATED NODE   READINESS GATES
milvus-operator-786f467cf-ztbws   0/1     CrashLoopBackOff   9 (4m21s ago)   52m   10.188.183.125   kylin-milvus-1   <none>           <none>
[root@kylin-milvus-1 crashbackloop]# kubectl -n milvus get pod -owide
NAME                                           READY   STATUS             RESTARTS         AGE    IP               NODE             NOMINATED NODE   READINESS GATES
my-attu-6b758665fd-sqzl9                       1/1     Running            0                18h    10.188.183.122   kylin-milvus-1   <none>           <none>
my-milvus-etcd-0                               1/1     Running            1 (2d1h ago)     7d1h   10.188.63.141    kylin-milvus-4   <none>           <none>
my-milvus-etcd-1                               1/1     Running            0                7d1h   10.188.7.104     kylin-milvus-2   <none>           <none>
my-milvus-etcd-2                               1/1     Running            0                7d1h   10.188.64.225    kylin-milvus-3   <none>           <none>
my-milvus-milvus-datanode-7bfcdccd4b-bmc2f     0/1     CrashLoopBackOff   11 (4m28s ago)   56m    10.188.63.144    kylin-milvus-4   <none>           <none>
my-milvus-milvus-indexnode-656bc5cffd-h48pw    1/1     Running            0                56m    10.188.7.65      kylin-milvus-2   <none>           <none>
my-milvus-milvus-mixcoord-566ffd94b8-8pq4c     1/1     Running            1 (50m ago)      56m    10.188.7.123     kylin-milvus-2   <none>           <none>
my-milvus-milvus-proxy-7dd474f7d-v6w7n         1/1     Running            3 (50m ago)      56m    10.188.63.145    kylin-milvus-4   <none>           <none>
my-milvus-milvus-querynode-0-fbb54cff9-q87v8   0/1     CrashLoopBackOff   4 (55s ago)      12m    10.188.183.123   kylin-milvus-1   <none>           <none>
my-milvus-milvus-querynode-1-fc64d5c96-sqx5j   0/1     CrashLoopBackOff   35 (4m38s ago)   7d1h   10.188.63.187    kylin-milvus-4   <none>           <none>
my-milvus-pulsar-bookie-0                      1/1     Running            0                6d1h   10.188.183.113   kylin-milvus-1   <none>           <none>
my-milvus-pulsar-bookie-1                      1/1     Running            0                6d1h   10.188.64.230    kylin-milvus-3   <none>           <none>
my-milvus-pulsar-bookie-2                      1/1     Running            173 (2d1h ago)   7d1h   10.188.63.139    kylin-milvus-4   <none>           <none>
my-milvus-pulsar-bookie-init-swns9             0/1     Completed          0                7d1h   <none>           kylin-milvus-4   <none>           <none>
my-milvus-pulsar-broker-0                      1/1     Running            0                7d1h   10.188.64.227    kylin-milvus-3   <none>           <none>
my-milvus-pulsar-broker-1                      1/1     Running            0                7d1h   10.188.7.111     kylin-milvus-2   <none>           <none>
my-milvus-pulsar-proxy-0                       1/1     Running            0                7d1h   10.188.7.99      kylin-milvus-2   <none>           <none>
my-milvus-pulsar-proxy-1                       1/1     Running            0                7d1h   10.188.183.110   kylin-milvus-1   <none>           <none>
my-milvus-pulsar-pulsar-init-xsp8q             0/1     Completed          0                7d1h   <none>           kylin-milvus-4   <none>           <none>
my-milvus-pulsar-recovery-0                    1/1     Running            1 (2d1h ago)     7d1h   10.188.63.132    kylin-milvus-4   <none>           <none>
my-milvus-pulsar-zookeeper-0                   1/1     Running            1 (2d1h ago)     7d1h   10.188.63.140    kylin-milvus-4   <none>           <none>
my-milvus-pulsar-zookeeper-1                   1/1     Running            0                7d1h   10.188.64.228    kylin-milvus-3   <none>           <none>
my-milvus-pulsar-zookeeper-2                   1/1     Running            0                7d1h   10.188.7.113     kylin-milvus-2   <none>           <none>
[root@kylin-milvus-1 crashbackloop]# kubectl -n milvus get milvus -owide
NAME        MODE      STATUS    UPDATED   AGE
my-milvus   cluster   Healthy   True      7d1h

2、describe milvus-operator

[root@kylin-milvus-1 crashbackloop]# kubectl describe pod milvus-operator-786f467cf-ztbws
Name:             milvus-operator-786f467cf-ztbws
Namespace:        default
Priority:         0
Service Account:  milvus-operator
Node:             kylin-milvus-1/10.194.174.91
Start Time:       Wed, 18 Jun 2025 08:48:07 +0800
Labels:           app.kubernetes.io/instance=milvus-operator
                  app.kubernetes.io/name=milvus-operator
                  pod-template-hash=786f467cf
Annotations:      cni.projectcalico.org/containerID: 4693da906e19ddea17ad755fa34118cdb0eb4b308a3d476868e1367f75331cae
                  cni.projectcalico.org/podIP: 10.188.183.125/32
                  cni.projectcalico.org/podIPs: 10.188.183.125/32
Status:           Running
IP:               10.188.183.125
IPs:
  IP:           10.188.183.125
Controlled By:  ReplicaSet/milvus-operator-786f467cf
Containers:
  manager:
    Container ID:  containerd://92cd147b9dc2c434fd8626d78c56a119aa3016b25cc0e6409677eda07c5c93bc
    Image:         repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus-operator:v1.2.6
    Image ID:      sha256:772be34179a45a846e5340e528be0ba2e8f83e5c5f56ce73ec997f87371db3da
    Ports:         9443/TCP, 8080/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /manager
    Args:
      -namespace
      default
      -name
      milvus-operator
      --health-probe-bind-address=:8081
      --metrics-bind-address=:8080
      --leader-elect
    State:          Running
      Started:      Wed, 18 Jun 2025 09:41:50 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 18 Jun 2025 09:32:04 +0800
      Finished:     Wed, 18 Jun 2025 09:36:40 +0800
    Ready:          False
    Restart Count:  10
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      100m
      memory:   100Mi
    Liveness:   http-get http://:8081/healthz delay=30s timeout=5s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8081/readyz delay=30s timeout=5s period=20s #success=1 #failure=3
    Environment:
      SSL_CERT_FILE:  /certs/minio-ca.crt
    Mounts:
      /certs from minio-ca-hostpath (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cbsgz (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  minio-ca-hostpath:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki/minio
    HostPathType:
  kube-api-access-cbsgz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  53m                    default-scheduler  Successfully assigned default/milvus-operator-786f467cf-ztbws to kylin-milvus-1
  Warning  Unhealthy  48m (x2 over 48m)      kubelet            Liveness probe failed: Get "http://10.188.183.125:8081/healthz": dial tcp 10.188.183.125:8081: connect: connection refused
  Warning  Unhealthy  47m (x4 over 48m)      kubelet            Readiness probe failed: Get "http://10.188.183.125:8081/readyz": dial tcp 10.188.183.125:8081: connect: connection refused
  Normal   Killing    47m                    kubelet            Container manager failed liveness probe, will be restarted
  Normal   Started    47m (x4 over 53m)      kubelet            Started container manager
  Normal   Created    43m (x5 over 53m)      kubelet            Created container manager
  Warning  Unhealthy  31m (x2 over 47m)      kubelet            Liveness probe failed: Get "http://10.188.183.125:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Pulled     18m (x9 over 53m)      kubelet            Container image "repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus-operator:v1.2.6" already present on machine
  Warning  BackOff    3m52s (x110 over 48m)  kubelet            Back-off restarting failed container manager in pod milvus-operator-786f467cf-ztbws_default(a0257897-600b-4766-841c-1b89d89cdbdb)

3、milvus-operator log before crashbackloop

[root@kylin-milvus-1 crashbackloop]# kubectl logs -p milvus-operator-786f467cf-ztbws
version: 1.2.6
milvus-helm version: milvus-4.2.48
{"level":"info","ts":"2025-06-18T01:32:04Z","logger":"controller","msg":"enable webhook","enable":false}
{"level":"info","ts":"2025-06-18T01:32:04Z","logger":"setup","msg":"starting manager","version":"1.2.6","milvus-helm version":"milvus-4.2.48"}
{"level":"info","ts":"2025-06-18T01:32:04Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2025-06-18T01:32:04Z","msg":"starting server","name":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2025-06-18T01:32:04Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
I0618 01:32:04.991402       1 leaderelection.go:254] attempting to acquire leader lease default/71808ec5.milvus.io...
I0618 01:32:22.399188       1 leaderelection.go:268] successfully acquired lease default/71808ec5.milvus.io
{"level":"info","ts":"2025-06-18T01:32:22Z","msg":"Starting EventSource","controller":"milvus","controllerGroup":"milvus.io","controllerKind":"Milvus","source":"kindsource: *v1beta1.Milvus"}
{"level":"info","ts":"2025-06-18T01:32:22Z","msg":"Starting EventSource","controller":"milvuscluster","controllerGroup":"milvus.io","controllerKind":"MilvusCluster","source":"kind source: *v1alpha1.MilvusCluster"}
{"level":"info","ts":"2025-06-18T01:32:22Z","msg":"Starting Controller","controller":"milvus","controllerGroup":"milvus.io","controllerKind":"Milvus"}
{"level":"info","ts":"2025-06-18T01:32:22Z","msg":"Starting Controller","controller":"milvuscluster","controllerGroup":"milvus.io","controllerKind":"MilvusCluster"}
{"level":"info","ts":"2025-06-18T01:32:22Z","msg":"Starting EventSource","controller":"milvusupgrade","controllerGroup":"milvus.io","controllerKind":"MilvusUpgrade","source":"kind source: *v1beta1.MilvusUpgrade"}
{"level":"info","ts":"2025-06-18T01:32:22Z","msg":"Starting EventSource","controller":"milvusupgrade","controllerGroup":"milvus.io","controllerKind":"MilvusUpgrade","source":"kind source: *v1.Pod"}
{"level":"info","ts":"2025-06-18T01:32:22Z","msg":"Starting Controller","controller":"milvusupgrade","controllerGroup":"milvus.io","controllerKind":"MilvusUpgrade"}
{"level":"info","ts":"2025-06-18T01:32:22Z","msg":"Starting workers","controller":"milvus","controllerGroup":"milvus.io","controllerKind":"Milvus","worker count":10}
{"level":"info","ts":"2025-06-18T01:32:22Z","logger":"controller.status-syncer","msg":"setup loop","func":"syncHealthyUpdated-fm","interval":"1m0s"}
{"level":"info","ts":"2025-06-18T01:32:22Z","logger":"controller.status-syncer","msg":"syncHealthyUpdated start","time":"2025-06-18T01:32:22Z"}
{"level":"info","ts":"2025-06-18T01:32:22Z","logger":"controller.status-syncer","msg":"setup loop","func":"syncUnealthyOrUpdating-fm","interval":"30s"}
{"level":"info","ts":"2025-06-18T01:32:22Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating start","time":"2025-06-18T01:32:22Z"}
{"level":"info","ts":"2025-06-18T01:32:22Z","logger":"controller.status-syncer","msg":"setup loop","func":"updateMetrics-fm","interval":"30s"}
{"level":"info","ts":"2025-06-18T01:32:22Z","logger":"controller.status-syncer","msg":"syncHealthyUpdated end","duration":0.0159226}
{"level":"info","ts":"2025-06-18T01:32:22Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating end","duration":0.01597374}
{"level":"info","ts":"2025-06-18T01:32:22Z","msg":"Starting workers","controller":"milvuscluster","controllerGroup":"milvus.io","controllerKind":"MilvusCluster","worker count":1}
{"level":"info","ts":"2025-06-18T01:32:22Z","msg":"Starting workers","controller":"milvusupgrade","controllerGroup":"milvus.io","controllerKind":"MilvusUpgrade","worker count":1}
{"level":"info","ts":"2025-06-18T01:32:22Z","logger":"controller.helm","msg":"getting history for release my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:32:22Z","logger":"controller.helm","msg":"getting history for release my-milvus-etcd"}
{"level":"info","ts":"2025-06-18T01:32:26Z","logger":"deploy-ctrl","msg":"rollout not finished","milvus":"my-milvus","component":"querynode","id":"5","reason":"updated replicas not equal to available replicas"}
{"level":"info","ts":"2025-06-18T01:32:52Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating start","time":"2025-06-18T01:32:52Z"}
{"level":"info","ts":"2025-06-18T01:32:52Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating end","duration":0.00202207}
{"level":"info","ts":"2025-06-18T01:33:20Z","logger":"controller.helm","msg":"update helm","namespace":"milvus","release":"my-milvus-pulsar","needUpdate":true,"deepEqual":true}
{"level":"info","ts":"2025-06-18T01:33:20Z","logger":"controller.helm","msg":"preparing upgrade for my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:33:22Z","logger":"controller.status-syncer","msg":"syncHealthyUpdated start","time":"2025-06-18T01:33:22Z"}
{"level":"info","ts":"2025-06-18T01:33:22Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating start","time":"2025-06-18T01:33:22Z"}
{"level":"info","ts":"2025-06-18T01:33:22Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating end","duration":0.00087814}
{"level":"info","ts":"2025-06-18T01:33:22Z","logger":"controller.status-syncer","msg":"syncHealthyUpdated end","duration":0.0080155}
{"level":"info","ts":"2025-06-18T01:33:41Z","logger":"controller.helm","msg":"performing update for my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"creating upgraded release for my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"checking 22 resources for changes"}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-recovery\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-bookie\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-broker-acct\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-proxy\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-zookeeper\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-recovery\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-bookie\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-broker\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-proxy\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-zookeeper\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for Role \"my-milvus-pulsar-broker-role\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for RoleBinding \"my-milvus-pulsar-broker-rolebinding\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-recovery\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-bookie\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-broker\""}
{"level":"info","ts":"2025-06-18T01:33:42Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-proxy\""}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-zookeeper\""}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-recovery\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-bookie\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.helm","msg":"error updating the resource \"my-milvus-pulsar-bookie\":\n\t cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-broker\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-proxy\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-zookeeper\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.helm","msg":"warning: Upgrade \"my-milvus-pulsar\" failed: cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.milvus","msg":"reconcileAll","err":"reconcile milvus: run group failed: groups error: reconcile milvus: run group failed: groups error: reconcile milvus deployments errs: %!w(<nil>);cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"}
{"level":"error","ts":"2025-06-18T01:33:43Z","msg":"Reconciler error","controller":"milvus","controllerGroup":"milvus.io","controllerKind":"Milvus","Milvus":{"name":"my-milvus","namespace":"milvus"},"namespace":"milvus","name":"my-milvus","reconcileID":"62b76ff8-ba7f-4f0c-90ea-17d00a12abe0","error":"reconcile milvus: run group failed: groups error: reconcile milvus: run group failed: groups error: reconcile milvus deployments errs: %!w(<nil>);cannot patch \"my-milvus-pulsar-bookie\" with kindStatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden","errorVerbose":"groups error: reconcile milvus: run group failed: groups error: reconcile milvus deployments errs: %!w(<nil>);cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*Group).Wait\n\t/workspace/pkg/controllers/groups.go:89\ngithub.com/zilliztech/milvus-operator/pkg/controllers.ParallelGroupRunner.Run\n\t/workspace/pkg/controllers/group_runner.go:60\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).ReconcileAll\n\t/workspace/pkg/controllers/milvus.go:43\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).Reconcile\n\t/workspace/pkg/controllers/milvus_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223\nrun group failed\ngithub.com/zilliztech/milvus-operator/pkg/controllers.ParallelGroupRunner.Run\n\t/workspace/pkg/controllers/group_runner.go:61\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).ReconcileAll\n\t/workspace/pkg/controllers/milvus.go:43\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).Reconcile\n\t/workspace/pkg/controllers/milvus_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223\nreconcile milvus\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).ReconcileAll\n\t/workspace/pkg/controllers/milvus.go:44\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).Reconcile\n\t/workspace/pkg/controllers/milvus_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224"}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.helm","msg":"getting history for release my-milvus-etcd"}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"controller.helm","msg":"getting history for release my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:33:43Z","logger":"deploy-ctrl","msg":"rollout not finished","milvus":"my-milvus","component":"querynode","id":"5","reason":"last deploy not scale to 0"}
{"level":"info","ts":"2025-06-18T01:33:52Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating start","time":"2025-06-18T01:33:52Z"}
{"level":"info","ts":"2025-06-18T01:33:52Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating end","duration":0.00066152}
{"level":"info","ts":"2025-06-18T01:34:22Z","logger":"controller.status-syncer","msg":"syncHealthyUpdated start","time":"2025-06-18T01:34:22Z"}
{"level":"info","ts":"2025-06-18T01:34:22Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating start","time":"2025-06-18T01:34:22Z"}
{"level":"info","ts":"2025-06-18T01:34:22Z","logger":"controller.status-syncer","msg":"syncHealthyUpdated end","duration":0.0009032}
{"level":"info","ts":"2025-06-18T01:34:22Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating end","duration":0.00101161}
{"level":"info","ts":"2025-06-18T01:34:39Z","logger":"controller.helm","msg":"update helm","namespace":"milvus","release":"my-milvus-pulsar","needUpdate":true,"deepEqual":true}
{"level":"info","ts":"2025-06-18T01:34:39Z","logger":"controller.helm","msg":"preparing upgrade for my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:34:52Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating start","time":"2025-06-18T01:34:52Z"}
{"level":"info","ts":"2025-06-18T01:34:52Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating end","duration":0.00059114}
{"level":"info","ts":"2025-06-18T01:35:00Z","logger":"controller.helm","msg":"performing update for my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"creating upgraded release for my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"checking 22 resources for changes"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-recovery\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-bookie\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-broker-acct\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-proxy\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-zookeeper\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-recovery\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-bookie\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-broker\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-proxy\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-zookeeper\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for Role \"my-milvus-pulsar-broker-role\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for RoleBinding \"my-milvus-pulsar-broker-rolebinding\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-recovery\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-bookie\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-broker\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-proxy\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-zookeeper\""}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-recovery\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-bookie\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"error updating the resource \"my-milvus-pulsar-bookie\":\n\t cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-broker\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-proxy\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-zookeeper\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"warning: Upgrade \"my-milvus-pulsar\" failed: cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.milvus","msg":"reconcileAll","err":"reconcile milvus: run group failed: groups error: reconcile milvus: run group failed: groups error: reconcile milvus deployments errs: %!w(<nil>);cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"}
{"level":"error","ts":"2025-06-18T01:35:02Z","msg":"Reconciler error","controller":"milvus","controllerGroup":"milvus.io","controllerKind":"Milvus","Milvus":{"name":"my-milvus","namespace":"milvus"},"namespace":"milvus","name":"my-milvus","reconcileID":"c4458072-725a-4a25-b662-6ffbd97549e2","error":"reconcile milvus: run group failed: groups error: reconcile milvus: run group failed: groups error: reconcile milvus deployments errs: %!w(<nil>);cannot patch \"my-milvus-pulsar-bookie\" with kindStatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden","errorVerbose":"groups error: reconcile milvus: run group failed: groups error: reconcile milvus deployments errs: %!w(<nil>);cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*Group).Wait\n\t/workspace/pkg/controllers/groups.go:89\ngithub.com/zilliztech/milvus-operator/pkg/controllers.ParallelGroupRunner.Run\n\t/workspace/pkg/controllers/group_runner.go:60\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).ReconcileAll\n\t/workspace/pkg/controllers/milvus.go:43\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).Reconcile\n\t/workspace/pkg/controllers/milvus_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223\nrun group failed\ngithub.com/zilliztech/milvus-operator/pkg/controllers.ParallelGroupRunner.Run\n\t/workspace/pkg/controllers/group_runner.go:61\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).ReconcileAll\n\t/workspace/pkg/controllers/milvus.go:43\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).Reconcile\n\t/workspace/pkg/controllers/milvus_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223\nreconcile milvus\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).ReconcileAll\n\t/workspace/pkg/controllers/milvus.go:44\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).Reconcile\n\t/workspace/pkg/controllers/milvus_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"getting history for release my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"controller.helm","msg":"getting history for release my-milvus-etcd"}
{"level":"info","ts":"2025-06-18T01:35:02Z","logger":"deploy-ctrl","msg":"rollout not finished","milvus":"my-milvus","component":"querynode","id":"5","reason":"last deploy not scale to 0"}
{"level":"info","ts":"2025-06-18T01:35:22Z","logger":"controller.status-syncer","msg":"syncHealthyUpdated start","time":"2025-06-18T01:35:22Z"}
{"level":"info","ts":"2025-06-18T01:35:22Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating start","time":"2025-06-18T01:35:22Z"}
{"level":"info","ts":"2025-06-18T01:35:22Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating end","duration":0.0007881}
{"level":"info","ts":"2025-06-18T01:35:22Z","logger":"controller.status-syncer","msg":"syncHealthyUpdated end","duration":0.00095022}
{"level":"info","ts":"2025-06-18T01:35:52Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating start","time":"2025-06-18T01:35:52Z"}
{"level":"info","ts":"2025-06-18T01:35:52Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating end","duration":0.00066806}
{"level":"info","ts":"2025-06-18T01:36:01Z","logger":"controller.helm","msg":"update helm","namespace":"milvus","release":"my-milvus-pulsar","needUpdate":true,"deepEqual":true}
{"level":"info","ts":"2025-06-18T01:36:01Z","logger":"controller.helm","msg":"preparing upgrade for my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:36:22Z","logger":"controller.status-syncer","msg":"syncHealthyUpdated start","time":"2025-06-18T01:36:22Z"}
{"level":"info","ts":"2025-06-18T01:36:22Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating start","time":"2025-06-18T01:36:22Z"}
{"level":"info","ts":"2025-06-18T01:36:22Z","logger":"controller.status-syncer","msg":"syncHealthyUpdated end","duration":0.00101795}
{"level":"info","ts":"2025-06-18T01:36:22Z","logger":"controller.status-syncer","msg":"syncUnealthyOrUpdating end","duration":0.00233381}
{"level":"info","ts":"2025-06-18T01:36:23Z","logger":"controller.helm","msg":"performing update for my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"creating upgraded release for my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"checking 22 resources for changes"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-recovery\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-bookie\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-broker-acct\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-proxy\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for ServiceAccount \"my-milvus-pulsar-zookeeper\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-recovery\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-bookie\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-broker\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-proxy\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for ConfigMap \"my-milvus-pulsar-zookeeper\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for Role \"my-milvus-pulsar-broker-role\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for RoleBinding \"my-milvus-pulsar-broker-rolebinding\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-recovery\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-bookie\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-broker\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-proxy\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Looks like there are no changes for Service \"my-milvus-pulsar-zookeeper\""}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-recovery\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-bookie\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"error updating the resource \"my-milvus-pulsar-bookie\":\n\t cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-broker\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-proxy\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"Patch StatefulSet \"my-milvus-pulsar-zookeeper\" in namespace milvus"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"warning: Upgrade \"my-milvus-pulsar\" failed: cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.milvus","msg":"reconcileAll","err":"reconcile milvus: run group failed: groups error: reconcile milvus: run group failed: groups error: reconcile milvus deployments errs: %!w(<nil>);cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"}
{"level":"error","ts":"2025-06-18T01:36:25Z","msg":"Reconciler error","controller":"milvus","controllerGroup":"milvus.io","controllerKind":"Milvus","Milvus":{"name":"my-milvus","namespace":"milvus"},"namespace":"milvus","name":"my-milvus","reconcileID":"12577eea-df4a-474a-b15d-62517672121f","error":"reconcile milvus: run group failed: groups error: reconcile milvus: run group failed: groups error: reconcile milvus deployments errs: %!w(<nil>);cannot patch \"my-milvus-pulsar-bookie\" with kindStatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden","errorVerbose":"groups error: reconcile milvus: run group failed: groups error: reconcile milvus deployments errs: %!w(<nil>);cannot patch \"my-milvus-pulsar-bookie\" with kind StatefulSet: StatefulSet.apps \"my-milvus-pulsar-bookie\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*Group).Wait\n\t/workspace/pkg/controllers/groups.go:89\ngithub.com/zilliztech/milvus-operator/pkg/controllers.ParallelGroupRunner.Run\n\t/workspace/pkg/controllers/group_runner.go:60\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).ReconcileAll\n\t/workspace/pkg/controllers/milvus.go:43\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).Reconcile\n\t/workspace/pkg/controllers/milvus_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223\nrun group failed\ngithub.com/zilliztech/milvus-operator/pkg/controllers.ParallelGroupRunner.Run\n\t/workspace/pkg/controllers/group_runner.go:61\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).ReconcileAll\n\t/workspace/pkg/controllers/milvus.go:43\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).Reconcile\n\t/workspace/pkg/controllers/milvus_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223\nreconcile milvus\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).ReconcileAll\n\t/workspace/pkg/controllers/milvus.go:44\ngithub.com/zilliztech/milvus-operator/pkg/controllers.(*MilvusReconciler).Reconcile\n\t/workspace/pkg/controllers/milvus_controller.go:184\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1223","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"getting history for release my-milvus-etcd"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"controller.helm","msg":"getting history for release my-milvus-pulsar"}
{"level":"info","ts":"2025-06-18T01:36:25Z","logger":"deploy-ctrl","msg":"rollout not finished","milvus":"my-milvus","component":"querynode","id":"5","reason":"updated replicas not equal to available replicas"}
E0618 01:36:35.736996       1 leaderelection.go:429] Failed to update lock optimitically: Put "https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/default/leases/71808ec5.milvus.io?timeout=5s": context deadline exceeded (Client.Timeout exceeded while awaiting headers), falling back to slow path
E0618 01:36:40.736064       1 leaderelection.go:436] error retrieving resource lock default/71808ec5.milvus.io: Get "https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/default/leases/71808ec5.milvus.io?timeout=5s": context deadline exceeded
I0618 01:36:40.736141       1 leaderelection.go:297] failed to renew lease default/71808ec5.milvus.io: timed out waiting for the condition
{"level":"error","ts":"2025-06-18T01:36:40Z","logger":"setup","msg":"problem running manager","error":"leader election lost","stacktrace":"main.main\n\t/workspace/main.go:115\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283"}

4、查看datanode状态

[root@kylin-milvus-1 crashbackloop]# kubectl -n milvus describe pod my-milvus-milvus-datanode-7bfcdccd4b-bmc2f
Name:             my-milvus-milvus-datanode-7bfcdccd4b-bmc2f
Namespace:        milvus
Priority:         0
Service Account:  default
Node:             kylin-milvus-4/10.194.174.94
Start Time:       Wed, 18 Jun 2025 08:48:47 +0800
Labels:           app.kubernetes.io/component=datanode
                  app.kubernetes.io/instance=my-milvus
                  app.kubernetes.io/managed-by=milvus-operator
                  app.kubernetes.io/name=milvus
                  pod-template-hash=7bfcdccd4b
Annotations:      checksum/config: 0ee1bb98b02eee31a17c554e55aae9274556f42f123e3d200c0f9394444e8061
                  cni.projectcalico.org/containerID: 4737b3615f29cd3717c59b4dcb842c3d8b0c0852fa2636a45750b62983eb1764
                  cni.projectcalico.org/podIP: 10.188.63.144/32
                  cni.projectcalico.org/podIPs: 10.188.63.144/32
                  milvus.io/using-configmap: my-milvus
Status:           Running
IP:               10.188.63.144
IPs:
  IP:           10.188.63.144
Controlled By:  ReplicaSet/my-milvus-milvus-datanode-7bfcdccd4b
Init Containers:
  config:
    Container ID:  containerd://6c8875fc5d9cbb7f112e5c0332e172c6876ae0a5649f0abd34ea4fbd6f67d7f6
    Image:         repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus-operator:v1.2.6
    Image ID:      repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus-operator@sha256:45694c5cc5b2734784fee11e7ea3b2509072a7a93a236d84234d593f0d9c2a02
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      /init.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 18 Jun 2025 08:48:48 +0800
      Finished:     Wed, 18 Jun 2025 08:48:48 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /milvus/configs/operator from milvus-config (ro)
      /milvus/tools from tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dxqzb (ro)
Containers:
  datanode:
    Container ID:  containerd://51678abb9cafe6a7cb12eea4e70c8484ab3aab63b1a1c9d263468666bd654002
    Image:         repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus:v2.5.12
    Image ID:      repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus@sha256:75224a1cd3912d1a261990a8f9d82459b55755835a68cd09290a7b15d3798cc1
    Port:          9091/TCP
    Host Port:     0/TCP
    Args:
      /milvus/tools/run.sh
      milvus
      run
      datanode
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    134
      Started:      Wed, 18 Jun 2025 09:45:11 +0800
      Finished:     Wed, 18 Jun 2025 09:46:36 +0800
    Ready:          False
    Restart Count:  12
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      2
      memory:   4Gi
    Liveness:   tcp-socket :9091 delay=0s timeout=10s period=15s #success=1 #failure=3
    Readiness:  http-get http://:9091/healthz delay=0s timeout=3s period=15s #success=1 #failure=2
    Startup:    tcp-socket :9091 delay=0s timeout=3s period=10s #success=1 #failure=18
    Environment:
      MINIO_ACCESS_KEY_ID:      <set to the key 'accesskey' in secret 'my-minio-secret'>  Optional: false
      MINIO_SECRET_ACCESS_KEY:  <set to the key 'secretkey' in secret 'my-minio-secret'>  Optional: false
      SSL_CERT_FILE:            /certs/minio-ca.crt
      CACHE_SIZE:               4 (limits.memory)
      MINIO_ACCESS_KEY:         <set to the key 'accesskey' in secret 'my-minio-secret'>  Optional: false
      MINIO_SECRET_KEY:         <set to the key 'secretkey' in secret 'my-minio-secret'>  Optional: false
    Mounts:
      /certs from minio-ca-hostpath (ro)
      /milvus/configs/operator from milvus-config (ro)
      /milvus/tools from tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dxqzb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  minio-ca-hostpath:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki/minio
    HostPathType:
  milvus-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      my-milvus
    Optional:  false
  tools:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-dxqzb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  59m                    default-scheduler  Successfully assigned milvus/my-milvus-milvus-datanode-7bfcdccd4b-bmc2f to kylin-milvus-4
  Normal   Pulled     59m                    kubelet            Container image "repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus-operator:v1.2.6" already present on machine
  Normal   Created    59m                    kubelet            Created container config
  Normal   Started    59m                    kubelet            Started container config
  Normal   Started    54m (x3 over 59m)      kubelet            Started container datanode
  Warning  Unhealthy  53m (x5 over 56m)      kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Normal   Created    51m (x4 over 59m)      kubelet            Created container datanode
  Normal   Pulled     29m (x9 over 59m)      kubelet            Container image "repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus:v2.5.12" already present on machine
  Warning  BackOff    4m32s (x176 over 54m)  kubelet            Back-off restarting failed container datanode in pod my-milvus-milvus-datanode-7bfcdccd4b-bmc2f_milvus(12dcf200-b6a5-4821-86f5-5486440aa227)
[root@kylin-milvus-1 crashbackloop]#

5、datanode崩溃前的日志

[2025/06/18 01:46:31.394 +00:00] [WARN] [msgstream/mq_msgstream.go:943] ["Failed to seek"] [channel=my-milvus-rootcoord-dml_5] [error="server error: UnknownError: Err
or when resetting subscription: unable to persist readPosition for cursor reset 350930:17179"]
[2025/06/18 01:46:31.395 +00:00] [WARN] [msgstream/mq_msgstream.go:943] ["Failed to seek"] [channel=my-milvus-rootcoord-dml_1] [error="server error: UnknownError: Err
or when resetting subscription: unable to persist readPosition for cursor reset 350931:17172"]
[2025/06/18 01:46:31.396 +00:00] [WARN] [msgstream/mq_msgstream.go:943] ["Failed to seek"] [channel=my-milvus-rootcoord-dml_11] [error="server error: UnknownError: Er
ror when resetting subscription: unable to persist readPosition for cursor reset 350936:498"]
[2025/06/18 01:46:36.372 +00:00] [ERROR] [msgdispatcher/dispatcher.go:122] ["seek failed"] [pchannel=my-milvus-rootcoord-dml_12] [id=1] [subName=my-milvus-rootcoord-dml_12-1-1750211115082006990] [error="failed to seek, error server error: UnknownError: Error when resetting subscription: unable to persist readPosition for cursor reset 350935:498"] [stack="github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.NewDispatcher\n\t/workspace/source/pkg/mq/msgdispatcher/dispatcher.go:122\ngithub.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).tryBuildDispatcher\n\t/workspace/source/pkg/mq/msgdispatcher/manager.go:259\ngithub.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).Run\n\t/workspace/source/pkg/mq/msgdispatcher/manager.go:141"]
[2025/06/18 01:46:36.372 +00:00] [INFO] [msgstream/mq_msgstream.go:227] ["start to close mq msg stream"] [producers="[]"] [consumers="[my-milvus-rootcoord-dml_12]"]
[2025/06/18 01:46:36.372 +00:00] [ERROR] [msgdispatcher/dispatcher.go:122] ["seek failed"] [pchannel=my-milvus-rootcoord-dml_7] [id=1] [subName=my-milvus-rootcoord-dml_7-1-1750211115081705070] [error="failed to seek, error server error: UnknownError: Error when resetting subscription: unable to persist readPosition for cursor reset 350926:17517"] [stack="github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.NewDispatcher\n\t/workspace/source/pkg/mq/msgdispatcher/dispatcher.go:122\ngithub.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).tryBuildDispatcher\n\t/workspace/source/pkg/mq/msgdispatcher/manager.go:259\ngithub.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).Run\n\t/workspace/source/pkg/mq/msgdispatcher/manager.go:141"]
[2025/06/18 01:46:36.377 +00:00] [ERROR] [msgdispatcher/dispatcher.go:122] ["seek failed"] [pchannel=my-milvus-rootcoord-dml_13] [id=1] [subName=my-milvus-rootcoord-dml_13-1-1750211115082325430] [error="failed to seek, error server error: UnknownError: Error when resetting subscription: unable to persist readPosition for cursor reset 350934:501"] [stack="github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.NewDispatcher\n\t/workspace/source/pkg/mq/msgdispatcher/dispatcher.go:122\ngithub.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).tryBuildDispatcher\n\t/workspace/source/pkg/mq/msgdispatcher/manager.go:259\ngithub.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).Run\n\t/workspace/source/pkg/mq/msgdispatcher/manager.go:141"]
[2025/06/18 01:46:36.377 +00:00] [INFO] [msgstream/mq_msgstream.go:227] ["start to close mq msg stream"] [producers="[]"] [consumers="[my-milvus-rootcoord-dml_13]"]
[2025/06/18 01:46:36.382 +00:00] [INFO] [msgstream/mq_msgstream.go:248] ["mq msg stream closed"] [producers="[]"] [consumers="[my-milvus-rootcoord-dml_7]"]
panic: failed to seek, error server error: UnknownError: Error when resetting subscription: unable to persist readPosition for cursor reset 350926:17517

goroutine 461 gp=0x4000cc0fc0 m=0 mp=0x8878f00 [running]:
panic({0x51affa0?, 0x4001ccd7d0?})
        /usr/local/go/src/runtime/panic.go:811 +0x154 fp=0x40013838e0 sp=0x4001383830 pc=0x1f94634
github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).tryBuildDispatcher(0x4001390ab0)
        /workspace/source/pkg/mq/msgdispatcher/manager.go:261 +0x1320 fp=0x4001383e10 sp=0x40013838e0 pc=0x4570960
github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).Run(0x4001390ab0)
        /workspace/source/pkg/mq/msgdispatcher/manager.go:141 +0x2c0 fp=0x4001383fb0 sp=0x4001383e10 pc=0x456ec20
github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*client).Register.gowrap2()
        /workspace/source/pkg/mq/msgdispatcher/client.go:96 +0x30 fp=0x4001383fd0 sp=0x4001383fb0 pc=0x4569740
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4001383fd0 sp=0x4001383fd0 pc=0x1f9e0d4
created by github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*client).Register in goroutine 421
        /workspace/source/pkg/mq/msgdispatcher/client.go:96 +0x334

6、querynode

[root@kylin-milvus-1 crashbackloop]# kubectl -n milvus describe pod my-milvus-milvus-querynode-0-fbb54cff9-q87v8
Name:             my-milvus-milvus-querynode-0-fbb54cff9-q87v8
Namespace:        milvus
Priority:         0
Service Account:  default
Node:             kylin-milvus-1/10.194.174.91
Start Time:       Wed, 18 Jun 2025 09:32:10 +0800
Labels:           app.kubernetes.io/component=querynode
                  app.kubernetes.io/instance=my-milvus
                  app.kubernetes.io/managed-by=milvus-operator
                  app.kubernetes.io/name=milvus
                  milvus.io/querynode-group-id=0
                  pod-template-hash=fbb54cff9
Annotations:      checksum/config: 0ee1bb98b02eee31a17c554e55aae9274556f42f123e3d200c0f9394444e8061
                  cni.projectcalico.org/containerID: 97ac283ac6e53623ce3d7c77eb065289a13cb8c5eb0a6fc9581140ef3200ac4c
                  cni.projectcalico.org/podIP: 10.188.183.123/32
                  cni.projectcalico.org/podIPs: 10.188.183.123/32
                  milvus.io/using-configmap: my-milvus
Status:           Running
IP:               10.188.183.123
IPs:
  IP:           10.188.183.123
Controlled By:  ReplicaSet/my-milvus-milvus-querynode-0-fbb54cff9
Init Containers:
  config:
    Container ID:  containerd://3fdd13b12875cfec9a814a418854e61620d2657d6619ca52bf22b5cd8c005e80
    Image:         repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus-operator:v1.2.6
    Image ID:      sha256:772be34179a45a846e5340e528be0ba2e8f83e5c5f56ce73ec997f87371db3da
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      /init.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 18 Jun 2025 09:32:12 +0800
      Finished:     Wed, 18 Jun 2025 09:32:12 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /milvus/configs/operator from milvus-config (ro)
      /milvus/tools from tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t5xlf (ro)
Containers:
  querynode:
    Container ID:  containerd://116f1e786d870ca011eae2dbe617bfc0ac7c53a7165fb986f0992d94d56477b3
    Image:         repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus:v2.5.12
    Image ID:      repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus@sha256:75224a1cd3912d1a261990a8f9d82459b55755835a68cd09290a7b15d3798cc1
    Port:          9091/TCP
    Host Port:     0/TCP
    Args:
      /milvus/tools/run.sh
      milvus
      run
      querynode
    State:          Running
      Started:      Wed, 18 Jun 2025 09:50:11 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    134
      Started:      Wed, 18 Jun 2025 09:45:17 +0800
      Finished:     Wed, 18 Jun 2025 09:47:31 +0800
    Ready:          True
    Restart Count:  6
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      2
      memory:   4Gi
    Liveness:   tcp-socket :9091 delay=0s timeout=10s period=15s #success=1 #failure=3
    Readiness:  http-get http://:9091/healthz delay=0s timeout=3s period=15s #success=1 #failure=2
    Startup:    tcp-socket :9091 delay=0s timeout=3s period=10s #success=1 #failure=18
    Environment:
      MINIO_ACCESS_KEY_ID:      <set to the key 'accesskey' in secret 'my-minio-secret'>  Optional: false
      MINIO_SECRET_ACCESS_KEY:  <set to the key 'secretkey' in secret 'my-minio-secret'>  Optional: false
      SSL_CERT_FILE:            /certs/minio-ca.crt
      CACHE_SIZE:               4 (limits.memory)
      MINIO_ACCESS_KEY:         <set to the key 'accesskey' in secret 'my-minio-secret'>  Optional: false
      MINIO_SECRET_KEY:         <set to the key 'secretkey' in secret 'my-minio-secret'>  Optional: false
    Mounts:
      /certs from minio-ca-hostpath (ro)
      /milvus/configs/operator from milvus-config (ro)
      /milvus/tools from tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t5xlf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  minio-ca-hostpath:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki/minio
    HostPathType:
  milvus-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      my-milvus
    Optional:  false
  tools:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-t5xlf:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  18m                   default-scheduler  Successfully assigned milvus/my-milvus-milvus-querynode-0-fbb54cff9-q87v8 to kylin-milvus-1
  Normal   Pulled     18m                   kubelet            Container image "repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus-operator:v1.2.6" already present on machine
  Normal   Created    18m                   kubelet            Created container config
  Normal   Started    18m                   kubelet            Started container config
  Normal   Pulling    18m                   kubelet            Pulling image "repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus:v2.5.12"
  Normal   Pulled     18m                   kubelet            Successfully pulled image "repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus:v2.5.12" in 25.261s (25.261s including waiting)
  Warning  Unhealthy  13m (x4 over 18m)     kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Normal   Created    11m (x4 over 18m)     kubelet            Created container querynode
  Normal   Started    11m (x4 over 18m)     kubelet            Started container querynode
  Normal   Pulled     8m27s (x4 over 16m)   kubelet            Container image "repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus:v2.5.12" already present on machine
  Warning  BackOff    3m12s (x20 over 14m)  kubelet            Back-off restarting failed container querynode in pod my-milvus-milvus-querynode-0-fbb54cff9-q87v8_milvus(7a48b9f1-90fe-4438-bcf3-9838512c593a)
[root@kylin-milvus-1 crashbackloop]#

7、querynode日志

[2025/06/18 01:51:32.393 +00:00] [WARN] [msgstream/mq_msgstream.go:943] ["Failed to seek"] [channel=my-milvus-rootcoord-dml_7] [error="server error: UnknownError: Err
or when resetting subscription: unable to persist readPosition for cursor reset 350926:17517"]
[2025/06/18 01:51:32.426 +00:00] [INFO] [msgstream/mq_msgstream.go:940] ["MsgStream begin to seek start msg: "] [channel=my-milvus-rootcoord-dml_6] [MessageID="CNC1FR
CKiQEYACAAMAE="]
[2025/06/18 01:51:32.988 +00:00] [WARN] [msgstream/mq_msgstream.go:943] ["Failed to seek"] [channel=my-milvus-rootcoord-dml_6] [error="server error: UnknownError: Err
or when resetting subscription: unable to persist readPosition for cursor reset 350928:17546"]
[2025/06/18 01:51:37.394 +00:00] [ERROR] [msgdispatcher/dispatcher.go:122] ["seek failed"] [pchannel=my-milvus-rootcoord-dml_7] [id=1] [subName=my-milvus-rootcoord-dml_7-1-1750211416099804790] [error="failed to seek, error server error: UnknownError: Error when resetting subscription: unable to persist readPosition for cursor reset 350926:17517"] [stack="github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.NewDispatcher\n\t/workspace/source/pkg/mq/msgdispatcher/dispatcher.go:122\ngithub.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).tryBuildDispatcher\n\t/workspace/source/pkg/mq/msgdispatcher/manager.go:259\ngithub.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).Run\n\t/workspace/source/pkg/mq/msgdispatcher/manager.go:141"]
[2025/06/18 01:51:37.395 +00:00] [INFO] [msgstream/mq_msgstream.go:227] ["start to close mq msg stream"] [producers="[]"] [consumers="[my-milvus-rootcoord-dml_7]"]
[2025/06/18 01:51:37.514 +00:00] [INFO] [msgstream/mq_msgstream.go:248] ["mq msg stream closed"] [producers="[]"] [consumers="[my-milvus-rootcoord-dml_7]"]
panic: failed to seek, error server error: UnknownError: Error when resetting subscription: unable to persist readPosition for cursor reset 350926:17517

goroutine 457 gp=0x400170ba40 m=17 mp=0x4001388008 [running]:
panic({0x51affa0?, 0x4001c26670?})
        /usr/local/go/src/runtime/panic.go:811 +0x154 fp=0x400119b8e0 sp=0x400119b830 pc=0x1f94634
github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).tryBuildDispatcher(0x4001de1a70)
        /workspace/source/pkg/mq/msgdispatcher/manager.go:261 +0x1320 fp=0x400119be10 sp=0x400119b8e0 pc=0x4570960
github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*dispatcherManager).Run(0x4001de1a70)
        /workspace/source/pkg/mq/msgdispatcher/manager.go:141 +0x2c0 fp=0x400119bfb0 sp=0x400119be10 pc=0x456ec20
github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*client).Register.gowrap2()
        /workspace/source/pkg/mq/msgdispatcher/client.go:96 +0x30 fp=0x400119bfd0 sp=0x400119bfb0 pc=0x4569740
runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1223 +0x4 fp=0x400119bfd0 sp=0x400119bfd0 pc=0x1f9e0d4
created by github.com/milvus-io/milvus/pkg/v2/mq/msgdispatcher.(*client).Register in goroutine 445
        /workspace/source/pkg/mq/msgdispatcher/client.go:96 +0x334

[yhmo]

有点类似这个issue
#39562
有可能是pulsar的retention设置ttl太短

[Aragorn1026]

有点类似这个issue
#39562
有可能是pulsar的retention设置ttl太短

@yhmo 你好，我检查retention和ttl都是null，是建议设置为多少吗？

pulsar@my-milvus-pulsar-broker-0:/pulsar$ /pulsar/bin/pulsar-admin namespaces get-retention pulsar/system
null
pulsar@my-milvus-pulsar-broker-0:/pulsar$
pulsar@my-milvus-pulsar-broker-0:/pulsar$ /pulsar/bin/pulsar-admin namespaces get-retention public/default
null
pulsar@my-milvus-pulsar-broker-0:/pulsar$ /pulsar/bin/pulsar-admin namespaces get-message-ttl pulsar/system
null
pulsar@my-milvus-pulsar-broker-0:/pulsar$
pulsar@my-milvus-pulsar-broker-0:/pulsar$ /pulsar/bin/pulsar-admin namespaces get-message-ttl public/default
null

[yhmo]

看一下这个讨论，get-retention pulsar/system可能没法返回正确的值，要进入pulasr容器里查看conf/broker.conf
#40578

[Aragorn1026]

看一下这个讨论，get-retention pulsar/system可能没法返回正确的值，要进入pulasr容器里查看conf/broker.conf #40578

@yhmo 你好，查看broker.conf配置，ttl为3天，retention为7天，这是部署milvus时的默认值，未改动过。
请问是否需要调整呢？如果调整的话，是通过pulsar-admin进行修改，还是如何修改呢？

pulsar@my-milvus-pulsar-broker-0:/pulsar/conf$ cat broker.conf  | egrep -v "^#|^$" | grep ttl
ttlDurationDefaultInSeconds=259200
topicPublisherThrottlingTickTimeMillis=10
brokerPublisherThrottlingTickTimeMillis=50
brokerPublisherThrottlingMaxMessageRate=0
brokerPublisherThrottlingMaxByteRate=0
subscribeThrottlingRatePerConsumer=0
dispatchThrottlingRateInMsg=0
dispatchThrottlingRateInByte=0
dispatchThrottlingRatePerTopicInMsg=0
dispatchThrottlingRatePerTopicInByte=0
dispatchThrottlingOnBatchMessageEnabled=false
dispatchThrottlingRatePerSubscriptionInMsg=0
dispatchThrottlingRatePerSubscriptionInByte=0
dispatchThrottlingRatePerReplicatorInMsg=0
dispatchThrottlingRatePerReplicatorInByte=0
dispatchThrottlingRateRelativeToPublishRate=false
dispatchThrottlingOnNonBacklogConsumerEnabled=true
dispatchThrottlingForFilteredEntriesEnabled=false
pulsar@my-milvus-pulsar-broker-0:/pulsar/conf$
pulsar@my-milvus-pulsar-broker-0:/pulsar/conf$ cat broker.conf  | egrep -v "^#|^$" | grep defaultRetentionTime
defaultRetentionTimeInMinutes=10080
pulsar@my-milvus-pulsar-broker-0:/pulsar/conf$

[Aragorn1026]

查看broker.conf中defaultRetentionSizeInMB=-1

[yhmo]

如果ttl/retention是默认值的话就没必要改。

这个报错是在milvus调用pulsar-client的Seek接口时，pulsar server端返回的报错信息，多半是pulsar那边出了什么变故。
server error: UnknownError: Error when resetting subscription: unable to persist readPosition for cursor reset 350926:17517

唯一有点相似的是这个issue： #39562

另外，milvus要求pulsar中的所有topics必须是non partitoned