k8s中创建milvus实例后协调器连接minio失败 #42540

Aragorn1026 · 2025-06-05T07:32:43Z

Aragorn1026
Jun 5, 2025

k8s环境部署milvus失败，相关信息如下，有如下疑问，请教下，谢谢~：

生成环境如果启用了mixcoord协调器，其他的indexcoord、datacoord和querynode是否还有必要？
milvus的协调器的deploy中env已经设置了正确的minio的ca.crt PEM证书，访问minio报错x509，应该如何修改配置？

1、环境信息（离线环境）

操作系统：
Release:
Kylin Linux Advanced Server release V10 (Sword)

Kernel:
4.19.90-24.4.v2101.ky10.aarch64

milvus-operator版本1.2.6
milvus版本：2.5.12

4节点k8s集群，尝试使用最小规格部署，验证部署方案，集群中minio使用的是自签名证书

2、查看创建的协调器的pod一直crash

[root@kylin-milvus-1 manifests]# kubectl -n milvus get pod -owide
NAME                                            READY   STATUS             RESTARTS         AGE    IP              NODE             NOMINATED NODE   READINESS GATES
my-milvus-etcd-0                                1/1     Running            0                119m   10.188.63.165   kylin-milvus-4   <none>           <none>
my-milvus-etcd-1                                1/1     Running            0                119m   10.188.64.248   kylin-milvus-3   <none>           <none>
my-milvus-etcd-2                                1/1     Running            0                119m   10.188.7.108    kylin-milvus-2   <none>           <none>
my-milvus-milvus-datanode-7597dcc8d9-cdsvb      0/1     CrashLoopBackOff   23 (3m33s ago)   117m   10.188.63.170   kylin-milvus-4   <none>           <none>
my-milvus-milvus-indexnode-56685dc65-f54rr      1/1     Running            0                117m   10.188.63.180   kylin-milvus-4   <none>           <none>
my-milvus-milvus-mixcoord-6fcdb56685-4bqkw      0/1     CrashLoopBackOff   23 (3m49s ago)   117m   10.188.63.174   kylin-milvus-4   <none>           <none>
my-milvus-milvus-proxy-89544779f-ghxzv          0/1     CrashLoopBackOff   27 (4m50s ago)   117m   10.188.63.176   kylin-milvus-4   <none>           <none>
my-milvus-milvus-querynode-0-84566c4ff6-8tcj8   0/1     CrashLoopBackOff   23 (4m12s ago)   117m   10.188.63.182   kylin-milvus-4   <none>           <none>
my-milvus-pulsar-bookie-0                       1/1     Running            0                119m   10.188.183.87   kylin-milvus-1   <none>           <none>
my-milvus-pulsar-bookie-1                       1/1     Running            0                119m   10.188.7.113    kylin-milvus-2   <none>           <none>
my-milvus-pulsar-bookie-2                       1/1     Running            0                119m   10.188.63.177   kylin-milvus-4   <none>           <none>
my-milvus-pulsar-bookie-init-f6k4g              0/1     Completed          0                119m   10.188.63.173   kylin-milvus-4   <none>           <none>
my-milvus-pulsar-broker-0                       1/1     Running            0                119m   10.188.7.109    kylin-milvus-2   <none>           <none>
my-milvus-pulsar-broker-1                       1/1     Running            0                119m   10.188.64.249   kylin-milvus-3   <none>           <none>
my-milvus-pulsar-proxy-0                        1/1     Running            0                119m   10.188.64.247   kylin-milvus-3   <none>           <none>
my-milvus-pulsar-proxy-1                        1/1     Running            0                119m   10.188.183.83   kylin-milvus-1   <none>           <none>
my-milvus-pulsar-pulsar-init-6qg84              0/1     Completed          0                119m   10.188.63.169   kylin-milvus-4   <none>           <none>
my-milvus-pulsar-recovery-0                     1/1     Running            0                119m   10.188.63.171   kylin-milvus-4   <none>           <none>
my-milvus-pulsar-zookeeper-0                    1/1     Running            0                119m   10.188.63.172   kylin-milvus-4   <none>           <none>
my-milvus-pulsar-zookeeper-1                    1/1     Running            0                119m   10.188.7.111    kylin-milvus-2   <none>           <none>
my-milvus-pulsar-zookeeper-2                    1/1     Running            0                119m   10.188.64.251   kylin-milvus-3   <none>           <none>
[root@kylin-milvus-1 manifests]#

3、在milvus-operator和milvus的yaml中添加了minio的ca.crt，作为环境变量SSL_CERT_FILE的值，describe Milvus实例可以看到连接minio正常StorageReady，在#42093中，如果milvus-operator没有设置minio的ca.crt会导致协调器都无法创建出来

  Conditions:
    Last Transition Time:  2025-06-05T06:05:06Z
    Message:               [mixcoord datanode querynode proxy] not ready, detail: component[mixcoord]: pod[my-milvus-milvus-mixcoord-6fcdb56685-4bqkw]: container[mixcoord]: restartCount[23] lastState[terminated] reason[Error]
    Reason:                MilvusComponentNotHealthy
    Status:                False
    Type:                  MilvusReady
    Last Transition Time:  2025-06-05T04:14:06Z
    Message:               Milvus components[datanode,proxy] are updating
    Reason:                MilvusComponentsUpdating
    Status:                False
    Type:                  MilvusUpdated
    Last Transition Time:  2025-06-05T04:10:41Z
    Message:               Etcd endpoints is healthy
    Reason:                EtcdReady
    Status:                True
    Type:                  EtcdReady
    Last Transition Time:  2025-06-05T04:10:41Z
    Reason:                StorageReady
    Status:                True
    Type:                  StorageReady
    Last Transition Time:  2025-06-05T04:12:08Z
    Reason:                MsgStreamReady
    Status:                True
    Type:                  MsgStreamReady
  Endpoint:                my-milvus-milvus.milvus:19530
  Ingress:
    Load Balancer:
  Observed Generation:   2
  Rolling Mode Version:  2
  Status:                Pending
Events:                  <none>

4、以mixcoord为例，查看pod日志，检查k8s集群中各节点时间正常，节点间网络通信正常

[2025/06/05 06:11:08.054 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="WaitForComponentStates, not meet, DataCoord current state: Initializing"]
[2025/06/05 06:11:08.060 +00:00] [WARN] [tso/tso.go:178] ["clock offset is huge, check network latency and clock skew"] [jet-lag=211.34808ms] [prev-physical=2025/06/05 06:11:07.849 +00:00] [now=2025/06/05 06:11:08.060 +00:00]
[2025/06/05 06:11:08.075 +00:00] [WARN] [storage/minio_object_storage.go:139] ["failed to check blob bucket exist"] [bucket=milvus] [error="Get \"https://milvus-minio-hl.minio.svc.cluster.local:9000/milvus/?location=\": tls: failed to verify certificate: x509: certificate signed by unknown authority"]
[2025/06/05 06:11:08.075 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="Get \"https://milvus-minio-hl.minio.svc.cluster.local:9000/milvus/?location=\": tls: failed to verify certificate: x509: certificate signed by unknown authority"]
[2025/06/05 06:11:08.210 +00:00] [WARN] [rootcoord/root_coord.go:208] ["zero ts was met, this should be only occurred in starting state"] [minBgDdlTs=458517084414935041] [minNormalDdlTs=0]
[2025/06/05 06:11:08.287 +00:00] [WARN] [storage/minio_object_storage.go:139] ["failed to check blob bucket exist"] [bucket=milvus] [error="Get \"https://milvus-minio-hl.minio.svc.cluster.local:9000/milvus/?location=\": tls: failed to verify certificate: x509: certificate signed by unknown authority"]
[2025/06/05 06:11:08.697 +00:00] [WARN] [storage/minio_object_storage.go:139] ["failed to check blob bucket exist"] [bucket=milvus] [error="Get \"https://milvus-minio-hl.minio.svc.cluster.local:9000/milvus/?location=\": tls: failed to verify certificate: x509: certificate signed by unknown authority"]
[2025/06/05 06:11:09.211 +00:00] [INFO] [healthz/healthz_handler.go:116] ["check health failed"] [UnhealthyComponent="[datacoord]"]
[2025/06/05 06:11:09.504 +00:00] [WARN] [storage/minio_object_storage.go:139] ["failed to check blob bucket exist"] [bucket=milvus] [error="Get \"https://milvus-minio-hl.minio.svc.cluster.local:9000/milvus/?location=\": tls: failed to verify certificate: x509: certificate signed by unknown authority"]
[2025/06/05 06:11:10.214 +00:00] [INFO] [healthz/healthz_handler.go:116] ["check health failed"] [UnhealthyComponent="[datacoord]"]
[2025/06/05 06:11:11.010 +00:00] [WARN] [proxyutil/proxy_client_manager.go:288] ["proxy client is empty, GetMetrics will not send to any client"]

5、在mixcoord crash之前，登录pod内检查minio证书挂载情况，可以看到mixcoord中证书挂载正常

[root@kylin-milvus-1 ~]# kubectl -n milvus exec -it my-milvus-milvus-mixcoord-6fcdb56685-5mmkl -c mixcoord -- bash
root@my-milvus-milvus-mixcoord-6fcdb56685-5mmkl:/milvus# curl https://milvus-minio-hl.minio.svc.cluster.local:9000
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/</Resource><RequestId>18461213D61FCD18</RequestId><HostId>c6e17e6732f9a75628830b204c0f5bef13b6283091981728f8696703d9a82784</HostId></Error>root@my-milvus-milvus-mixcoord-6fcdb56685-5mmkl:/milvus#
root@my-milvus-milvus-mixcoord-6fcdb56685-5mmkl:/milvus#
root@my-milvus-milvus-mixcoord-6fcdb56685-5mmkl:/milvus# curl --cacert $SSL_CERT_FILE https://milvus-minio-hl.minio.svc.cluster.local:9000
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/</Resource><RequestId>18461217E5B71720</RequestId><HostId>90dccc0ba4ab43444a6bf31db958eaa0b36c3ec1555910985f23c0cc9e32bde2</HostId></Error>root@my-milvus-milvus-mixcoord-6fcdb56685-5mmkl:/milvus#
root@my-milvus-milvus-mixcoord-6fcdb56685-5mmkl:/milvus#
root@my-milvus-milvus-mixcoord-6fcdb56685-5mmkl:/milvus# echo $MINIO_ACCESS_KEY_ID $MINIO_SECRET_ACCESS_KEY
xxxx yyyyyy
# 如果设置为错误的ca证书，访问会报509错误
root@my-milvus-milvus-mixcoord-6fcdb56685-5mmkl:/milvus# curl --cacert /root/eca.crt https://milvus-minio-hl.minio.svc.cluster.local:9000
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@my-milvus-milvus-mixcoord-6fcdb56685-5mmkl:/milvus# cat $SSL_CERT_FILE
-----BEGIN CERTIFICATE-----
MIIC8jCCAdqgAwIBAgIQP0+UGtl7wsKBLr9WaDOczTANBgkqhkiG9w0BAQsFADAT
MREwDwYDVQQDEwhtaW5pby1jYTAeFw0yNTA1MjgwMDM0MTJaFw00NTA1MjMwMDM0
MTJaMBMxETAPBgNVBAMTCG1pbmlvLWNhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEApJbxQ+WbvNE06pp8SZIXpTKRiN2b9YXcvdVs1yw1I/lF7/RqU9Cx
Qm+8VMgWO/5SajCZ+nyf3jLkkRg1hOP1WeTjtz7pKIYNyUh17D9KbGi957ZfKR4E
Ou103u7FwqSZ+M3Q+Q5kyyyyg2PDTT/62B62JPjVewmIUa//UXsRU002itBZJBb8
ccAfRlPIXW2Z1d/t1I40Eg77lWra/TUnQUAwS0LtOdxqtsQbZCjjyufIa5zRv1D9
M5auqqKd077PtX9s8xfrckPRsUAQeRgy0/SCGlbSuyUrZ06+TM4TLrKm8Q2nqQhp
183Mly3FqIjA/c3Nfpwr2Cy+xxxx9y/RkQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMC
AqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUib1o3xVySimNw1yOZxEvPUxs
lgAwDQYJKoZIhvcNAQELBQADggEBAAKOMfCE4xadLUcdN2ypUoIfEXpgxny2lgPB
h/n9VVDtjnChJ/iUKYkc0PThmVpuJav9i5yhNXKOih7g57JXxZePFauN2XYjrN/Z
spFKNfusv44De3NfkeuIH7uzP6k3mvFW+e4jO1cT8eZhKaZx9RJJ73w5UsuUjm2p
sm1WvVxbm+wyUEkQMr5W6XWI4bsD4z8cVo4TT2gEPVlr5Cqz5KrWYXaLq0x1dOSY
pJqHcgyEIwiUnmvSHWjZ9FXYrpeiQpedRVsfzwDdpk4kepZ/KbkxHD2q43zykoen
NDciSyZoCfz7/Jik1XlxS9v6Qqlpksxzurvd+b5CDp2HPmlqIsA=
-----END CERTIFICATE-----

6、通过minio的mc工具也可以验证证书的正确性

# 正确的证书
bash-4.4# mc alias set qmyminio https://milvus-minio-hl.minio.svc.cluster.local:9000 $MINIO_ACCESS_KEY_ID $MINIO_SECRET_ACCESS_KEY
Added `qmyminio` successfully.

# 错误的证书
bash-4.4# mc alias set qmyminio https://milvus-minio-hl.minio.svc.cluster.local:9000 $MINIO_ACCESS_KEY_ID $MINIO_SECRET_ACCESS_KEY
mc: <ERROR> Unable to initialize new alias from the provided credentials. Get "https://milvus-minio-hl.minio.svc.cluster.local:9000": tls: failed to verifycertificate: x509: certificate signed by unknown authority.

7、完整的milvus-demo的yaml如下

---
apiVersion: "v1"
kind: Namespace
metadata:
  name: milvus
---
apiVersion: v1
kind: Secret
metadata:
  name: my-minio-secret
  namespace: milvus
type: Opaque
stringData:
  ak: xxxx
  sk: yyyyyy
  accesskey: xxxx
  secretkey: yyyyyy
---
apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
  annotations:
    milvus.io/dependency-values-merged: "true"
    milvus.io/pod-service-label-added: "true"
  finalizers:
  - milvus.milvus.io/finalizer
  generation: 3
  labels:
    app: milvus
    milvus.io/operator-version: 1.2.6
  name: my-milvus
  namespace: milvus
spec:
  components:
    volumes:
      - hostPath:
          path: /etc/kubernetes/pki/minio
          type: ""
        name: minio-ca-hostpath
    volumeMounts:
      - mountPath: /certs
        name: minio-ca-hostpath
        readOnly: true
    env:
    - name: MINIO_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: my-minio-secret
          key: accesskey
    - name: MINIO_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: my-minio-secret
          key: secretkey
    - name: SSL_CERT_FILE
      value: /certs/minio-ca.crt
    dataNode:
      paused: false
      probes: null
      replicas: 1
    disableMetric: false
    enableRollingUpdate: true
    image: repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/milvus:v2.5.12
    imageUpdateMode: rollingUpgrade
    indexNode:
      paused: false
      probes: null
      replicas: 1
    metricInterval: ""
    mixCoord:
      replicas: 1
    paused: false
    probes: null
    proxy:
      paused: false
      probes: null
      replicas: 1
      serviceType: ClusterIP
    queryNode:
      paused: false
      probes: null
      replicas: 1
    rollingMode: 2
    standalone:
      paused: false
      probes: null
      replicas: 0
      serviceType: ClusterIP
  config:
    common:
      metrics:
        port: 9091
      security:
        tlsMode: 1
    metrics:
      enabled: true
      enableRootPath: true
      port: 9091
      path: /metrics
    minio:
      bucketName: milvus
      # rootPath: milvus/demo
      useSSL: true
      tls:
        enabled: true
      ssl:
        tlsCACert: /certs/ca.crt
      useVirtualHost: false
    dataCoord:
      enableActiveStandby: true
    indexCoord:
      enableActiveStandby: true
    queryCoord:
      enableActiveStandby: true
    rootCoord:
      enableActiveStandby: true
  dependencies:
    customMsgStream: null
    etcd:
      endpoints:
      - my-milvus-etcd-0.my-milvus-etcd-headless.milvus:2379
      - my-milvus-etcd-1.my-milvus-etcd-headless.milvus:2379
      - my-milvus-etcd-2.my-milvus-etcd-headless.milvus:2379
      external: false
      inCluster:
        deletionPolicy: Retain
        values:
          auth:
            rbac:
              enabled: false
          autoCompactionMode: revision
          autoCompactionRetention: "1000"
          enabled: true
          extraEnvVars:
          - name: ETCD_QUOTA_BACKEND_BYTES
            value: "4294967296"
          - name: ETCD_HEARTBEAT_INTERVAL
            value: "500"
          - name: ETCD_ELECTION_TIMEOUT
            value: "2500"
          image:
            pullPolicy: IfNotPresent
            repository: repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/etcd
            tag: v3.5.18-r1
          livenessProbe:
            enabled: true
            timeoutSeconds: 10
          name: etcd
          pdb:
            create: false
          persistence:
            accessMode: ReadWriteOnce
            enabled: true
            size: 1Gi
            storageClass: null
          readinessProbe:
            enabled: true
            periodSeconds: 20
            timeoutSeconds: 10
          replicaCount: 3
          service:
            peerPort: 2380
            port: 2379
            type: ClusterIP
    kafka:
      external: false
    msgStreamType: pulsar
    natsmq:
      persistence:
        persistentVolumeClaim:
          spec: null
    pulsar:
      endpoint: my-milvus-pulsar-proxy.milvus:6650
      external: false
      inCluster:
        chartVersion: pulsar-v3
        deletionPolicy: Retain
        values:
          autorecovery:
            affinity:
              anti_affinity: false
            component: recovery
            configData:
              BOOKIE_MEM: |
                -Xms128m -Xmx128m
              PULSAR_PREFIX_useV2WireProtocol: "true"
            podMonitor:
              enabled: false
            replicaCount: 1
            resources:
              requests:
                cpu: 0.1
                memory: 128Mi
          bookkeeper:
            affinity:
              anti_affinity: false
            component: bookie
            configData:
              PULSAR_GC: |
                -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem
              PULSAR_MEM: |
                -Xms4096m -Xmx4096m -XX:MaxDirectMemorySize=8192m
              nettyMaxFrameSizeBytes: "104867840"
            pdb:
              usePolicy: false
            podManagementPolicy: Parallel
            podMonitor:
              enabled: false
            replicaCount: 3
            resources:
              requests:
                cpu: 0.5
                memory: 2048Mi
            updateStrategy:
              type: RollingUpdate
            volumes:
              journal:
                local_storage: false
                name: journal
                size: 1Gi
              ledgers:
                local_storage: false
                name: ledgers
                size: 1Gi
              persistence: true
          broker:
            affinity:
              anti_affinity: false
            autoscaling:
              enabled: false
            component: broker
            configData:
              PULSAR_GC: |
                -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem
              PULSAR_MEM: |
                -Xms4096m -Xmx4096m -XX:MaxDirectMemorySize=8192m
              backlogQuotaDefaultLimitGB: "8"
              backlogQuotaDefaultRetentionPolicy: producer_exception
              defaultRetentionSizeInMB: "-1"
              defaultRetentionTimeInMinutes: "10080"
              managedLedgerDefaultAckQuorum: "2"
              managedLedgerDefaultEnsembleSize: "3"
              managedLedgerDefaultWriteQuorum: "3"
              maxMessageSize: "104857600"
              subscriptionExpirationTimeMinutes: "3"
              ttlDurationDefaultInSeconds: "259200"
            pdb:
              usePolicy: false
            podMonitor:
              enabled: false
            replicaCount: 2
            resources:
              requests:
                cpu: 0.5
                memory: 2048Mi
          components:
            autorecovery: true
            bookkeeper: true
            broker: true
            functions: false
            proxy: true
            pulsar_manager: false
            toolset: false
            zookeeper: true
          enabled: true
          extra: {}
          images:
            autorecovery:
              pullPolicy: IfNotPresent
              repository: repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/pulsar
              tag: 3.0.7
            bookie:
              pullPolicy: IfNotPresent
              repository: repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/pulsar
              tag: 3.0.7
            broker:
              pullPolicy: IfNotPresent
              repository: repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/pulsar
              tag: 3.0.7
            proxy:
              pullPolicy: IfNotPresent
              repository: repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/pulsar
              tag: 3.0.7
            zookeeper:
              pullPolicy: IfNotPresent
              repository: repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/pulsar
              tag: 3.0.7
          kube-prometheus-stack:
            crds:
              enabled: false
            enabled: false
            grafana:
              enabled: false
            prometheus:
              enabled: false
          monitoring: {}
          name: pulsar
          nameOverride: ""
          persistence: true
          proxy:
            affinity:
              anti_affinity: false
            autoscaling:
              enabled: false
            component: proxy
            configData:
              PULSAR_GC: |
                -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem
              PULSAR_MEM: |
                -Xms512m -Xmx512m -XX:MaxDirectMemorySize=2048m
              httpNumThreads: "8"
            pdb:
              usePolicy: false
            podMonitor:
              enabled: false
            ports:
              containerPorts:
                http: 8080
                https: 8443
              http: 80
              https: 443
              pulsar: 6650
              pulsarssl: 6651
            replicaCount: 2
            resources:
              requests:
                cpu: 0.5
                memory: 1024Mi
            service:
              annotations: {}
              type: ClusterIP
          pulsar_metadata:
            component: pulsar-init
            image:
              pullPolicy: IfNotPresent
              repository: repo.xxbank.cn/baseimage-xclxc-uat-local/milvus/pulsar
              tag: 3.0.7
          volumes:
            local_storage: false
            persistence: true
          zookeeper:
            affinity:
              anti_affinity: false
            component: zookeeper
            configData:
              PULSAR_GC: |
                -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -Dcom.sun.management.jmxremote -Djute.maxbuffer=10485760 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:+DisableExplicitGC -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem
              PULSAR_MEM: |
                -Xms256m -Xmx256m
            pdb:
              usePolicy: false
            podManagementPolicy: Parallel
            podMonitor:
              enabled: false
            replicaCount: 3
            resources:
              requests:
                cpu: 0.2
                memory: 256Mi
            updateStrategy:
              type: RollingUpdate
            volumes:
              data:
                local_storage: false
                name: data
                size: 2Gi
              persistence: true
    rocksmq:
      persistence:
        persistentVolumeClaim:
          spec: null
    storage:
      endpoint: milvus-minio-hl.minio.svc.cluster.local:9000
      # endpoint: minio.xxbank.cn:443
      external: true
      type: S3
      secretRef: "my-minio-secret"
  hookConfig: null
  mode: cluster

yhmo · 2025-06-05T07:58:02Z

yhmo
Jun 5, 2025
Collaborator

mixcoord = rootcoord + indexcoord + datacoord + querycoord
所以如果有mixcoord的话，其他所有带有"coord"字样的节点都不需要。

minio证书的问题还是请 @LoveEachDay @haorenfsa 来看一下。

0 replies

kiyor · 2025-06-06T00:12:36Z

kiyor
Jun 6, 2025

这一步 “6、通过minio的mc工具也可以验证证书的正确性” 你是在哪里做的, 也是在mixcoord还是别的pod上. 这也许是个功能需求 #27461

1 reply

Aragorn1026 Jun 6, 2025
Author

你好，mixcoord中没有mc工具，是在milvus的namespace内创建了一个包含mc工具的临时pod，然后在该pod内进行连接测试的

Aragorn1026 · 2025-06-06T01:35:16Z

Aragorn1026
Jun 6, 2025
Author

issue#27461中提到了config.verifySSL: false，该版本比较古老，测试了下当前版本milvus2.5.12和milvus-operator1.2.6中无效，是否有其他方式可以忽略证书或者其他什么方法呢？

0 replies

kiyor · 2025-06-09T20:19:35Z

kiyor
Jun 9, 2025

这两天又测试了一下, 方向大致定位不是milvus本身的问题, 本身调用了minio客户端, 理论上调用就能跑通ssl, 很有可能是milvus-operator里的check有问题, 我继续追一下.

8 replies

kiyor Jun 17, 2025

@Aragorn1026 最新版的operator已经支持了, 你可以参考这个配置https://github.com/zilliztech/milvus-operator/blob/main/config/samples/milvus_external_minio_ssl.yaml

Aragorn1026 Jun 18, 2025
Author

@Aragorn1026 最新版的operator已经支持了, 你可以参考这个配置https://github.com/zilliztech/milvus-operator/blob/main/config/samples/milvus_external_minio_ssl.yaml

@kiyor 你好，我更新milvus-operator版本为1.3.0-rc1后，按链接进行milvus实例的配置，卡到创建不出协调器这一步，describe milvus显示storage这里有SSL CA Ceritificate secret not exist，我检查创建了指定的configMap：minio-ca-cert，secret只有minio的accessKey和secretKey。milvus的yaml中，除了链接中指定的部分外，还有什么需要调整的吗？

Aragorn1026 Jun 19, 2025
Author

@kiyor 你好，这是我用新版本部署时，遇到的SSL CA certificate secret not exist问题，目前是无法创建出各种node和mixcoord，麻烦看下，谢谢~

这是创建milvus-operator的yaml:
milvus-deployment.yaml.txt
这是创建milvus实例的yaml
milvus-demo9.yaml.txt
这个describe milvus和milvus-operator的部分日志
k8s-get-describe.txt

kiyor Jun 19, 2025

了解, 我再研究一下, 需要点时间

kiyor Jun 20, 2025

@Aragorn1026 你的配置把ca放进了configmap, 在milvus配置中, 只要ca存在于/etc/ssl/certs/minio-ca.crt这个路径即可, 不要关注是secret还是configmap, 所以你的配置没有问题. 但在operator这里caCertificateRef: minio-ca-cert, 不是通过configmap获取ca, 而是强制使用secret使用ca, 这边我的文档似乎写的有问题. 我会更新一下.
简单的结论是, 你把configmap.minio-ca-cert复制到secret.minio-ca-cert,一样的ca.crt, 就可以实现了.
另外建议更新operator到1.2.7稳定版, 我是这个版本测试的.
最佳实践是

---
apiVersion: v1
data:
  ca.crt: <base64 ca cert>
kind: Secret
metadata:
  name: minio-ca-cert
  namespace: milvus
type: Opaque
---
apiVersion: milvus.io/v1beta1
kind: Milvus
...
spec:
  components:
    env:
    - name: SSL_CERT_FILE
      value: /etc/ssl/certs/minio-ca.crt
    volumeMounts:
    - mountPath: /etc/ssl/certs/minio-ca.crt
      name: minio-ca-cert
      readOnly: true
      subPath: ca.crt
    volumes:
    - name: minio-ca-cert
      secret:
        secretName: minio-ca-cert
...
    storage:
      endpoint: minio.minio.svc.cluster.local:443
      external: true
      secretRef: minio-credentials
      ssl:
        caCertificateRef: minio-ca-cert <这里是secret不是configmap>
        enabled: true
        insecureSkipVerify: false
      type: MinIO

k8s中创建milvus实例后协调器连接minio失败 #42540

Uh oh!

Aragorn1026 Jun 5, 2025

Replies: 4 comments · 9 replies

Uh oh!

yhmo Jun 5, 2025 Collaborator

Uh oh!

kiyor Jun 6, 2025

Uh oh!

Aragorn1026 Jun 6, 2025 Author

Uh oh!

Aragorn1026 Jun 6, 2025 Author

Uh oh!

kiyor Jun 9, 2025

Uh oh!

kiyor Jun 17, 2025

Uh oh!

Aragorn1026 Jun 18, 2025 Author

Uh oh!

Aragorn1026 Jun 19, 2025 Author

Uh oh!

kiyor Jun 19, 2025

Uh oh!

Uh oh!

kiyor Jun 20, 2025

Aragorn1026
Jun 5, 2025

Replies: 4 comments 9 replies

yhmo
Jun 5, 2025
Collaborator

kiyor
Jun 6, 2025

Aragorn1026 Jun 6, 2025
Author

Aragorn1026
Jun 6, 2025
Author

kiyor
Jun 9, 2025

Aragorn1026 Jun 18, 2025
Author

Aragorn1026 Jun 19, 2025
Author