集群部署时，minio pod总是失败 #40234

qin-hua-20230331 · 2025-02-27T07:24:23Z

qin-hua-20230331
Feb 27, 2025

其他pod都是好的，只有minio pod 总是有几个失败的。

查看目录/export 权限, 发现每一个pod里的/export 目录，似乎不是共享的，改了minio-0 的chmod -R 777 /export的权限，再查看 minio-1 的/export目录，还是老的权限。

想问下各位有没有头绪，到底是啥问题。。。

EKS信息:

Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.28.15-eks-8cce635

部署的yaml

yaml文件从 https://milvus.io/tools/sizing 获取

apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
  name: my-release
  namespace: milvus-operator
  labels:
    app: milvus
spec:
  components:
    mixCoord:
      replicas: 2
      resources:
        limits:
          cpu: 2
          memory: 8Gi
    indexNode:
      replicas: 2
      resources:
        limits:
          cpu: 4
          memory: 16Gi
    proxy:
      replicas: 2
      resources:
        limits:
          cpu: 2
          memory: 8Gi
  config: {}
  dependencies:
    etcd:
      inCluster:
        values:
          autoCompactionMode: revision
          autoCompactionRetention: "1000"
          extraEnvVars:
          - name: ETCD_QUOTA_BACKEND_BYTES
            value: "4294967296"
          - name: ETCD_HEARTBEAT_INTERVAL
            value: "500"
          - name: ETCD_ELECTION_TIMEOUT
            value: "25000"
          - name: ETCD_SNAPSHOT_COUNT
            value: "10000"
          - name: ETCD_ENABLE_PPROF
            value: "true"
          persistence:
            accessMode: ReadWriteOnce
            enabled: true
            size: 20Gi   #SSD Required
            storageClass: "efs-sc"
          replicaCount: 3
          resources:
            limits:
              cpu: 1
              memory: 4Gi
            requests:
              cpu: 1
              memory: 4Gi

    msgStreamType: kafka
    kafka:
      inCluster:
        values:
          persistence:
            enabled: true
            storageClass: "efs-sc"
            accessMode: ReadWriteOnce
            size: 400Gi
          resources:
            limits:
              cpu: 2
              memory: 8Gi
          zookeeper:
            enabled: true
            replicaCount: 3
            persistence:
              enabled: true
              storageClass: "efs-sc"
              accessModes:
                - ReadWriteOnce
              size: 30Gi #SSD Required
            resources:
              limits:
                cpu: 0.5
                memory: 2Gi

    storage:
      inCluster:
        values:
          mode: distributed
          resources:
            limits:
              cpu: 1
              memory: 8Gi
          persistence:
            storageClass: "efs-sc"
            accessMode: ReadWriteMany
            size: 300Gi

错误日志

my-release-minio-0 的pod是成功，但里面的日志如下:

API: SYSTEM()

Time: 07:03:57 UTC 02/27/2025

DeploymentID: xxxxxxxx

Error: Get "http://my-release-minio-3.my-release-minio-svc.milvus-operator.svc.cluster.local:9000/minio/health/live": dial tcp 192.178.14.19:9000: connect: connection refused (*url.Error)

       5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()

       4: internal/logger/logonce.go:149:logger.LogOnceIf()

       3: cmd/admin-server-info.go:64:cmd.getLocalServerProperty()

       2: cmd/peer-rest-server.go:358:cmd.(*peerRESTServer).ServerInfoHandler()

       1: net/http/server.go:2109:http.HandlerFunc.ServeHTTP()

Waiting for all MinIO sub-systems to be initialized.. trying to acquire lock

API: SYSTEM()

Time: 07:04:27 UTC 02/27/2025

DeploymentID: xxxxxxxx

Error: Get "http://my-release-minio-2.my-release-minio-svc.milvus-operator.svc.cluster.local:9000/minio/health/live": dial tcp 192.178.174.28:9000: connect: connection refused (*url.Error)

       5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()

       4: internal/logger/logonce.go:149:logger.LogOnceIf()

       3: cmd/admin-server-info.go:64:cmd.getLocalServerProperty()

       2: cmd/peer-rest-server.go:358:cmd.(*peerRESTServer).ServerInfoHandler()

       1: net/http/server.go:2109:http.HandlerFunc.ServeHTTP()

API: SYSTEM()

Time: 07:04:27 UTC 02/27/2025

DeploymentID: 0593b6cc-a415-477d-93d0-cc2ba7c74658

Error: Get "http://my-release-minio-3.my-release-minio-svc.milvus-operator.svc.cluster.local:9000/minio/health/live": dial tcp 192.178.14.19:9000: connect: connection refused (*url.Error)

       5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()

       4: internal/logger/logonce.go:149:logger.LogOnceIf()

       3: cmd/admin-server-info.go:64:cmd.getLocalServerProperty()

       2: cmd/peer-rest-server.go:358:cmd.(*peerRESTServer).ServerInfoHandler()

       1: net/http/server.go:2109:http.HandlerFunc.ServeHTTP()

Waiting for all MinIO sub-systems to be initialized.. trying to acquire lock

失败pod 的错误日志:

my-release-minio-2和my-release-minio-3一直失败

Time: 06:57:16 UTC 02/27/2025

Error: Marking my-release-minio-3.my-release-minio-svc.milvus-operator.svc.cluster.local:9000 offline temporarily; caused by Post "http://my-release-minio-3.my-release-minio-svc.milvus-operator.svc.cluster.local:9000/minio/storage/export/v49/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 192.178.14.19:9000: connect: connection refused (*fmt.wrapError)

       8: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()

       7: internal/logger/logonce.go:149:logger.LogOnceIf()

       6: internal/rest/client.go:259:rest.(*Client).Call()

       5: cmd/storage-rest-client.go:164:cmd.(*storageRESTClient).call()

       4: cmd/storage-rest-client.go:551:cmd.(*storageRESTClient).ReadAll()

       3: cmd/format-erasure.go:391:cmd.loadFormatErasure()

       2: cmd/format-erasure.go:327:cmd.loadFormatErasureAll.func1()

       1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

API: SYSTEM()

Time: 06:57:16 UTC 02/27/2025

Error: Drive http://my-release-minio-1.my-release-minio-svc.milvus-operator.svc.cluster.local:9000/export, path (.minio.sys/pool.bin) returned an error (file access denied) (*fmt.wrapError)

      13: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()

      12: internal/logger/logonce.go:149:logger.LogOnceIf()

      11: cmd/erasure-object.go:560:cmd.readAllXL()

      10: cmd/erasure-object.go:628:cmd.erasureObjects.getObjectFileInfo()

       9: cmd/erasure-object.go:227:cmd.erasureObjects.GetObjectNInfo()

       8: cmd/erasure-sets.go:761:cmd.(*erasureSets).GetObjectNInfo()

       7: cmd/config-common.go:38:cmd.readConfigWithMetadata()

       6: cmd/config-common.go:59:cmd.readConfig()

       5: cmd/erasure-server-pool-decom.go:384:cmd.(*poolMeta).load()

       4: cmd/erasure-server-pool-decom.go:497:cmd.(*erasureServerPools).Init()

       3: cmd/erasure-server-pool.go:146:cmd.newErasureServerPools()

       2: cmd/server-main.go:785:cmd.newObjectLayer()

       1: cmd/server-main.go:611:cmd.serverMain()

API: SYSTEM()

Time: 06:57:16 UTC 02/27/2025

Error: Drive /export, path (.minio.sys/pool.bin) returned an error (file access denied) (*fmt.wrapError)

      13: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()

      12: internal/logger/logonce.go:149:logger.LogOnceIf()

      11: cmd/erasure-object.go:560:cmd.readAllXL()

      10: cmd/erasure-object.go:628:cmd.erasureObjects.getObjectFileInfo()

       9: cmd/erasure-object.go:227:cmd.erasureObjects.GetObjectNInfo()

       8: cmd/erasure-sets.go:761:cmd.(*erasureSets).GetObjectNInfo()

       7: cmd/config-common.go:38:cmd.readConfigWithMetadata()

       6: cmd/config-common.go:59:cmd.readConfig()

       5: cmd/erasure-server-pool-decom.go:384:cmd.(*poolMeta).load()

       4: cmd/erasure-server-pool-decom.go:497:cmd.(*erasureServerPools).Init()

       3: cmd/erasure-server-pool.go:146:cmd.newErasureServerPools()

       2: cmd/server-main.go:785:cmd.newObjectLayer()

       1: cmd/server-main.go:611:cmd.serverMain()

ERROR Unable to initialize backend: Prefix access is denied: .minio.sys/pool.bin

PVC 状态都是Bound, 没有异常

Answered by LoveEachDay

Feb 27, 2025

@qin-hua-20230331 看起来是efs-sc provision 的pv 的权限不对，另外minio 不推荐使用efs 来作为底层存储。
这里都建议使用块存储作为minio/kafka/etcd 的存储。

View full answer

LoveEachDay · 2025-02-27T08:43:55Z

LoveEachDay
Feb 27, 2025
Maintainer

@qin-hua-20230331 看起来是efs-sc provision 的pv 的权限不对，另外minio 不推荐使用efs 来作为底层存储。
这里都建议使用块存储作为minio/kafka/etcd 的存储。

1 reply

qin-hua-20230331 Feb 27, 2025
Author

@qin-hua-20230331 看起来是efs-sc provision 的pv 的权限不对，另外minio 不推荐使用efs 来作为底层存储。这里都建议使用块存储作为minio/kafka/etcd 的存储。

好的，谢谢，我来试试gp3看看

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

集群部署时，minio pod总是失败 #40234

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

集群部署时，minio pod总是失败 #40234

Uh oh!

Uh oh!

qin-hua-20230331 Feb 27, 2025

EKS信息:

部署的yaml

错误日志

my-release-minio-0 的pod是成功，但里面的日志如下:

失败pod 的错误日志:

PVC 状态都是Bound, 没有异常

Replies: 1 comment · 1 reply

Uh oh!

LoveEachDay Feb 27, 2025 Maintainer

Uh oh!

qin-hua-20230331 Feb 27, 2025 Author

qin-hua-20230331
Feb 27, 2025

Replies: 1 comment 1 reply

LoveEachDay
Feb 27, 2025
Maintainer

qin-hua-20230331 Feb 27, 2025
Author