单机milvus-v2.5.1版本，原本都正常使用的，数据也不多，用着用着就报这个了database space exceeded #41539

jhyever · 2025-04-27T02:26:03Z

jhyever
Apr 27, 2025

@yhmo 麻烦大佬帮忙看下，感谢，之前也有遇到过，增加了之后过几天还是报这个错呢
数据量也不多啊，就报这个error="etcdserver: mvcc: database space exceeded"

open pid file: /run/milvus/standalone.pid
lock pid file: /run/milvus/standalone.pid
[2025/04/27 02:17:41.026 +00:00] [INFO] [roles/roles.go:345] ["starting running Milvus components"]
[2025/04/27 02:17:41.026 +00:00] [INFO] [roles/roles.go:187] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/:/milvus/lib/libjemalloc.so]
[2025/04/27 02:17:41.037 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]
[2025/04/27 02:17:41.042 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]
[2025/04/27 02:17:41.042 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
{"level":"info","ts":"2025-04-27T02:17:41.043Z","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2025-04-27T02:17:41.044Z","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2025-04-27T02:17:41.044Z","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.5","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.22.0","go-os":"linux","go-arch":"amd64","max-cpu-set":32,"max-cpu-available":32,"member-initialized":true,"name":"default","data-dir":"/var/lib/milvus/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/milvus/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://0.0.0.0:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":[""],"host-whitelist":[""],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"1µs","auto-compaction-interval":"1µs","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2025-04-27T02:17:50.739Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/milvus/etcd/member/snap/db","took":"9.693845132s"}
{"level":"info","ts":"2025-04-27T02:17:52.367Z","caller":"etcdserver/server.go:509","msg":"recovered v2 store from snapshot","snapshot-index":700007,"snapshot-size":"7.1 kB"}
{"level":"info","ts":"2025-04-27T02:17:52.367Z","caller":"etcdserver/server.go:522","msg":"recovered v3 backend from snapshot","backend-size-bytes":2147463168,"backend-size":"2.1 GB","backend-size-in-use-bytes":2147442688,"backend-size-in-use":"2.1 GB"}
{"level":"info","ts":"2025-04-27T02:17:52.801Z","caller":"etcdserver/raft.go:529","msg":"restarting local member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","commit-index":722232}
{"level":"info","ts":"2025-04-27T02:17:52.804Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
{"level":"info","ts":"2025-04-27T02:17:52.804Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became follower at term 2"}
{"level":"info","ts":"2025-04-27T02:17:52.804Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8e9e05c52164694d [peers: [8e9e05c52164694d], term: 2, commit: 722232, applied: 700007, lastindex: 722232, lastterm: 2]"}
{"level":"info","ts":"2025-04-27T02:17:52.804Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2025-04-27T02:17:52.804Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","recovered-remote-peer-id":"8e9e05c52164694d","recovered-remote-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2025-04-27T02:17:52.805Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
{"level":"warn","ts":"2025-04-27T02:17:52.806Z","caller":"auth/store.go:1233","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2025-04-27T02:17:57.844Z","caller":"mvcc/kvstore.go:393","msg":"kvstore restored","current-rev":172923}
{"level":"info","ts":"2025-04-27T02:17:57.851Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2025-04-27T02:17:57.854Z","caller":"etcdserver/server.go:845","msg":"starting etcd server","local-member-id":"8e9e05c52164694d","local-server-version":"3.5.5","cluster-id":"cdf818194e3a8c32","cluster-version":"3.5"}
{"level":"info","ts":"2025-04-27T02:17:57.854Z","caller":"etcdserver/server.go:738","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"8e9e05c52164694d","forward-ticks":9,"forward-duration":"900ms","election-ticks":10,"election-timeout":"1s"}
{"level":"info","ts":"2025-04-27T02:17:57.856Z","caller":"embed/etcd.go:275","msg":"now serving peer/client/metrics","local-member-id":"8e9e05c52164694d","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://0.0.0.0:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2025-04-27T02:17:57.856Z","caller":"embed/etcd.go:584","msg":"serving peer traffic","address":"127.0.0.1:2380"}
{"level":"info","ts":"2025-04-27T02:17:57.856Z","caller":"embed/etcd.go:556","msg":"cmux::serve","address":"127.0.0.1:2380"}
[2025/04/27 02:17:57.856 +00:00] [INFO] [etcd/etcd_server.go:58] ["finish init Etcd config"] [path=/milvus/configs/embedEtcd.yaml] [data=/var/lib/milvus/etcd]
[2025/04/27 02:17:57.857 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]
[2025/04/27 02:17:57.857 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]
[2025/04/27 02:17:57.857 +00:00] [INFO] [roles/roles.go:294] [setupPrometheusHTTPServer]
[2025/04/27 02:17:57.857 +00:00] [INFO] [http/server.go:240] ["management listen"] [addr=:9091]
[2025/04/27 02:17:57.859 +00:00] [INFO] [gc/gc_tuner.go:138] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=45425464934]
[2025/04/27 02:17:57.859 +00:00] [INFO] [rootcoord/root_coord.go:160] ["update rootcoord state"] [state=Abnormal]
[2025/04/27 02:17:57.859 +00:00] [INFO] [rootcoord/service.go:154] ["RootCoord listen on"] [address="[::]:53100"] [port=53100]
[2025/04/27 02:17:57.859 +00:00] [INFO] [rootcoord/service.go:184] ["init params done.."]
[2025/04/27 02:17:57.859 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/04/27 02:17:57.860 +00:00] [INFO] [rootcoord/service.go:204] ["etcd connect done ..."]
[2025/04/27 02:17:57.860 +00:00] [INFO] [rootcoord/service.go:265] ["start grpc "] [port=53100]
[2025/04/27 02:17:57.860 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/04/27 02:17:57.861 +00:00] [INFO] [datacoord/service.go:99] ["DataCoord listen on"] [address="[::]:13333"] [port=13333]
[2025/04/27 02:17:57.861 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/04/27 02:17:57.861 +00:00] [INFO] [datacoord/server.go:1213] ["register metrics actions finished"]
[2025/04/27 02:17:57.861 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2025/04/27 02:17:57.861 +00:00] [INFO] [components/index_coord.go:42] ["IndexCoord running ..."]
[2025/04/27 02:17:57.862 +00:00] [INFO] [querycoord/service.go:104] ["QueryCoord listen on"] [address="[::]:19531"] [port=19531]
[2025/04/27 02:17:57.862 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/04/27 02:17:57.863 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/04/27 02:17:57.865 +00:00] [INFO] [roles/roles.go:106] ["Clean local data cache"] [path=/var/lib/milvus/data/mmap]
[2025/04/27 02:17:57.865 +00:00] [INFO] [querynode/service.go:102] ["QueryNode listen on"] [address="[::]:21123"] [port=21123]
[2025/04/27 02:17:57.866 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/04/27 02:17:57.866 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/04/27 02:17:57.867 +00:00] [INFO] [datanode/service.go:102] ["DataNode listen on"] [address="[::]:21124"] [port=21124]
[2025/04/27 02:17:57.867 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/04/27 02:17:57.867 +00:00] [INFO] [datanode/service.go:257] ["DataNode address"] [address=172.17.0.2:21124]
[2025/04/27 02:17:57.867 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/04/27 02:17:57.868 +00:00] [INFO] [indexnode/service.go:78] ["IndexNode listen on"] [address="[::]:21121"] [port=21121]
[2025/04/27 02:17:57.868 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/04/27 02:17:57.869 +00:00] [INFO] [proxy/look_aside_balancer.go:233] ["Start check query node health loop"]
[2025/04/27 02:17:57.869 +00:00] [INFO] [hookutil/hook.go:70] ["empty so path, skip to load plugin"]
[2025/04/27 02:17:57.870 +00:00] [INFO] [vecindexmgr/vector_index_mgr.go:124] ["init vector indexes with features : BINFLAT : 720897,BIN_FLAT : 720897,BIN_IVF_FLAT : 524289,DISKANN : 2097166,FLAT : 720910,HNSW : 1572910,HNSWLIB_DEPRECATED : 1572879,HNSW_PQ : 524334,HNSW_PRQ : 524334,HNSW_SQ : 524334,IVFBIN : 524289,IVFFLAT : 524302,IVFFLATCC : 14,IVFPQ : 524302,IVFSQ : 524302,IVF_FLAT : 524302,IVF_FLAT_CC : 14,IVF_PQ : 524302,IVF_SQ : 524302,IVF_SQ8 : 524302,IVF_SQ_CC : 14,SCANN : 524302,SPARSE_INVERTED_INDEX : 524304,SPARSE_INVERTED_INDEX_CC : 524304,SPARSE_WAND : 524304,SPARSE_WAND_CC : 524304,"]
[2025/04/27 02:17:57.871 +00:00] [INFO] [proxy/listener_manager.go:53] ["Proxy listen on external grpc listener"] [address=172.17.0.2:19530] [port=19530]
[2025/04/27 02:17:57.871 +00:00] [INFO] [proxy/listener_manager.go:63] ["Proxy listen on internal grpc listener"] [address=172.17.0.2:19529] [port=19529]
[2025/04/27 02:17:57.871 +00:00] [INFO] [proxy/listener_manager.go:98] ["Proxy server(http) and external grpc server share the same port"]
[2025/04/27 02:17:57.871 +00:00] [INFO] [proxy/service.go:411] ["init Proxy server"]
[2025/04/27 02:17:57.871 +00:00] [INFO] [proxy/service.go:431] ["Proxy init service's parameter table done"]
[2025/04/27 02:17:57.871 +00:00] [INFO] [proxy/service.go:433] ["Proxy init http server's parameter table done"]
[2025/04/27 02:17:57.872 +00:00] [INFO] [accesslog/global.go:146] ["Init access logger success"]
[2025/04/27 02:17:57.872 +00:00] [INFO] [proxy/service.go:437] ["init Proxy's tracer done"] ["service name"="Proxy ip: 172.17.0.2, port: 19530"]
[2025/04/27 02:17:57.872 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/04/27 02:17:57.872 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/04/27 02:17:57.872 +00:00] [INFO] [proxy/service.go:387] ["create Proxy internal grpc server"] ["enforcement policy"="{"MinTime":5000000000,"PermitWithoutStream":true}"] ["server parameters"="{"MaxConnectionIdle":0,"MaxConnectionAge":0,"MaxConnectionAgeGrace":0,"Time":60000000000,"Timeout":10000000000}"]
[2025/04/27 02:17:57.872 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
{"level":"info","ts":"2025-04-27T02:17:57.873Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d no leader at term 2; dropping index reading msg"}
[2025/04/27 02:17:57.960 +00:00] [INFO] [rootcoord/service.go:221] ["grpc init done ..."]
[2025/04/27 02:17:57.961 +00:00] [INFO] [rootcoord/service.go:224] ["RootCoord start to create DataCoord client"]
[2025/04/27 02:17:57.966 +00:00] [INFO] [querynodev2/server.go:294] ["register metrics actions finished"]
[2025/04/27 02:17:57.966 +00:00] [INFO] [querynodev2/server.go:304] ["QueryNode session info"] [metaPath=by-dev/meta]
[2025/04/27 02:17:57.968 +00:00] [INFO] [datanode/service.go:266] ["initializing RootCoord client for DataNode"]
[2025/04/27 02:17:57.969 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/04/27 02:17:57.969 +00:00] [INFO] [indexnode/indexnode.go:214] ["IndexNode init"] [state=Initializing]
{"level":"info","ts":"2025-04-27T02:17:58.305Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d is starting a new election at term 2"}
{"level":"info","ts":"2025-04-27T02:17:58.305Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became pre-candidate at term 2"}
{"level":"info","ts":"2025-04-27T02:17:58.305Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgPreVoteResp from 8e9e05c52164694d at term 2"}
{"level":"info","ts":"2025-04-27T02:17:58.305Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became candidate at term 3"}
{"level":"info","ts":"2025-04-27T02:17:58.305Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3"}
{"level":"info","ts":"2025-04-27T02:17:58.305Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became leader at term 3"}
{"level":"info","ts":"2025-04-27T02:17:58.305Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3"}
{"level":"warn","ts":"2025-04-27T02:17:58.306Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"432.996546ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"","error":"etcdserver: leader changed"}
{"level":"info","ts":"2025-04-27T02:17:58.306Z","caller":"traceutil/trace.go:171","msg":"trace[1435231681] range","detail":"{range_begin:health; range_end:; }","duration":"433.505284ms","start":"2025-04-27T02:17:57.872Z","end":"2025-04-27T02:17:58.306Z","steps":["trace[1435231681] 'agreement among raft nodes before linearized reading' (duration: 432.983637ms)"],"step_count":1}
[2025/04/27 02:17:58.306 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="etcdserver: leader changed"]
{"level":"info","ts":"2025-04-27T02:17:58.307Z","caller":"etcdserver/server.go:2054","msg":"published local member to cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http://0.0.0.0:2379]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-timeout":"7s"}
{"level":"info","ts":"2025-04-27T02:17:58.307Z","caller":"embed/serve.go:100","msg":"ready to serve client requests"}
{"level":"warn","ts":"2025-04-27T02:17:58.307Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"346.861071ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"range_response_count:0 size:6"}
{"level":"warn","ts":"2025-04-27T02:17:58.308Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"339.527294ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"range_response_count:0 size:6"}
{"level":"warn","ts":"2025-04-27T02:17:58.308Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"341.631379ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"range_response_count:0 size:6"}
{"level":"info","ts":"2025-04-27T02:17:58.308Z","caller":"traceutil/trace.go:171","msg":"trace[2091278540] range","detail":"{range_begin:health; range_end:; response_count:0; response_revision:172923; }","duration":"346.908791ms","start":"2025-04-27T02:17:57.961Z","end":"2025-04-27T02:17:58.308Z","steps":["trace[2091278540] 'agreement among raft nodes before linearized reading' (duration: 346.768017ms)"],"step_count":1}
{"level":"info","ts":"2025-04-27T02:17:58.308Z","caller":"traceutil/trace.go:171","msg":"trace[2088362134] range","detail":"{range_begin:health; range_end:; response_count:0; response_revision:172923; }","duration":"341.654321ms","start":"2025-04-27T02:17:57.966Z","end":"2025-04-27T02:17:58.308Z","steps":["trace[2088362134] 'agreement among raft nodes before linearized reading' (duration: 341.615327ms)"],"step_count":1}
{"level":"info","ts":"2025-04-27T02:17:58.308Z","caller":"traceutil/trace.go:171","msg":"trace[2127301141] range","detail":"{range_begin:health; range_end:; response_count:0; response_revision:172923; }","duration":"339.576846ms","start":"2025-04-27T02:17:57.968Z","end":"2025-04-27T02:17:58.308Z","steps":["trace[2127301141] 'agreement among raft nodes before linearized reading' (duration: 339.45079ms)"],"step_count":1}
{"level":"warn","ts":"2025-04-27T02:17:58.308Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"338.406913ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"range_response_count:0 size:6"}
{"level":"warn","ts":"2025-04-27T02:17:58.308Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"343.875015ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"range_response_count:0 size:6"}
{"level":"info","ts":"2025-04-27T02:17:58.308Z","caller":"traceutil/trace.go:171","msg":"trace[367823818] range","detail":"{range_begin:health; range_end:; response_count:0; response_revision:172923; }","duration":"338.466318ms","start":"2025-04-27T02:17:57.969Z","end":"2025-04-27T02:17:58.308Z","steps":["trace[367823818] 'agreement among raft nodes before linearized reading' (duration: 338.359824ms)"],"step_count":1}
{"level":"info","ts":"2025-04-27T02:17:58.308Z","caller":"traceutil/trace.go:171","msg":"trace[1055165004] range","detail":"{range_begin:health; range_end:; response_count:0; response_revision:172923; }","duration":"343.940188ms","start":"2025-04-27T02:17:57.964Z","end":"2025-04-27T02:17:58.308Z","steps":["trace[1055165004] 'agreement among raft nodes before linearized reading' (duration: 343.830757ms)"],"step_count":1}
[2025/04/27 02:17:58.308 +00:00] [INFO] [rootcoord/service.go:233] ["RootCoord start to create QueryCoord client"]
{"level":"info","ts":"2025-04-27T02:17:58.309Z","caller":"embed/serve.go:146","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"[::]:2379"}
{"level":"warn","ts":"2025-04-27T02:17:58.309Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"18.275µs","request":"header:<ID:7587886376602281221 > txn:<compare:<key:"by-dev/meta/session/id" version:0 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}
{"level":"warn","ts":"2025-04-27T02:17:58.309Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.23µs","request":"header:<ID:7587886376602281222 > txn:<compare:<key:"by-dev/meta/session/id" version:0 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}
[2025/04/27 02:17:58.309 +00:00] [INFO] [querycoordv2/server.go:249] ["QueryCoord start init"] [meta-root-path=by-dev/meta] [address=172.17.0.2:19531]
[2025/04/27 02:17:58.309 +00:00] [INFO] [querycoordv2/server.go:244] ["register metrics actions finished"]
[2025/04/27 02:17:58.309 +00:00] [WARN] [client/client.go:104] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/04/27 02:17:58.309 +00:00] [WARN] [grpcclient/client.go:262] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/04/27 02:17:58.309 +00:00] [INFO] [rootcoord/root_coord.go:513] ["register metrics actions finished"]
[2025/04/27 02:17:58.309 +00:00] [WARN] [grpcclient/client.go:473] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/04/27 02:17:58.309 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2025/04/27 02:17:58.309 +00:00] [WARN] [grpcclient/client.go:494] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:493\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:128\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:486\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:589\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:135\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n | \t/workspace/source/internal/distributed/datanode/service.go:273\n | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n | \t/workspace/source/internal/distributed/datanode/service.go:188\n | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n | \t/workspace/source/cmd/components/data_node.go:59\n | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n | \t/workspace/source/cmd/roles/roles.go:130\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
{"level":"warn","ts":"2025-04-27T02:17:58.310Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.96µs","request":"header:<ID:7587886376602281226 > txn:<compare:<target:VALUE key:"by-dev/meta/session/id" value_size:1 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}
[2025/04/27 02:17:58.310 +00:00] [WARN] [sessionutil/session_util.go:392] ["Session Txn failed"] [key=id] [error="etcdserver: mvcc: database space exceeded"]
panic: etcdserver: mvcc: database space exceeded

goroutine 473 gp=0xc002f4c700 m=3 mp=0xc0000ed008 [running]:
panic({0x6635b60?, 0xc002f69440?})
/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/panic.go:779 +0x158 fp=0xc010c07350 sp=0xc010c072a0 pc=0x1fd6b18
github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Init(0xc010ade340, {0x6a4776d, 0x9}, {0xc001c28d30, 0x10}, 0xdd?, 0x2?)
/workspace/source/internal/util/sessionutil/session_util.go:292 +0x451 fp=0xc010c074d0 sp=0xc010c07350 pc=0x5093cf1
github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).initSession(0xc00d95c000)
/workspace/source/internal/querynodev2/server.go:164 +0xf1 fp=0xc010c075b8 sp=0xc010c074d0 pc=0x5cb02f1
github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1()
/workspace/source/internal/querynodev2/server.go:305 +0x14b fp=0xc010c07a78 sp=0xc010c075b8 pc=0x5cb134b
sync.(*Once).doSlow(0x74a68c0?, 0xc000d80e10?)
/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/sync/once.go:74 +0xc2 fp=0xc010c07ad8 sp=0xc010c07a78 pc=0x203c062
sync.(*Once).Do(...)
/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/sync/once.go:65
github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init(0xc00d95c000)
/workspace/source/internal/querynodev2/server.go:301 +0x72 fp=0xc010c07b28 sp=0xc010c07ad8 pc=0x5cb11d2
github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init(0xc00d9588a0)
/workspace/source/internal/distributed/querynode/service.go:144 +0x793 fp=0xc010c07e50 sp=0xc010c07b28 pc=0x5cb6f73
github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run(0xc00d9588a0)
/workspace/source/internal/distributed/querynode/service.go:227 +0x1c fp=0xc010c07e90 sp=0xc010c07e50 pc=0x5cb8afc
github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run(0xc00d48d4e8)
/workspace/source/cmd/components/query_node.go:59 +0x2e fp=0xc010c07f70 sp=0xc010c07e90 pc=0x5dbf40e
github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1()
/workspace/source/cmd/roles/roles.go:130 +0xfc fp=0xc010c07fe0 sp=0xc010c07f70 pc=0x5dc88fc
runtime.goexit({})
/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc010c07fe8 sp=0xc010c07fe0 pc=0x2015981
created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] in goroutine 1
/workspace/source/cmd/roles/roles.go:119 +0x129

goroutine 1 gp=0xc0000061c0 m=nil [semacquire]:
runtime.gopark(0x4a05532eca5194?, 0x5dc8d49?, 0x80?, 0xa1?, 0xc018d14080?)
/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/proc.go:402 +0xce fp=0xc001b2f560 sp=0xc001b2f540 pc=0x1fdb28e
runtime.goparkunlock(...)
/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/proc.go:408
runtime.semacquire1(0xc00488a598, 0x0, 0x1, 0x0, 0x12)
/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/sema.go:160 +0x225 fp=0xc001b2f5c8 sp=0xc001b2f560 pc=0x1fee805
sync.runtime_Semacquire(0xc00488a590?)
/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/sema.go:62 +0x25 fp=0xc001b2f600 sp=0xc001b2f5c8 pc=0x20110e5
sync.(*WaitGroup).Wait(0x6300e80?)
/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/sync/waitgroup.go:116 +0x48 fp=0xc001b2f628 sp=0xc001b2f600 pc=0x203d648
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc000d818b0)
/workspace/source/cmd/roles/roles.go:493 +0x11bb fp=0xc001b2fba8 sp=0xc001b2f628 pc=0x5dc617b
github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0xa20d020, {0xc0001a2810, 0x3, 0x3}, 0xc000d888c0)

yhmo · 2025-04-27T06:11:20Z

yhmo
Apr 27, 2025
Collaborator

查看一下volumes/etcd这个目录占用了多少磁盘空间？
docker-compose.yaml文件里etcd的环境变量应该有那么几项：

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.16
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000

如果你的docker-compose.yaml里没有这几项，就补上。如果有，就把ETCD_QUOTA_BACKEND_BYTES设置为8589934592，然后docker compose restart重启

6 replies

yhmo Apr 28, 2025
Collaborator

我怀疑那个quota-backend-bytes: 8589934592没有实际设置到etcd。etcd默认的quota-backend-bytes是2GB，当磁盘用量超过2GB就会报database space exceeded。你截图里的etcd占用了2.4GB，如果它还报这个错的话，我怀疑它仍用着默认设置。

jhyever Apr 28, 2025
Author

用的都是正常配置的啊，不知道为啥就还是启动不起来

yhmo Apr 28, 2025
Collaborator

你用文本编辑器把standalone-embed.sh里的这行

-e ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml \

改成

-e ETCD_CONFIG_FILE=/milvus/configs/embedEtcd.yaml \

然后再重启看看

xiaofan-luan Apr 28, 2025
Maintainer

etcd盘满了，我建议你要检查几个问题

是否有频繁flush等操作？
etcd的清理是否在进行，也就是ETCD_AUTO_COMPACTION_MODE是否有打开？

jhyever Apr 29, 2025
Author

1、我进到容器里看是生效的啊

2、我都没有暴露flush接口调用呢
3、如何看ETCD_AUTO_COMPACTION_MODE有没有打开呀，配置里配的看的是在容器里是有的

xiaofan-luan · 2025-04-29T05:13:11Z

xiaofan-luan
Apr 29, 2025
Maintainer

目前看起来，你这个etcd的大小远远大于数据的大小，意味着你有很多的元信息。

一般就是两种可能：

etcd的gc没有正常工作
存在大量小文件/segment，一般都是因为flush导致的

你可以考虑调用etcd的defrag，看看会不会有效果。如果数据没有降低下去，可以考虑用birdwatcher备份一份meta，我们可以帮忙分析一下

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

单机milvus-v2.5.1版本，原本都正常使用的，数据也不多，用着用着就报这个了database space exceeded #41539

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

单机milvus-v2.5.1版本，原本都正常使用的，数据也不多，用着用着就报这个了database space exceeded #41539

Uh oh!

Uh oh!

jhyever Apr 27, 2025

Replies: 2 comments · 6 replies

Uh oh!

Uh oh!

yhmo Apr 27, 2025 Collaborator

Uh oh!

yhmo Apr 28, 2025 Collaborator

Uh oh!

jhyever Apr 28, 2025 Author

Uh oh!

yhmo Apr 28, 2025 Collaborator

Uh oh!

xiaofan-luan Apr 28, 2025 Maintainer

Uh oh!

jhyever Apr 29, 2025 Author

Uh oh!

xiaofan-luan Apr 29, 2025 Maintainer

jhyever
Apr 27, 2025

Replies: 2 comments 6 replies

yhmo
Apr 27, 2025
Collaborator

yhmo Apr 28, 2025
Collaborator

jhyever Apr 28, 2025
Author

yhmo Apr 28, 2025
Collaborator

xiaofan-luan Apr 28, 2025
Maintainer

jhyever Apr 29, 2025
Author

xiaofan-luan
Apr 29, 2025
Maintainer