内存足够，但是报etcdserver no space启动不起来，milvus版本2.5.1 #42048

jhyever · 2025-05-23T08:14:13Z

jhyever
May 23, 2025

@yhmo 麻烦大佬帮忙看下，感谢
正常的文档里面的片段生成向量1024纬，正常入库，

配置的8589934592，也是生效的，但是etcd数据到2.5G就启动不起来了，
而且配置是生效的
TotalMem: 1081825304576
UsedMem: 39337984

open pid file: /run/milvus/standalone.pid
lock pid file: /run/milvus/standalone.pid
[2025/05/23 08:04:11.213 +00:00] [INFO] [roles/roles.go:345] ["starting running Milvus components"]
[2025/05/23 08:04:11.213 +00:00] [INFO] [roles/roles.go:187] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/:/milvus/lib/libjemalloc.so]
[2025/05/23 08:04:11.220 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]
[2025/05/23 08:04:11.222 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2025/05/23 08:04:11.222 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]
{"level":"info","ts":"2025-05-23T08:04:11.223Z","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2025-05-23T08:04:11.223Z","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2025-05-23T08:04:11.223Z","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.5","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.22.0","go-os":"linux","go-arch":"amd64","max-cpu-set":128,"max-cpu-available":128,"member-initialized":true,"name":"default","data-dir":"/var/lib/milvus/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/milvus/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://0.0.0.0:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":[""],"host-whitelist":[""],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":8589934592,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"1ns","auto-compaction-interval":"1ns","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2025-05-23T08:04:18.114Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/milvus/etcd/member/snap/db","took":"6.890591518s"}
{"level":"info","ts":"2025-05-23T08:04:19.073Z","caller":"etcdserver/server.go:509","msg":"recovered v2 store from snapshot","snapshot-index":2100021,"snapshot-size":"8.1 kB"}
{"level":"info","ts":"2025-05-23T08:04:19.074Z","caller":"etcdserver/server.go:522","msg":"recovered v3 backend from snapshot","backend-size-bytes":2147487744,"backend-size":"2.1 GB","backend-size-in-use-bytes":2147450880,"backend-size-in-use":"2.1 GB"}
{"level":"info","ts":"2025-05-23T08:04:19.360Z","caller":"etcdserver/raft.go:529","msg":"restarting local member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","commit-index":2152649}
{"level":"info","ts":"2025-05-23T08:04:19.364Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
{"level":"info","ts":"2025-05-23T08:04:19.364Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became follower at term 8"}
{"level":"info","ts":"2025-05-23T08:04:19.364Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8e9e05c52164694d [peers: [8e9e05c52164694d], term: 8, commit: 2152649, applied: 2100021, lastindex: 2152649, lastterm: 8]"}
{"level":"info","ts":"2025-05-23T08:04:19.365Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2025-05-23T08:04:19.365Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","recovered-remote-peer-id":"8e9e05c52164694d","recovered-remote-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2025-05-23T08:04:19.365Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
{"level":"warn","ts":"2025-05-23T08:04:19.365Z","caller":"auth/store.go:1233","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2025-05-23T08:04:27.746Z","caller":"mvcc/kvstore.go:393","msg":"kvstore restored","current-rev":2107209}
{"level":"info","ts":"2025-05-23T08:04:27.747Z","caller":"etcdserver/quota.go:117","msg":"enabled backend quota","quota-name":"v3-applier","quota-size-bytes":8589934592,"quota-size":"8.6 GB"}
{"level":"info","ts":"2025-05-23T08:04:27.747Z","caller":"etcdserver/server.go:845","msg":"starting etcd server","local-member-id":"8e9e05c52164694d","local-server-version":"3.5.5","cluster-id":"cdf818194e3a8c32","cluster-version":"3.5"}
{"level":"info","ts":"2025-05-23T08:04:27.747Z","caller":"etcdserver/server.go:738","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"8e9e05c52164694d","forward-ticks":9,"forward-duration":"900ms","election-ticks":10,"election-timeout":"1s"}
{"level":"info","ts":"2025-05-23T08:04:27.749Z","caller":"embed/etcd.go:584","msg":"serving peer traffic","address":"127.0.0.1:2380"}
{"level":"info","ts":"2025-05-23T08:04:27.749Z","caller":"embed/etcd.go:275","msg":"now serving peer/client/metrics","local-member-id":"8e9e05c52164694d","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://0.0.0.0:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2025-05-23T08:04:27.749Z","caller":"embed/etcd.go:556","msg":"cmux::serve","address":"127.0.0.1:2380"}
[2025/05/23 08:04:27.749 +00:00] [INFO] [etcd/etcd_server.go:58] ["finish init Etcd config"] [path=/milvus/configs/embedEtcd.yaml] [data=/var/lib/milvus/etcd]
[2025/05/23 08:04:27.750 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]
[2025/05/23 08:04:27.750 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]
[2025/05/23 08:04:27.750 +00:00] [INFO] [roles/roles.go:294] [setupPrometheusHTTPServer]
[2025/05/23 08:04:27.750 +00:00] [INFO] [http/server.go:240] ["management listen"] [addr=:9091]
[2025/05/23 08:04:27.752 +00:00] [INFO] [gc/gc_tuner.go:138] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=973642774118]
[2025/05/23 08:04:27.752 +00:00] [INFO] [rootcoord/root_coord.go:160] ["update rootcoord state"] [state=Abnormal]
[2025/05/23 08:04:27.753 +00:00] [INFO] [rootcoord/service.go:154] ["RootCoord listen on"] [address="[::]:53100"] [port=53100]
[2025/05/23 08:04:27.753 +00:00] [INFO] [rootcoord/service.go:184] ["init params done.."]
[2025/05/23 08:04:27.753 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/05/23 08:04:27.753 +00:00] [INFO] [rootcoord/service.go:204] ["etcd connect done ..."]
[2025/05/23 08:04:27.753 +00:00] [INFO] [rootcoord/service.go:265] ["start grpc "] [port=53100]
[2025/05/23 08:04:27.753 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/05/23 08:04:27.754 +00:00] [INFO] [datacoord/service.go:99] ["DataCoord listen on"] [address="[::]:13333"] [port=13333]
[2025/05/23 08:04:27.755 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/05/23 08:04:27.755 +00:00] [INFO] [datacoord/server.go:1213] ["register metrics actions finished"]
[2025/05/23 08:04:27.755 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2025/05/23 08:04:27.756 +00:00] [INFO] [components/index_coord.go:42] ["IndexCoord running ..."]
[2025/05/23 08:04:27.756 +00:00] [INFO] [querycoord/service.go:104] ["QueryCoord listen on"] [address="[::]:19531"] [port=19531]
[2025/05/23 08:04:27.756 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/05/23 08:04:27.757 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/05/23 08:04:27.758 +00:00] [INFO] [querynode/service.go:102] ["QueryNode listen on"] [address="[::]:21123"] [port=21123]
[2025/05/23 08:04:27.758 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/05/23 08:04:27.758 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/05/23 08:04:27.760 +00:00] [INFO] [datanode/service.go:102] ["DataNode listen on"] [address="[::]:21124"] [port=21124]
[2025/05/23 08:04:27.760 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/05/23 08:04:27.760 +00:00] [INFO] [datanode/service.go:257] ["DataNode address"] [address=172.17.0.2:21124]
[2025/05/23 08:04:27.761 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/05/23 08:04:27.761 +00:00] [INFO] [indexnode/service.go:78] ["IndexNode listen on"] [address="[::]:21121"] [port=21121]
[2025/05/23 08:04:27.762 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/05/23 08:04:27.762 +00:00] [INFO] [proxy/look_aside_balancer.go:233] ["Start check query node health loop"]
[2025/05/23 08:04:27.763 +00:00] [INFO] [hookutil/hook.go:70] ["empty so path, skip to load plugin"]
[2025/05/23 08:04:27.763 +00:00] [INFO] [vecindexmgr/vector_index_mgr.go:124] ["init vector indexes with features : BINFLAT : 720897,BIN_FLAT : 720897,BIN_IVF_FLAT : 524289,DISKANN : 2097166,FLAT : 720910,HNSW : 1572910,HNSWLIB_DEPRECATED : 1572879,HNSW_PQ : 524334,HNSW_PRQ : 524334,HNSW_SQ : 524334,IVFBIN : 524289,IVFFLAT : 524302,IVFFLATCC : 14,IVFPQ : 524302,IVFSQ : 524302,IVF_FLAT : 524302,IVF_FLAT_CC : 14,IVF_PQ : 524302,IVF_SQ : 524302,IVF_SQ8 : 524302,IVF_SQ_CC : 14,SCANN : 524302,SPARSE_INVERTED_INDEX : 524304,SPARSE_INVERTED_INDEX_CC : 524304,SPARSE_WAND : 524304,SPARSE_WAND_CC : 524304,"]
[2025/05/23 08:04:27.765 +00:00] [INFO] [proxy/listener_manager.go:53] ["Proxy listen on external grpc listener"] [address=172.17.0.2:19530] [port=19530]
[2025/05/23 08:04:27.765 +00:00] [INFO] [proxy/listener_manager.go:63] ["Proxy listen on internal grpc listener"] [address=172.17.0.2:19529] [port=19529]
[2025/05/23 08:04:27.765 +00:00] [INFO] [proxy/listener_manager.go:98] ["Proxy server(http) and external grpc server share the same port"]
[2025/05/23 08:04:27.765 +00:00] [INFO] [proxy/service.go:411] ["init Proxy server"]
[2025/05/23 08:04:27.765 +00:00] [INFO] [proxy/service.go:431] ["Proxy init service's parameter table done"]
[2025/05/23 08:04:27.765 +00:00] [INFO] [proxy/service.go:433] ["Proxy init http server's parameter table done"]
[2025/05/23 08:04:27.765 +00:00] [INFO] [accesslog/global.go:146] ["Init access logger success"]
[2025/05/23 08:04:27.765 +00:00] [INFO] [proxy/service.go:437] ["init Proxy's tracer done"] ["service name"="Proxy ip: 172.17.0.2, port: 19530"]
[2025/05/23 08:04:27.765 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/05/23 08:04:27.766 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/05/23 08:04:27.766 +00:00] [INFO] [proxy/service.go:387] ["create Proxy internal grpc server"] ["enforcement policy"="{"MinTime":5000000000,"PermitWithoutStream":true}"] ["server parameters"="{"MaxConnectionIdle":0,"MaxConnectionAge":0,"MaxConnectionAgeGrace":0,"Time":60000000000,"Timeout":10000000000}"]
[2025/05/23 08:04:27.766 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
{"level":"info","ts":"2025-05-23T08:04:27.767Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d no leader at term 8; dropping index reading msg"}
[2025/05/23 08:04:27.854 +00:00] [INFO] [rootcoord/service.go:221] ["grpc init done ..."]
[2025/05/23 08:04:27.854 +00:00] [INFO] [rootcoord/service.go:224] ["RootCoord start to create DataCoord client"]
[2025/05/23 08:04:27.859 +00:00] [INFO] [querynodev2/server.go:294] ["register metrics actions finished"]
[2025/05/23 08:04:27.859 +00:00] [INFO] [querynodev2/server.go:304] ["QueryNode session info"] [metaPath=by-dev/meta]
[2025/05/23 08:04:27.861 +00:00] [INFO] [datanode/service.go:266] ["initializing RootCoord client for DataNode"]
[2025/05/23 08:04:27.862 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/05/23 08:04:27.862 +00:00] [INFO] [indexnode/indexnode.go:214] ["IndexNode init"] [state=Initializing]
{"level":"info","ts":"2025-05-23T08:04:27.921Z","caller":"etcdserver/v3_server.go:832","msg":"first commit in current term: resending ReadIndex request"}
{"level":"info","ts":"2025-05-23T08:04:27.921Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d no leader at term 8; dropping index reading msg"}
{"level":"info","ts":"2025-05-23T08:04:27.921Z","caller":"etcdserver/v3_server.go:832","msg":"first commit in current term: resending ReadIndex request"}
{"level":"info","ts":"2025-05-23T08:04:27.921Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d no leader at term 8; dropping index reading msg"}
{"level":"warn","ts":"2025-05-23T08:04:27.922Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.18µs","request":"header:<ID:7587886956732803077 > txn:<compare:<key:"by-dev/meta/session/id" version:0 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}
{"level":"warn","ts":"2025-05-23T08:04:27.922Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.047µs","request":"header:<ID:7587886956732803078 > txn:<compare:<key:"by-dev/meta/session/id" version:0 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}
{"level":"warn","ts":"2025-05-23T08:04:27.922Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.035µs","request":"header:<ID:7587886956732803081 > txn:<compare:<target:VALUE key:"by-dev/meta/session/id" value_size:1 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}
{"level":"warn","ts":"2025-05-23T08:04:27.922Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.483µs","request":"header:<ID:7587886956732803083 > txn:<compare:<key:"by-dev/meta/session/id" version:0 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}
{"level":"warn","ts":"2025-05-23T08:04:27.922Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.091µs","request":"header:<ID:7587886956872042501 > txn:<compare:<key:"by-dev/meta/session/id" version:0 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}
{"level":"warn","ts":"2025-05-23T08:04:27.922Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"941ns","request":"header:<ID:7587886956872042502 > txn:<compare:<key:"by-dev/meta/session/id" version:0 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}
{"level":"warn","ts":"2025-05-23T08:04:27.922Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"813ns","request":"header:<ID:7587886956872042506 > txn:<compare:<target:VALUE key:"by-dev/meta/session/id" value_size:1 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}
{"level":"info","ts":"2025-05-23T08:04:27.922Z","caller":"etcdserver/v3_server.go:832","msg":"first commit in current term: resending ReadIndex request"}
{"level":"info","ts":"2025-05-23T08:04:27.922Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d no leader at term 8; dropping index reading msg"}
{"level":"warn","ts":"2025-05-23T08:04:28.423Z","caller":"etcdserver/v3_server.go:840","msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":7587886957002040

LoveEachDay · 2025-05-23T08:43:04Z

LoveEachDay
May 23, 2025
Maintainer

@jhyever After updating the quota-backend-bytes setting on your etcd server, you’ll need to manually disarm the quota alarm. You can do this using the following command:

etcdctl alarm disarm --endpoints=<etcd-member-ip>:2379

This step is necessary to clear the alarm state and resume normal write operations.

8 replies

yhmo May 23, 2025
Collaborator

我感觉是embed etcd本身的问题，但我没有证据。
etcd的日志里已经检查到占用的backend-size是2.1GB

{"level":"info","ts":"2025-05-23T08:04:19.074Z","caller":"etcdserver/server.go:522","msg":"recovered v3 backend from snapshot","backend-size-bytes":2147487744,"backend-size":"2.1 GB","backend-size-in-use-bytes":2147450880,"backend-size-in-use":"2.1 GB"}

然后，它也确认了quota-size-bytes确实是设置到了8.6GB

{"level":"info","ts":"2025-05-23T08:04:27.747Z","caller":"etcdserver/quota.go:117","msg":"enabled backend quota","quota-name":"v3-applier","quota-size-bytes":8589934592,"quota-size":"8.6 GB"}

然后也没做什么事情，put请求的value_size=1，居然就no space了

{"level":"warn","ts":"2025-05-23T08:04:27.922Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.18µs","request":"header:<ID:7587886956732803077 > txn:<compare:<key:"by-dev/meta/session/id" version:0 > success:<request_put:<key:"by-dev/meta/session/id" value_size:1 >> failure:<>>","response":"","error":"etcdserver: no space"}

jhyever May 23, 2025
Author

1、都是standalone_embed.sh启动的，和etcd有啥关系，元数据是etcd，但是milvus也不可能这点数据都不支持啊，
2、milvus之前都没遇到过这样么，
3、上述我回复的都没起的来milvus， etcd都没法连接

yhmo May 23, 2025
Collaborator

standalone_embed 使用的是一个内嵌的etcd，也就是这个 https://github.com/etcd-io/etcd/tree/main/server/embed
元数据仍然是由etcd存储，只不过这个etcd server是内嵌于milvus的进程中，milvus各个组件和它的交互依然是以etcd-client调用etcd server的方式。
因为这个内嵌的etcd报错说自己no space了，etcd-client调用它的put请求全都失败，这个etcd server无法工作了，milvus也就无法工作。

我所列举的那些日志全都是这个embed etcd的日志，不是milvus的日志。

jhyever May 23, 2025
Author

那你们肯定也有遇到过这种no space错啊，是怎么解决的，
这种情况遇到后数据怎么恢复呢

yhmo May 23, 2025
Collaborator

Etcd的工程师写的这句和 @LoveEachDay 是一个意思：
etcd-io/etcd#19204 (reply in thread)
"If the NOSPACE alarm has already been triggered, then it won't disappear automatically when you set a bigger storage quota."
一旦这个alarm触发，就算是把quota改大也不会停止。所以你的standalone_embed里的etcd一直报这个错。

现在的问题是这个etcd起不来，就没法通过这个命令来禁止那个alarm:
etcdctl alarm disarm --endpoints=:2379

yhmo · 2025-05-23T11:18:38Z

yhmo
May 23, 2025
Collaborator

@LoveEachDay 建议了一个方案可以尝试一下。

1.先把volumes目录备份好，以防万一，以防万一，以防万一
2. 在standalone_embed.sh同一个目录下创建一个docker-compose.yml文件，内容如下：

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.16
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    ports:
      - "2379:2379"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

networks:
  default:
    name: milvus

可以看到我在这里启动了一个etcd server，使用的数据目录是volumes/milvus/etcd，也就是standalone_embed所创建的etcd数据目录。

启动etcd server

docker compose up -d

用这个命令看一下这个etcd server是能读出那个embed etcd的数据的，你会看到很多带有by-dev前缀的key

etcdctl --endpoints="http://127.0.0.1:2379" get by-dev/ --prefix --keys-only

用这个命令看一下是否有alarm，我觉得应该会显示“alarm:NOSPACE”这样的字样

etcdctl alarm list --endpoints=127.0.0.1:2379

用这个命令去删除alarm

etcdctl alarm disarm --endpoints=127.0.0.1:2379

停止当前这个临时的etcd server

docker compose down

启动原先的standalone_embed milvus，看能否成功

bash standalone_embed.sh start

15 replies

jhyever May 28, 2025
Author

1、那比如我新增一条数据，如上图数据，新增一条，对版本来说，是增加了一个版本碎片麽，我又增加了10条，但是没修改任何数据，都是新增的，这时候会有10个版本碎片麽，假如配置的auto-compaction-retention是10，是不是就可以理解为要处理这10个版本的碎片元数据了；如果这10条数据，我update了5条，版本碎片数量是5，还是多少
2、查询不会增加版本是吧
3、如果一次性新增了50w数据，期间没做任何其他的操作，知识insert，这时候版本是咋体现的

yhmo May 28, 2025
Collaborator

Milvus自己没有revision这个概念。这里说的revision都是etcd的。
etcd为每个key都会维护一组revision，milvus往etcd的某个key里更新值，revision就会加1
milvus维护自己的segments，各个segments的信息，各个collection/partition信息要记录在etcd里。你往milvus里写数据还是改数据，milvus里都会产生segment或者存储delete id的delta_log，这些东西的信息和状态都要更新到etcd的很多key-value里，segment的状态变更都会更新etcd里某个key的值，这些对etcd的操作会在etcd内部产生新的revision。这没这么难理解吧。

jhyever May 28, 2025
Author

"etcd为每个key都会维护一组revision，milvus往etcd的某个key里更新值，revision就会加1"

1、如果就一条数据，如果不更新，是不是版本就是1，也不会+1；
2、revision配置100的话，
如果就一条数据，更改100次，那版本就是有100个，那就是会压缩清理；
如果有100条数据，每个数据更新一次，算版本碎片总数是100个麽，会进行压缩麽

yhmo May 28, 2025
Collaborator

etcd的某个key的数据如果不更新，这个key的revision就不会+1
auto-compaction-retention=100，如果某个key只有1个revision，或者说小于100个revision，就无影响；如果某个key有500个revision，则丢弃从1到400的版本数据，只保留401到500的版本数据。这是我理解的etcd机制。

jhyever May 28, 2025
Author

好的，感谢解答

内存足够，但是报etcdserver no space启动不起来，milvus版本2.5.1 #42048

Uh oh!

Uh oh!

jhyever May 23, 2025

Replies: 2 comments · 23 replies

Uh oh!

Uh oh!

LoveEachDay May 23, 2025 Maintainer

Uh oh!

yhmo May 23, 2025 Collaborator

Uh oh!

Uh oh!

jhyever May 23, 2025 Author

Uh oh!

Uh oh!

yhmo May 23, 2025 Collaborator

Uh oh!

Uh oh!

jhyever May 23, 2025 Author

Uh oh!

yhmo May 23, 2025 Collaborator

Uh oh!

Uh oh!

yhmo May 23, 2025 Collaborator

Uh oh!

Uh oh!

jhyever May 28, 2025 Author

Uh oh!

Uh oh!

yhmo May 28, 2025 Collaborator

Uh oh!

jhyever May 28, 2025 Author

Uh oh!

Uh oh!

yhmo May 28, 2025 Collaborator

Uh oh!

jhyever May 28, 2025 Author

jhyever
May 23, 2025

Replies: 2 comments 23 replies

LoveEachDay
May 23, 2025
Maintainer

yhmo May 23, 2025
Collaborator

jhyever May 23, 2025
Author

yhmo May 23, 2025
Collaborator

jhyever May 23, 2025
Author

yhmo May 23, 2025
Collaborator

yhmo
May 23, 2025
Collaborator

jhyever May 28, 2025
Author

yhmo May 28, 2025
Collaborator

jhyever May 28, 2025
Author

yhmo May 28, 2025
Collaborator

jhyever May 28, 2025
Author