Data insertion completed, error loading into memory #24432

ingale726 · 2023-05-26T05:23:51Z

ingale726
May 26, 2023

datanode error:

[2023/05/26 05:11:26.496 +00:00] [INFO] [datacoord/services.go:282] ["received request to get collection statistics"] [traceID=638935b0837e33d2] [module=DataCoord]
[2023/05/26 05:11:26.496 +00:00] [INFO] [datacoord/services.go:295] ["success to get collection statistics"] [traceID=638935b0837e33d2] [module=DataCoord] [response="stats:<key:\"row_count\" value:\"147991460\" > status:<> "]
[2023/05/26 05:11:31.298 +00:00] [INFO] [datacoord/services.go:282] ["received request to get collection statistics"] [traceID=6e4bb6b873dfbbd6] [module=DataCoord]
[2023/05/26 05:11:31.298 +00:00] [INFO] [datacoord/services.go:295] ["success to get collection statistics"] [traceID=6e4bb6b873dfbbd6] [module=DataCoord] [response="stats:<key:\"row_count\" value:\"147991460\" > status:<> "]
[2023/05/26 05:11:36.545 +00:00] [INFO] [datacoord/services.go:282] ["received request to get collection statistics"] [traceID=45d7bf20607d7424] [module=DataCoord]
[2023/05/26 05:11:36.545 +00:00] [INFO] [datacoord/services.go:295] ["success to get collection statistics"] [traceID=45d7bf20607d7424] [module=DataCoord] [response="stats:<key:\"row_count\" value:\"147991460\" > status:<> "]
[2023/05/26 05:11:41.926 +00:00] [INFO] [datacoord/services.go:282] ["received request to get collection statistics"] [traceID=2b6e460c6008b037] [module=DataCoord]
[2023/05/26 05:11:52.446 +00:00] [INFO] [datacoord/services.go:282] ["received request to get collection statistics"] [traceID=3adee3e56fbef477] [module=DataCoord]
[2023/05/26 05:12:02.445 +00:00] [INFO] [datacoord/services.go:282] ["received request to get collection statistics"] [traceID=d8a62d81c2b3252] [module=DataCoord]
{"level":"warn","ts":"2023-05-26T05:12:07.932Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002e7340/139.9.248.174:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2023/05/26 05:12:07.933 +00:00] [DEBUG] [etcd/etcd_kv.go:794] [tr/putEtcdMeta] [msg="putEtcdMeta done"] [duration=30.000370482s]
[2023/05/26 05:12:07.933 +00:00] [WARN] [etcd/etcd_kv.go:729] ["Slow etcd operation save"] ["time spent"=30.000395912s] [key=by-dev/meta/datacoord-meta/channel-cp/by-dev-rootcoord-dml_5_441708466037026885v1]
[2023/05/26 05:12:07.933 +00:00] [WARN] [datacoord/services.go:1208] ["failed to UpdateChannelCheckpoint"] [vChannel=by-dev-rootcoord-dml_5_441708466037026885v1] [error="context deadline exceeded"]
[2023/05/26 05:12:07.933 +00:00] [INFO] [datacoord/services.go:295] ["success to get collection statistics"] [traceID=3adee3e56fbef477] [module=DataCoord] [response="stats:<key:\"row_count\" value:\"147991460\" > status:<> "]
[2023/05/26 05:12:07.933 +00:00] [WARN] [datacoord/metrics_info.go:167] ["invalid metrics of DataNode was found"] [error="context deadline exceeded"]
[2023/05/26 05:12:07.933 +00:00] [INFO] [datacoord/services.go:295] ["success to get collection statistics"] [traceID=d8a62d81c2b3252] [module=DataCoord] [response="stats:<key:\"row_count\" value:\"147991460\" > status:<> "]
[2023/05/26 05:12:07.933 +00:00] [WARN] [datacoord/metrics_info.go:167] ["invalid metrics of DataNode was found"] [error="context deadline exceeded"]
[2023/05/26 05:12:07.933 +00:00] [INFO] [datacoord/services.go:295] ["success to get collection statistics"] [traceID=2b6e460c6008b037] [module=DataCoord] [response="stats:<key:\"row_count\" value:\"147991460\" > status:<> "]
[2023/05/26 05:12:07.933 +00:00] [WARN] [datacoord/metrics_info.go:167] ["invalid metrics of DataNode was found"] [error="context canceled"]
[2023/05/26 05:12:07.933 +00:00] [WARN] [datacoord/metrics_info.go:167] ["invalid metrics of DataNode was found"] [error="context canceled"]
[2023/05/26 05:12:08.554 +00:00] [INFO] [datacoord/services.go:742] ["received get flushed segments request"] [collectionID=441708466037026885] [partitionID=-1]
[2023/05/26 05:12:31.990 +00:00] [WARN] [sessionutil/session_util.go:399] ["session keepalive channel closed"]
[2023/05/26 05:12:31.990 +00:00] [WARN] [sessionutil/session_util.go:666] ["connection lost detected, shuting down"]
[2023/05/26 05:12:31.990 +00:00] [ERROR] [datacoord/server.go:232] ["disconnected from etcd and exited"] [serverID=7] [stack="github.com/milvus-io/milvus/internal/datacoord.(*Server).Register.func1\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:232"]
[2023/05/26 05:12:31.990 +00:00] [INFO] [datacoord/server.go:864] ["server shutdown"]
[2023/05/26 05:12:31.990 +00:00] [WARN] [datacoord/garbage_collector.go:106] ["garbage collector quit"]
[2023/05/26 05:12:31.990 +00:00] [INFO] [datacoord/server.go:543] ["DataNode timetick loop shutdown"]
[2023/05/26 05:12:31.990 +00:00] [INFO] [msgstream/mq_msgstream.go:182] ["start to close mq msg stream"] ["producer num"=0] ["consumer num"=1]
[2023/05/26 05:12:31.990 +00:00] [INFO] [datacoord/server.go:781] ["flush loop shutdown"]
[2023/05/26 05:12:31.990 +00:00] [INFO] [datacoord/server.go:697] ["watch service shutdown"]
time="2023-05-26T05:12:32Z" level=info msg="The consumer[1] successfully unsubscribed" consumerID=1 name=sbdmc subscription=by-dev-dataCoord-7-datanodeTl topic="persistent://public/default/by-dev-datacoord-timetick-channel"
time="2023-05-26T05:12:32Z" level=info msg="Closing consumer=1" consumerID=1 name=sbdmc subscription=by-dev-dataCoord-7-datanodeTl topic="persistent://public/default/by-dev-datacoord-timetick-channel"
time="2023-05-26T05:12:32Z" level=info msg="[Closed consumer]" consumerID=1 name=sbdmc subscription=by-dev-dataCoord-7-datanodeTl topic="persistent://public/default/by-dev-datacoord-timetick-channel"
[2023/05/26 05:12:32.093 +00:00] [WARN] [timerecord/time_recorder.go:128] ["long term checker [dataTtChecker] shutdown"]
{"level":"warn","ts":"2023-05-26T05:12:33.093Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002e7340/139.9.248.174:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2023/05/26 05:12:33.093 +00:00] [INFO] [datacoord/compaction_trigger.go:150] ["global compaction loop exit"]
{"level":"warn","ts":"2023-05-26T05:12:37.934Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002e7340/139.9.248.174:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2023/05/26 05:12:37.934 +00:00] [DEBUG] [etcd/etcd_kv.go:794] [tr/putEtcdMeta] [msg="putEtcdMeta done"] [duration=30.000699148s]
[2023/05/26 05:12:37.934 +00:00] [WARN] [etcd/etcd_kv.go:729] ["Slow etcd operation save"] ["time spent"=30.000735832s] [key=by-dev/meta/datacoord-meta/channel-cp/by-dev-rootcoord-dml_4_441708466037026885v0]
[2023/05/26 05:12:37.934 +00:00] [WARN] [datacoord/services.go:1208] ["failed to UpdateChannelCheckpoint"] [vChannel=by-dev-rootcoord-dml_4_441708466037026885v0] [error="context deadline exceeded"]
[2023/05/26 05:12:37.934 +00:00] [WARN] [datacoord/metrics_info.go:82] ["fails to get DataNode metrics"] [error="client is disposed"]
[2023/05/26 05:12:37.934 +00:00] [WARN] [datacoord/metrics_info.go:82] ["fails to get DataNode metrics"] [error="client is disposed"]
[2023/05/26 05:12:37.934 +00:00] [WARN] [datacoord/metrics_info.go:82] ["fails to get DataNode metrics"] [error="client is disposed"]
[2023/05/26 05:12:37.934 +00:00] [WARN] [datacoord/metrics_info.go:82] ["fails to get DataNode metrics"] [error="client is disposed"]
[2023/05/26 05:12:37.934 +00:00] [DEBUG] [log/mlogger.go:63] [DataCoord.GetMetrics] [traceID=1e50dcb4fcda43b1] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [metricType=system_info] [metrics="status:<> response:\"{\\\"cluster\\\":{\\\"self\\\":{\\\"has_error\\\":false,\\\"error_reason\\\":\\\"\\\",\\\"name\\\":\\\"datacoord7\\\",\\\"hardware_infos\\\":{\\\"ip\\\":\\\"192.168.0.190:13333\\\",\\\"cpu_core_count\\\":32,\\\"cpu_core_usage\\\":0.8560945784202948,\\\"memory\\\":270307250176,\\\"memory_usage\\\":315219968,\\\"disk\\\":104857600,\\\"disk_usage\\\":2097152},\\\"system_info\\\":{\\\"system_version\\\":\\\"6313a455\\\",\\\"deploy_mode\\\":\\\"DISTRIBUTED\\\",\\\"build_version\\\":\\\"v2.2.3\\\",\\\"build_time\\\":\\\"Fri Feb 10 09:59:53 UTC 2023\\\",\\\"used_go_version\\\":\\\"go version go1.18.3 linux/amd64\\\"},\\\"created_time\\\":\\\"2023-05-25 03:08:07.638757811 +0000 UTC m=+0.331019178\\\",\\\"updated_time\\\":\\\"2023-05-25 03:08:07.638757903 +0000 UTC m=+0.331019258\\\",\\\"type\\\":\\\"datacoord\\\",\\\"id\\\":7,\\\"system_configurations\\\":{\\\"segment_max_size\\\":512},\\\"quota_metrics\\\":{\\\"TotalBinlogSize\\\":72178150947}},\\\"connected_nodes\\\":[]},\\\"connections\\\":{\\\"name\\\":\\\"datacoord7\\\",\\\"connected_components\\\":[]}}\" component_name:\"datacoord7\" "] []
[2023/05/26 05:12:37.939 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=62391f0abdb437e6] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:12:37.941 +00:00] [INFO] [datacoord/services.go:742] ["received get flushed segments request"] [collectionID=441708466037026885] [partitionID=-1]
[2023/05/26 05:12:39.176 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=6ff3c07df76f01f9] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:12:42.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=7ef1ce83f73301bd] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:12:45.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=dc482f134a7f0f8] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:12:48.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=12b01e8f0575b96f] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:12:51.176 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=43e8c340835387c6] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:12:54.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=7976ab3f1bc307cf] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:12:57.176 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=75c062bad4ca38b5] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:00.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=4d63bddafb8e068f] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:03.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=15db152651ccff49] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:06.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=3a694431a935b547] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
{"level":"warn","ts":"2023-05-26T05:13:07.935Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002e7340/139.9.248.174:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2023/05/26 05:13:07.935 +00:00] [DEBUG] [etcd/etcd_kv.go:794] [tr/putEtcdMeta] [msg="putEtcdMeta done"] [duration=30.000719124s]
[2023/05/26 05:13:07.935 +00:00] [WARN] [etcd/etcd_kv.go:729] ["Slow etcd operation save"] ["time spent"=30.00074717s] [key=by-dev/meta/datacoord-meta/channel-cp/by-dev-rootcoord-dml_4_441708466037026885v0]
[2023/05/26 05:13:07.935 +00:00] [WARN] [datacoord/services.go:1208] ["failed to UpdateChannelCheckpoint"] [vChannel=by-dev-rootcoord-dml_4_441708466037026885v0] [error="context deadline exceeded"]
[2023/05/26 05:13:09.176 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=365cd4454f846058] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:12.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=5f263332908857bf] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:15.176 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=3475949f419ce382] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:18.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=5989a696e9cc92e7] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:21.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=3e1384b124b8f84f] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:24.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=5953ccacba96129e] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:27.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=70342352c2108102] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:30.176 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=3a709bb30d08baa4] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:30.809 +00:00] [WARN] [datacoord/services.go:1201] ["failed to update channel position for closed server"]
[2023/05/26 05:13:30.809 +00:00] [WARN] [datacoord/services.go:1201] ["failed to update channel position for closed server"]
[2023/05/26 05:13:33.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=453ed2926dcbf31e] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
[2023/05/26 05:13:36.177 +00:00] [WARN] [datacoord/services.go:849] ["DataCoord.GetMetrics failed"] [traceID=5d56baf1e2ede852] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [error="DataCoord 7 is not ready"]
{"level":"warn","ts":"2023-05-26T05:13:37.935Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002e7340/139.9.248.174:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2023/05/26 05:13:37.935 +00:00] [DEBUG] [etcd/etcd_kv.go:794] [tr/putEtcdMeta] [msg="putEtcdMeta done"] [duration=30.000124779s]
[2023/05/26 05:13:37.935 +00:00] [WARN] [etcd/etcd_kv.go:729] ["Slow etcd operation save"] ["time spent"=30.000145716s] [key=by-dev/meta/datacoord-meta/channel-cp/by-dev-rootcoord-dml_5_441708466037026885v1]
[2023/05/26 05:13:37.935 +00:00] [WARN] [datacoord/services.go:1208] ["failed to UpdateChannelCheckpoint"] [vChannel=by-dev-rootcoord-dml_5_441708466037026885v1] [error="context deadline exceeded"]
[2023/05/26 05:13:37.935 +00:00] [WARN] [datacoord/metrics_info.go:82] ["fails to get DataNode metrics"] [error="client is disposed"]
[2023/05/26 05:13:37.935 +00:00] [WARN] [datacoord/metrics_info.go:82] ["fails to get DataNode metrics"] [error="client is disposed"]
[2023/05/26 05:13:37.935 +00:00] [DEBUG] [log/mlogger.go:63] [DataCoord.GetMetrics] [traceID=34a9ec263ecfb46a] [nodeID=7] [req="{\"metric_type\":\"system_info\"}"] [metricType=system_info] [metrics="status:<> response:\"{\\\"cluster\\\":{\\\"self\\\":{\\\"has_error\\\":false,\\\"error_reason\\\":\\\"\\\",\\\"name\\\":\\\"datacoord7\\\",\\\"hardware_infos\\\":{\\\"ip\\\":\\\"192.168.0.190:13333\\\",\\\"cpu_core_count\\\":32,\\\"cpu_core_usage\\\":0.6816030297603742,\\\"memory\\\":270307250176,\\\"memory_usage\\\":315219968,\\\"disk\\\":104857600,\\\"disk_usage\\\":2097152},\\\"system_info\\\":{\\\"system_version\\\":\\\"6313a455\\\",\\\"deploy_mode\\\":\\\"DISTRIBUTED\\\",\\\"build_version\\\":\\\"v2.2.3\\\",\\\"build_time\\\":\\\"Fri Feb 10 09:59:53 UTC 2023\\\",\\\"used_go_version\\\":\\\"go version go1.18.3 linux/amd64\\\"},\\\"created_time\\\":\\\"2023-05-25 03:08:07.638757811 +0000 UTC m=+0.331019178\\\",\\\"updated_time\\\":\\\"2023-05-25 03:08:07.638757903 +0000 UTC m=+0.331019258\\\",\\\"type\\\":\\\"datacoord\\\",\\\"id\\\":7,\\\"system_configurations\\\":{\\\"segment_max_size\\\":512},\\\"quota_metrics\\\":{\\\"TotalBinlogSize\\\":72178150947}},\\\"connected_nodes\\\":[]},\\\"connections\\\":{\\\"name\\\":\\\"datacoord7\\\",\\\"connected_components\\\":[]}}\" component_name:\"datacoord7\" "] []
[2023/05/26 05:13:37.936 +00:00] [INFO] [datacoord/services.go:742] ["received get flushed segments request"] [collectionID=441708466037026885] [partitionID=-1]
[2023/05/26 05:13:37.939 +00:00] [INFO] [datacoord/services.go:742] ["received get flushed segments request"] [collectionID=441708466037026885] [partitionID=-1]
[2023/05/26 05:13:37.939 +00:00] [INFO] [datacoord/compaction_trigger.go:119] ["compaction trigger quit"]
[2023/05/26 05:13:37.939 +00:00] [INFO] [datacoord/compaction.go:136] ["compaction handler quit"]
[2023/05/26 05:13:37.939 +00:00] [ERROR] [roles/roles.go:320] ["Get signal to exit\n"] [signal=interrupt] [stack="github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:320\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:112\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
[2023/05/26 05:13:37.939 +00:00] [INFO] [datacoord/service.go:212] ["Datacoord stop"] [Address=192.168.0.190:13333]
[2023/05/26 05:13:37.939 +00:00] [INFO] [datacoord/service.go:225] ["Graceful stop grpc server..."]
[2023/05/26 05:13:37.939 +00:00] [INFO] [datacoord/channel_manager.go:663] ["watch etcd loop quit"]

#################################################################################################
indexnode error:

[2023/05/26 03:41:11.593 +00:00] [DEBUG] [indexnode/task.go:446] ["tr/IndexBuildID: 441708466201592888, ClusterID: by-dev"] [msg="index building all done"] [duration=4.046596482s]
[2023/05/26 03:41:11.593 +00:00] [INFO] [indexnode/task.go:447] ["Successfully save index files"] [buildID=441708466201592888] [Collection=441708466037026885] [partition=441708466037026886] [SegmentId=441708466201592867]
[2023/05/26 03:41:11.593 +00:00] [INFO] [indexnode/taskinfo_ops.go:42] ["IndexNode store task state"] [clusterID=by-dev] [buildID=441708466201592888] [state=Finished] ["fail reason"=]
[2023/05/26 03:41:12.553 +00:00] [DEBUG] [indexnode/indexnode_service.go:158] ["querying index build task"] [traceID=789768fb7d6ce82d] [ClusterID=by-dev] [IndexBuildID=441708466201592888] [state=Finished] ["fail reason"=]
[2023/05/26 03:41:13.553 +00:00] [INFO] [indexnode/indexnode_service.go:167] ["drop index build jobs"] [traceID=579eee61749416a6] [ClusterID=by-dev] [IndexBuildIDs="[441708466201592888]"]
[2023/05/26 03:41:13.553 +00:00] [DEBUG] [indexnode/indexnode_service.go:186] ["drop index build jobs success"] [traceID=579eee61749416a6] [ClusterID=by-dev] [IndexBuildIDs="[441708466201592888]"]
[2023/05/26 05:07:49.492 +00:00] [DEBUG] [log/mlogger.go:63] [IndexNode.GetMetrics] [traceID=6583a244b7703ad] [nodeID=2] [req="{\"metric_type\":\"system_info\"}"] [metric_type=system_info] []
{"level":"warn","ts":"2023-05-26T05:09:51.618Z","logger":"etcd-client","caller":"v3@v3.5.5/lease.go:533","msg":"lease keepalive response queue is full; dropping response send","queue-size":16,"queue-capacity":16}
{"level":"warn","ts":"2023-05-26T05:09:51.618Z","logger":"etcd-client","caller":"v3@v3.5.5/lease.go:533","msg":"lease keepalive response queue is full; dropping response send","queue-size":16,"queue-capacity":16}
{"level":"warn","ts":"2023-05-26T05:09:51.618Z","logger":"etcd-client","caller":"v3@v3.5.5/lease.go:533","msg":"lease keepalive response queue is full; dropping response send","queue-size":16,"queue-capacity":16}
{"level":"warn","ts":"2023-05-26T05:11:42.393Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00052a540/139.9.248.174:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2023/05/26 05:11:42.393 +00:00] [ERROR] [config/etcd_source.go:151] ["can not pull configs"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/config.(*EtcdSource).refreshConfigurationsPeriodically\n\t/go/src/github.com/milvus-io/milvus/internal/config/etcd_source.go:151"]
[2023/05/26 05:11:43.073 +00:00] [INFO] [config/etcd_source.go:155] ["stop refreshing configurations"]
[2023/05/26 05:11:50.659 +00:00] [WARN] [sessionutil/session_util.go:399] ["session keepalive channel closed"]
[2023/05/26 05:11:50.659 +00:00] [WARN] [sessionutil/session_util.go:666] ["connection lost detected, shuting down"]
[2023/05/26 05:11:50.659 +00:00] [ERROR] [indexnode/indexnode.go:135] ["Index Node disconnected from etcd, process will exit"] ["Server Id"=2] [stack="github.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Register.func1\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:135"]
[2023/05/26 05:11:50.660 +00:00] [INFO] [indexnode/indexnode.go:240] ["Index node stopping"]
[2023/05/26 05:13:45.730 +00:00] [INFO] [indexnode/indexnode.go:250] ["Index node abnormal"]
[2023/05/26 05:13:45.733 +00:00] [INFO] [indexnode/indexnode.go:264] ["Index node stopped."]
{"level":"warn","ts":"2023-05-26T05:13:45.733Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00052a700/139.9.248.174:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: requested lease not found"}
[2023/05/26 05:13:45.733 +00:00] [ERROR] [roles/roles.go:320] ["Get signal to exit\n"] [signal=interrupt] [stack="github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:320\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:112\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
[2023/05/26 05:13:45.733 +00:00] [INFO] [indexnode/service.go:199] ["IndexNode stop"] [Address=192.168.0.190:21121]
[2023/05/26 05:13:45.733 +00:00] [INFO] [indexnode/service.go:213] ["Graceful stop grpc server..."]

yhmo · 2023-05-26T06:34:46Z

yhmo
May 26, 2023
Collaborator

All these errors indicate the Etcd server is down:

[2023/05/26 05:12:31.990 +00:00] [WARN] [sessionutil/session_util.go:666] ["connection lost detected, shuting down"]
[2023/05/26 05:12:31.990 +00:00] [ERROR] [datacoord/server.go:232] ["disconnected from etcd and exited"] [

[2023/05/26 05:11:42.393 +00:00] [ERROR] [config/etcd_source.go:151] ["can not pull configs"] [error="context deadline exceeded"]
[2023/05/26 05:11:50.659 +00:00] [ERROR] [indexnode/indexnode.go:135] ["Index Node disconnected from etcd, process will exit"]

6 replies

yhmo May 26, 2023
Collaborator

You can check the log of the etcd server.

FiveTreesHigh Mar 20, 2025

I met this problem too. I deployed milvus using docker(https://milvus.io/docs/zh/install_standalone-docker.md). Different from deploying milvus using docker-compose, I could only find one container when milvus starts:

How can I get the logs of etcd in this situation?

yhmo Mar 20, 2025
Collaborator

Docker standalone only has one container, etcd is embedded in the milvus container.
The doc already mentioned:

Export logs by

docker logs [milvus_container_id] > 1.log

FiveTreesHigh Mar 20, 2025

Ok, it means logs exported by docker logs [milvus_container_id] > 1.log include all of the logs generated by etcd,minio and standalone?

yhmo Mar 20, 2025
Collaborator

I think so.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Data insertion completed, error loading into memory #24432

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Data insertion completed, error loading into memory #24432

Uh oh!

ingale726 May 26, 2023

Replies: 1 comment · 6 replies

Uh oh!

yhmo May 26, 2023 Collaborator

Uh oh!

yhmo May 26, 2023 Collaborator

Uh oh!

FiveTreesHigh Mar 20, 2025

Uh oh!

yhmo Mar 20, 2025 Collaborator

Uh oh!

FiveTreesHigh Mar 20, 2025

Uh oh!

yhmo Mar 20, 2025 Collaborator

ingale726
May 26, 2023

Replies: 1 comment 6 replies

yhmo
May 26, 2023
Collaborator

yhmo May 26, 2023
Collaborator

yhmo Mar 20, 2025
Collaborator

yhmo Mar 20, 2025
Collaborator