Mix coordinator unhealthy in the distributed provision mode.

[jubingc]

Hi,

This is a new distributed cluster provision. The mix coordinator is always unhealthy. The debug logs show that rootcoord is the unhealthy component. Not able to connect to the root coordinator.

{"level":"INFO","time":"2025/04/18 20:29:28.720 +00:00","caller":"healthz/healthz_handler.go:116","message":"check health failed","UnhealthyComponent":["rootcoord"[]}

[2025/04/18 20:27:49.354 +00:00] [WARN[] [resolver/watch_based_grpc_resolver.go:54] ["fail to update resolver state"[] [component=grpc-resolver] [scheme=milvus-session] [role=by-dev/meta/session/rootcoord] [exclusive=true] [id=1] [state="{\"Addresses\":null,\"Endpoints\":null,\"ServiceConfig\":null,\"Attributes\":null}"] [error="bad resolver state"]

{"level":"WARN","time":"2025/04/18 20:29:18.791 +00:00","caller":"grpclog/grpclog.go:46","message":"[core[][Channel #15 SubChannel #28[]grpc: addrConn.createTransport failed to connect to {Addr: \"100.65.19.85:53100\", ServerName: \"rootcoord\", Attributes: {\"<%!p(attributes.attributesKeyType=0)>\": \"<%!p(int64=19)>\" }, }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 100.65.19.85:53100: connect: connection refused\""}

{"level":"DEBUG","time":"2025/04/18 20:29:48.001 +00:00","caller":"rootcoord/root_coord.go:857","message":"RootCoord current state","traceID":"90b45613fa1e524d15ea14732fc6c97c","StateCode":"Initializing"}
{"level":"WARN","time":"2025/04/18 20:29:48.001 +00:00","caller":"retry/retry.go:46","message":"retry func failed","retried":4,"error":"WaitForComponentStates, not meet, RootCoord current state: Initializing"}

But logs also show that root coordinator started successfully.

{"level":"INFO","time":"2025/04/18 20:27:49.501 +00:00","caller":"rootcoord/service.go:175","message":"RootCoord start done ..."}
{"level":"INFO","time":"2025/04/18 20:27:49.501 +00:00","caller":"components/root_coord.go:62","message":"RootCoord successfully started"}

The debug log is attached.
ns-db-milvus-aias-stg-milvus-aias-stg-mixcoord-59598d9b4c-f7sbf-1745008786854638000.log

Can I have some insights on what may go wrong?