milvus operator部署milvus集群如何设置外部pulsar服务？

[bozai111]

使用的外部pulsar服务,带有认证,想知道下如何配置认证?
externalPulsar:
enabled: true
authPlugin: ''
authParams: ''

如果设置这两个参数?

[xiaofan-luan]

pulsar:
authPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken
authParams: token:xxx

or helm can be done like this

externalPulsar:
enabled: true
host: localhost
port: 6650
tenant: public
namespace: default
authPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken
authParams: token:xxx

[bozai111]

目前看认证通过了，我们预先创建topic1-topic16，从日志中看还有topic not found的报错。帮忙确认下是否还有topic未预先创建。

[bozai111]

目前看认证通过了，我们预先创建topic1-topic16，从日志中看还有topic not found的报错。帮忙确认下是否还有topic未预先创建。

[bozai111]

preCreatedTopic指定的那些topic不需要手动创建，milvus会自动创建
-- 那这个参数只是用来指定milvus使用的topic的名称？感觉作用和预期不一样。

我们目前想使用公司内部的pulsar服务，公司内部的pulsar服务是不支持应用程序自动创建topic的，想借助预先创建topic的方式，来规避这个问题，但是实际看这个参数preCreatedTopic和预期的不一样，仍然要自动创建topic。有没有别的办法可以解决？

[yhmo]

预先创建topic也没问题，这也是已有的功能。
在第4步那个pulsar-manager创建environment后，我在Topics页面里手动创建了topic1和topic2两个topic，然后启动milvus建表查数据搜索都是正常的：

[bozai111]

Subscription does not exist

-- 这个报错看着像订阅时，指定的消费组名称没找到，topic是能正常找到的，Subscription name内部是怎么约定的？

[xiaofan-luan]

Subscription does not exist

-- 这个报错看着像订阅时，指定的消费组名称没找到，topic是能正常找到的，Subscription name内部是怎么约定的？

请发一下milvus日志吧，完整上下文。
看你这个像是pulsar报错，但是信息比较少

[bozai111]

目前从rootcoord节点内，能看到日志只有这么多。我们公司内部pulsar服务内的topic对应的消费组名称，也是要预先定义的，这个在步骤没有并没有说明。目前一直怀疑是消费组名称未定义准确。详细的日志可以看下面的内容。

[bozai111]

2025/03/28 01:18:29 missing source yaml
2025/03/28 01:18:29 useIAM is false
[2025/03/28 01:18:29.635 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"]
2025/03/28 01:18:29 maxprocs: Updating GOMAXPROCS=2: determined from CPU quota

__  _________ _   ____  ______    

/ |/ / / /| | / / / / / __/
/ /|/ // // /| |/ / // /\ \
// /////_/__/

Welcome to use Milvus!
Version: 2.5.7
Built: Sat Mar 22 03:26:11 UTC 2025
GitCommit: 31f9e3c
GoVersion: go version go1.22.7 linux/amd64

TotalMem: 2147483648
UsedMem: 109985792

open pid file: /run/milvus/rootcoord.pid
lock pid file: /run/milvus/rootcoord.pid
[2025/03/28 01:18:29.651 +00:00] [INFO] [roles/roles.go:344] ["starting running Milvus components"]
[2025/03/28 01:18:29.651 +00:00] [INFO] [roles/roles.go:186] ["Enable Jemalloc"] ["Jemalloc Path"=:/milvus/lib/libjemalloc.so]
[2025/03/28 01:18:29.656 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]
[2025/03/28 01:18:29.657 +00:00] [DEBUG] [paramtable/base_table.go:213] ["init etcd source"] [etcdInfo="{"UseEmbed":false,"EnableAuth":false,"UserName":"","PassWord":"","UseSSL":false,"Endpoints":["my-release-etcd-0.my-release-etcd-headless.milvus-test-operator:2379","my-release-etcd-1.my-release-etcd-headless.milvus-test-operator:2379","my-release-etcd-2.my-release-etcd-headless.milvus-test-operator:2379"],"KeyPrefix":"my-release","CertFile":"/path/to/etcd-client.pem","KeyFile":"/path/to/etcd-client-key.pem","CaCertFile":"/path/to/ca.pem","MinVersion":"1.3","RefreshInterval":5000000000}"]
[2025/03/28 01:18:29.657 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-release-etcd-0.my-release-etcd-headless.milvus-test-operator:2379,my-release-etcd-1.my-release-etcd-headless.milvus-test-operator:2379,my-release-etcd-2.my-release-etcd-headless.milvus-test-operator:2379]"] [minVersion=1.3]
[2025/03/28 01:18:29.659 +00:00] [DEBUG] [config/etcd_source.go:92] ["etcd refreshConfigurations"] [prefix=my-release/config] [endpoints="[my-release-etcd-0.my-release-etcd-headless.milvus-test-operator:2379,my-release-etcd-1.my-release-etcd-headless.milvus-test-operator:2379,my-release-etcd-2.my-release-etcd-headless.milvus-test-operator:2379]"]
[2025/03/28 01:18:29.659 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=EtcdSource]
[2025/03/28 01:18:29.660 +00:00] [INFO] [paramtable/component_param.go:4778] ["DeployModeEnv is not set, use default"] [default=0.5]
[2025/03/28 01:18:29.661 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]
[2025/03/28 01:18:29.661 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2025/03/28 01:18:29.661 +00:00] [INFO] [tracer/tracer.go:50] ["Init tracer finished"] [Exporter=noop]
[2025/03/28 01:18:29.661 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-release-etcd-0.my-release-etcd-headless.milvus-test-operator:2379,my-release-etcd-1.my-release-etcd-headless.milvus-test-operator:2379,my-release-etcd-2.my-release-etcd-headless.milvus-test-operator:2379]"] [minVersion=1.3]
[2025/03/28 01:18:29.663 +00:00] [INFO] [resolver/resolver_with_discoverer.go:128] ["service discover task started, listening..."] [component=grpc-resolver] [scheme=milvus-session] [role=my-release/meta/session/rootcoord] [exclusive=true]
[2025/03/28 01:18:29.663 +00:00] [INFO] [resolver/resolver_with_discoverer.go:161] ["service discover update, update resolver"] [component=grpc-resolver] [scheme=milvus-session] [role=my-release/meta/session/rootcoord] [exclusive=true] [state="{"Version":8809,"State":{"Addresses":[],"Endpoints":null,"ServiceConfig":null,"Attributes":null}}"] [resolver_count=0]
[2025/03/28 01:18:29.663 +00:00] [INFO] [resolver/resolver_with_discoverer.go:171] ["update resolver done"] [component=grpc-resolver] [scheme=milvus-session] [role=my-release/meta/session/rootcoord] [exclusive=true]
[2025/03/28 01:18:29.668 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]
[2025/03/28 01:18:29.668 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]
[2025/03/28 01:18:29.668 +00:00] [INFO] [roles/roles.go:293] [setupPrometheusHTTPServer]
[2025/03/28 01:18:29.668 +00:00] [INFO] [resolver/resolver_with_discoverer.go:133] ["new grpc resolver registered"] [component=grpc-resolver] [scheme=milvus-session] [role=my-release/meta/session/rootcoord] [exclusive=true] [id=1]
[2025/03/28 01:18:29.668 +00:00] [WARN] [resolver/watch_based_grpc_resolver.go:54] ["fail to update resolver state"] [component=grpc-resolver] [scheme=milvus-session] [role=my-release/meta/session/rootcoord] [exclusive=true] [id=1] [state="{"Addresses":[],"Endpoints":null,"ServiceConfig":null,"Attributes":null}"] [error="bad resolver state"]
[2025/03/28 01:18:29.668 +00:00] [INFO] [http/server.go:240] ["management listen"] [addr=:9091]
[2025/03/28 01:18:29.668 +00:00] [INFO] [gc/gc_tuner.go:138] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=1932735283]
[2025/03/28 01:18:29.668 +00:00] [INFO] [rootcoord/root_coord.go:165] ["update rootcoord state"] [state=Abnormal]
[2025/03/28 01:18:29.669 +00:00] [INFO] [rootcoord/service.go:155] ["RootCoord listen on"] [address="[::]:53100"] [port=53100]
[2025/03/28 01:18:29.669 +00:00] [INFO] [rootcoord/service.go:185] ["init params done.."]
[2025/03/28 01:18:29.669 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-release-etcd-0.my-release-etcd-headless.milvus-test-operator:2379,my-release-etcd-1.my-release-etcd-headless.milvus-test-operator:2379,my-release-etcd-2.my-release-etcd-headless.milvus-test-operator:2379]"] [minVersion=1.3]
[2025/03/28 01:18:29.670 +00:00] [INFO] [rootcoord/service.go:205] ["etcd connect done ..."]
[2025/03/28 01:18:29.670 +00:00] [INFO] [rootcoord/service.go:219] ["RootCoord start to create DataCoord client"]
[2025/03/28 01:18:29.671 +00:00] [INFO] [rootcoord/service.go:228] ["RootCoord start to create QueryCoord client"]
[2025/03/28 01:18:29.671 +00:00] [INFO] [rootcoord/root_coord.go:524] ["register metrics actions finished"]
[2025/03/28 01:18:29.671 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=false] [mqType=pulsar]
[2025/03/28 01:18:29.725 +00:00] [INFO] [sessionutil/session_util.go:305] ["start server"] [name=rootcoord] [address=10.136.10.97:53100] [id=1375] [server_labels={}]
[2025/03/28 01:18:29.725 +00:00] [INFO] [rootcoord/root_coord.go:165] ["update rootcoord state"] [state=StandBy]
[2025/03/28 01:18:29.725 +00:00] [INFO] [rootcoord/root_coord.go:561] ["RootCoord enter standby mode successfully"]
[2025/03/28 01:18:29.725 +00:00] [INFO] [rootcoord/service.go:239] ["RootCoord init done ..."]
[2025/03/28 01:18:29.725 +00:00] [INFO] [rootcoord/service.go:270] ["start grpc "] [port=53100]
[2025/03/28 01:18:29.725 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/03/28 01:18:29.725 +00:00] [INFO] [coordclient/registry.go:77] ["register root coord server"] [enableLocalClient="{"ServerType":"","EnableQueryCoord":false,"EnableDataCoord":false,"EnableRootCoord":false}"]
[2025/03/28 01:18:29.826 +00:00] [INFO] [rootcoord/service.go:245] ["grpc init done ..."]
[2025/03/28 01:18:29.826 +00:00] [INFO] [rootcoord/service.go:170] ["RootCoord init done ..."]
[2025/03/28 01:18:29.826 +00:00] [INFO] [rootcoord/service.go:319] ["RootCoord Core start ..."]
[2025/03/28 01:18:29.861 +00:00] [INFO] [sessionutil/session_util.go:503] ["put session key into etcd"] [key=my-release/meta/session/rootcoord-1375] [value="{"ServerID":1375,"ServerName":"rootcoord","Address":"10.136.10.97:53100","Exclusive":true,"TriggerKill":true,"Version":"2.5.7","IndexEngineVersion":{},"ScalarIndexEngineVersion":{},"LeaseID":1627934498087346728,"HostName":"my-release-milvus-rootcoord-cb8bd6b57-6jg9z"}"]
[2025/03/28 01:18:29.861 +00:00] [INFO] [sessionutil/session_util.go:513] ["Service registered successfully"] [ServerName=rootcoord] [serverID=1375]
[2025/03/28 01:18:29.861 +00:00] [INFO] [rootcoord/service.go:175] ["RootCoord start done ..."]
[2025/03/28 01:18:29.861 +00:00] [INFO] [components/root_coord.go:62] ["RootCoord successfully started"]
[2025/03/28 01:18:29.861 +00:00] [INFO] [sessionutil/session_util.go:1097] ["serverName: rootcoord enter STANDBY mode"]
[2025/03/28 01:18:29.861 +00:00] [INFO] [sessionutil/session_util.go:1071] ["try to register as ACTIVE rootcoord service..."]
[2025/03/28 01:18:29.868 +00:00] [WARN] [lazygrpc/conn.go:60] ["async dial failed, wait for retry..."] [error="context deadline exceeded"]
[2025/03/28 01:18:29.869 +00:00] [INFO] [sessionutil/session_util.go:1089] ["register ACTIVE rootcoord"]
[2025/03/28 01:18:29.870 +00:00] [INFO] [sessionutil/session_util.go:1142] ["serverName: rootcoord quit STANDBY mode, this node will become ACTIVE, ID: 1375"]
[2025/03/28 01:18:29.870 +00:00] [INFO] [rootcoord/root_coord.go:541] ["RootCoord switch from standby to active, activating"]
[2025/03/28 01:18:29.870 +00:00] [INFO] [rootcoord/root_coord.go:165] ["update rootcoord state"] [state=Initializing]
[2025/03/28 01:18:29.878 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2025/03/28 01:12:02.953 +00:00] [save=2025/03/28 01:18:32.870 +00:00] [next=2025/03/28 01:18:29.870 +00:00]
[2025/03/28 01:18:29.878 +00:00] [INFO] [rootcoord/root_coord.go:414] ["id allocator initialized"] [role=rootcoord] [intent=initInternal] [traceID=6a0529b33978f35c9f4d37968e63724c] [root_path=my-release/kv] [sub_path=gid] [key=idTimestamp]
[2025/03/28 01:18:29.893 +00:00] [INFO] [discoverer/session_discoverer.go:118] ["new server modification"] [prefix=my-release/meta/session/rootcoord] [exclusive=true] [expectedVersion=2.4.0] [event=PUT] [sessionKey=my-release/meta/session/rootcoord] [sessionValue="{"ServerID":1375,"ServerName":"rootcoord","Address":"10.136.10.97:53100","Exclusive":true,"TriggerKill":true,"Version":"2.5.7","IndexEngineVersion":{},"ScalarIndexEngineVersion":{},"LeaseID":1627934498087346728,"HostName":"my-release-milvus-rootcoord-cb8bd6b57-6jg9z"}"]
[2025/03/28 01:18:29.893 +00:00] [INFO] [resolver/resolver_with_discoverer.go:161] ["service discover update, update resolver"] [component=grpc-resolver] [scheme=milvus-session] [role=my-release/meta/session/rootcoord] [exclusive=true] [state="{"Version":8812,"State":{"Addresses":[{"Addr":"10.136.10.97:53100","ServerName":"","Attributes":{"<%!p(attributes.attributesKeyType=0)>": "<%!p(int64=1375)>" },"BalancerAttributes":{"<%!p(attributes.attributesKeyType=2)>": "<0xc002084480>" , "<%!p(attributes.attributesKeyType=0)>": "<%!p(int64=1375)>" },"Metadata":null}],"Endpoints":null,"ServiceConfig":null,"Attributes":null}}"] [resolver_count=1]
[2025/03/28 01:18:29.893 +00:00] [INFO] [resolver/resolver_with_discoverer.go:167] ["resolver is closed, unregister the resolver"] [component=grpc-resolver] [scheme=milvus-session] [role=my-release/meta/session/rootcoord] [exclusive=true] [updateError="resolver is closed"]
[2025/03/28 01:18:29.893 +00:00] [INFO] [resolver/resolver_with_discoverer.go:171] ["update resolver done"] [component=grpc-resolver] [scheme=milvus-session] [role=my-release/meta/session/rootcoord] [exclusive=true]
[2025/03/28 01:18:29.894 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2025/03/28 01:12:02.970 +00:00] [save=2025/03/28 01:18:32.884 +00:00] [next=2025/03/28 01:18:29.884 +00:00]
[2025/03/28 01:18:29.894 +00:00] [INFO] [rootcoord/root_coord.go:438] ["tso allocator initialized"] [role=rootcoord] [intent=initInternal] [traceID=6a0529b33978f35c9f4d37968e63724c] [root_path=my-release/kv] [sub_path=gid] [key=idTimestamp]
[2025/03/28 01:18:29.894 +00:00] [INFO] [rootcoord/root_coord.go:365] ["Using etcd as meta storage."] [role=rootcoord] [intent=initInternal] [traceID=6a0529b33978f35c9f4d37968e63724c]
[2025/03/28 01:18:29.895 +00:00] [INFO] [rootcoord/meta_table.go:167] ["recover databases"] ["num of dbs"=1]
[2025/03/28 01:18:29.896 +00:00] [INFO] [rootcoord/kv_catalog.go:834] ["unmarshal all collection details cost"] [db=0] [cost=1.288µs]
[2025/03/28 01:18:29.898 +00:00] [INFO] [rootcoord/kv_catalog.go:834] ["unmarshal all collection details cost"] [db=1] [cost=559ns]
[2025/03/28 01:18:29.898 +00:00] [INFO] [rootcoord/meta_table.go:218] ["collections recovered from db"] [db_name=default] [collection_num=0] [partition_num=0] [dur=451.438µs]
[2025/03/28 01:18:29.899 +00:00] [INFO] [rootcoord/meta_table.go:235] ["RootCoord meta table reload done"] [duration=4.515364ms]
[2025/03/28 01:18:29.899 +00:00] [INFO] [rootcoord/dml_channels.go:151] ["new DmlChannels"] [role=rootcoord] [intent=initInternal] [traceID=6a0529b33978f35c9f4d37968e63724c] [chanNamePrefixDefault=my-release-rootcoord-dml] [chanNumDefault=0]
[2025/03/28 01:18:29.900 +00:00] [INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true]
[2025/03/28 01:18:29.900 +00:00] [INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true]
[2025/03/28 01:18:29.906 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [role=rootcoord] [intent=initInternal] [traceID=6a0529b33978f35c9f4d37968e63724c] [retried=0] [error="server error: UnknownError: Subscription does not exist"]
[2025/03/28 01:18:30.577 +00:00] [INFO] [resolver/resolver_with_discoverer.go:133] ["new grpc resolver registered"] [component=grpc-resolver] [scheme=milvus-session] [role=my-release/meta/session/rootcoord] [exclusive=true] [id=2]
[2025/03/28 01:18:30.577 +00:00] [INFO] [resolver/watch_based_grpc_resolver.go:57] ["update resolver state success"] [component=grpc-resolver] [scheme=milvus-session] [role=my-release/meta/session/rootcoord] [exclusive=true] [id=2] [state="{"Addresses":[{"Addr":"10.136.10.97:53100","ServerName":"","Attributes":{"<%!p(attributes.attributesKeyType=0)>": "<%!p(int64=1375)>" },"BalancerAttributes":{"<%!p(attributes.attributesKeyType=2)>": "<0xc002084480>" , "<%!p(attributes.attributesKeyType=0)>": "<%!p(int64=1375)>" },"Metadata":null}],"Endpoints":null,"ServiceConfig":null,"Attributes":null}"]
[2025/03/28 01:18:32.914 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [role=rootcoord] [intent=initInternal] [traceID=6a0529b33978f35c9f4d37968e63724c] [retried=4] [error="server error: UnknownError: Subscription does not exist"]
[2025/03/28 01:18:38.654 +00:00] [INFO] [healthz/healthz_handler.go:116] ["check health failed"] [UnhealthyComponent="[rootcoord]"]
[2025/03/28 01:18:48.654 +00:00] [INFO] [healthz/healthz_handler.go:116] ["check health failed"] [UnhealthyComponent="[rootcoord]"]
[2025/03/28 01:18:51.123 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [role=rootcoord] [intent=initInternal] [traceID=6a0529b33978f35c9f4d37968e63724c] [retried=8] [error="server error: UnknownError: Subscription does not exist"]
[2025/03/28 01:18:58.654 +00:00] [INFO] [healthz/healthz_handler.go:116] ["check health failed"] [UnhealthyComponent="[rootcoord]"]
[2025/03/28 01:18:59.652 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=83]
[2025/03/28 01:19:08.654 +00:00] [INFO] [healthz/healthz_handler.go:116] ["check health failed"] [UnhealthyComponent="[rootcoord]"]
[2025/03/28 01:19:11.133 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [role=rootcoord] [intent=initInternal] [traceID=6a0529b33978f35c9f4d37968e63724c] [retried=12] [error="server error: UnknownError: Subscription does not exist"]
[2025/03/28 01:19:18.654 +00:00] [INFO] [healthz/healthz_handler.go:116] ["check health failed"] [UnhealthyComponent="[rootcoord]"]
[2025/03/28 01:19:28.654 +00:00] [INFO] [healthz/healthz_handler.go:116] ["check health failed"] [UnhealthyComponent="[rootcoord]"]
[2025/03/28 01:19:29.653 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=83]
[2025/03/28 01:19:31.141 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [role=rootcoord] [intent=initInternal] [traceID=6a0529b33978f35c9f4d37968e63724c] [retried=16] [error="server error: UnknownError: Subscription does not exist"]

[xiaofan-luan]

你可以确定下你的pulsar能创建订阅么？
allowAutoSubscriptionCreation=true
如果订阅也不能创建，就比较麻烦了。。。

[yhmo]

订阅只跟你创建的集合和partiiton以及shards_num数量相关。
假设有一个集合，它有10个partition，这个集合创建时设置的shards_num=2(如果你不设置shards_num，默认为1)，则它产生的订阅数量为：
只要集合创建完成，datanode节点们必然会对这个集合产生订阅，订阅数量=10 * 2 = 20
如果你调用了collection.load()，querynode节点们就会对这个集合产生订阅，订阅数量=10 * 2 = 20
所以，你不调用load()，订阅数量为20；如果调用了load()，订阅数量为40
如果你有10张这样的集合，那么订阅数量就为200 ～ 400 之间，根据你load的表的数量浮动。
理想地，milvus会将这些订阅均匀地分布在256个topic上。如果订阅数量不足256，那么有些topic就处于闲置状态。

[bozai111]

你好，想再问下什么场景下Milvus会主动会去主动删除订阅？还是说一直不会删除？

[yhmo]

当你release_collection的时候，querynode为该表的订阅会被删除
当你drop_collection的时候，querynode和datanode为该表的订阅会被删除

[syyy-111]

你好，请教下关于milvus架构图中，proxy是先将DDL/DCL发送到root coord，root coord为每个DDL/DCL分配一个时间戳。root coord会将元数据的变更（如集合、索引等）持久化到etcd。是这么理解吗？然后etcd将元数据的变更通知到log broker中吗？control signal这个交互又怎么理解？

[yhmo]

你看的这张图是两年前的2.3版本，DDL/DCL所作的修改会被记录到etcd里，分配时间戳主要是为了保证修改操作的时序性。每张表的信息都必须记录在etcd里。打个比方，milvus是一家银行，etcd就是它的账本，minio则是它的金库，银行没账本就会乱套。不太清楚那个Notification是做什么用的。
在2.4/2.5里，没有这个Notification，https://milvus.io/docs/architecture_overview.md