milvus-2.5.1集群150w数据，三台机器，IVF_FLAT索引，30路压测的时候，queryNode节点CPU利用率波动幅度很大

[jhyever]

milvus-2.5.1集群150w数据，三台机器，IVF_FLAT索引，30路压测的时候，queryNode节点CPU利用率波动幅度很大，这是为啥 @yhmo 麻烦大佬帮忙解答一下 感谢

[jhyever]

queryNode日志

TotalMem: 17179869184
UsedMem: 33386496

open pid file: /run/milvus/querynode.pid
lock pid file: /run/milvus/querynode.pid
[2025/04/16 03:04:33.562 +00:00] [INFO] [roles/roles.go:345] ["starting running Milvus components"]
[2025/04/16 03:04:33.562 +00:00] [INFO] [roles/roles.go:187] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/:/milvus/lib/libjemalloc.so]
[2025/04/16 03:04:33.571 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]
[2025/04/16 03:04:33.572 +00:00] [DEBUG] [paramtable/base_table.go:213] ["init etcd source"] [etcdInfo="{"UseEmbed":false,"EnableAuth":false,"UserName":"","PassWord":"","UseSSL" :false,"Endpoints":["my-release-etcd:2379"],"KeyPrefix":"by-dev","CertFile":"/path/to/etcd-client.pem","KeyFile":"/path/to/etcd-client-key.pem","CaCertFile":"/path/to/ca.pe m","MinVersion":"1.3","RefreshInterval":5000000000}"]
[2025/04/16 03:04:33.573 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-release-etcd:2379]"] [minVersion=1.3]
[2025/04/16 03:04:33.576 +00:00] [DEBUG] [config/etcd_source.go:92] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[my-release-etcd:2379]"]
[2025/04/16 03:04:33.576 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=EtcdSource]
[2025/04/16 03:04:33.577 +00:00] [INFO] [paramtable/component_param.go:4471] ["DeployModeEnv is not set, use default"] [default=0.5]
[2025/04/16 03:04:33.579 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2025/04/16 03:04:33.579 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]
[2025/04/16 03:04:33.580 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]
[2025/04/16 03:04:33.580 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]
[2025/04/16 03:04:33.580 +00:00] [INFO] [roles/roles.go:294] [setupPrometheusHTTPServer]
[2025/04/16 03:04:33.580 +00:00] [INFO] [http/server.go:240] ["management listen"] [addr=:9091]
[2025/04/16 03:04:33.581 +00:00] [INFO] [gc/gc_tuner.go:138] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=15461882265]
[2025/04/16 03:04:33.582 +00:00] [INFO] [querynode/service.go:102] ["QueryNode listen on"] [address="[::]:21123"] [port=21123]
[2025/04/16 03:04:33.582 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-release-etcd:2379]"] [minVersion=1.3]
[2025/04/16 03:04:33.585 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/04/16 03:04:33.685 +00:00] [INFO] [querynodev2/server.go:294] ["register metrics actions finished"]
[2025/04/16 03:04:33.685 +00:00] [INFO] [querynodev2/server.go:304] ["QueryNode session info"] [metaPath=by-dev/meta]
[2025/04/16 03:04:33.685 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-release-etcd:2379]"] [minVersion=1.3]
[2025/04/16 03:04:33.697 +00:00] [INFO] [sessionutil/session_util.go:296] ["start server"] [name=querynode] [address=100.82.231.16:21123] [id=1981] [server_labels={}]
[2025/04/16 03:04:33.697 +00:00] [INFO] [sessionutil/session_util.go:1264] ["save server info into file"] [content="querynode-1981\n"] [filePath=/tmp/milvus/server_id_8]
[2025/04/16 03:04:33.697 +00:00] [INFO] [querynodev2/server.go:168] ["QueryNode init session"] [nodeID=1981] ["node address"=100.82.231.16:21123]
[2025/04/16 03:04:33.698 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=false] [mqType=pulsar]
[2025/04/16 03:04:33.704 +00:00] [INFO] [storage/remote_chunk_manager.go:97] ["remote chunk manager init success."] [remote=aws] [bucketname=milvus-bucket] [root=file]
[2025/04/16 03:04:33.704 +00:00] [INFO] [scheduler/concurrent_safe_scheduler.go:27] ["query node use concurrent safe scheduler"] [max_concurrency=10]
[2025/04/16 03:04:33.705 +00:00] [INFO] [querynodev2/server.go:345] ["queryNode init scheduler"] [policy=fifo]
[2025/04/16 03:04:33.705 +00:00] [INFO] [segments/segment_loader.go:168] ["SegmentLoader created"] [ioPoolSize=80]
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250416 03:04:33.705492 56 knowhere_config.cc:107] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384
I20250416 03:04:33.705684 56 knowhere_config.cc:118] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0
I20250416 03:04:33.705736 56 knowhere_config.cc:54] [KNOWHERE][ShowVersion][milvus] Knowhere Version: v2.5.0
I20250416 03:04:33.706166 56 thread_pool.h:198] [KNOWHERE][InitGlobalSearchThreadPool][milvus] Init global search thread pool with size 40
I20250416 03:04:33.706274 56 knowhere_config.cc:64] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO
I20250416 03:04:33.706298 56 knowhere_config.cc:89] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX512
I20250416 03:04:33.706327 56 Common.cpp:35] [SERVER][SetIndexSliceSize][milvus][]set config index slice size (byte): 16777216
I20250416 03:04:33.706413 56 Common.cpp:41] [SERVER][SetHighPriorityThreadCoreCoefficient][milvus][]set high priority thread pool core coefficient: 10
I20250416 03:04:33.706436 56 Common.cpp:48] [SERVER][SetMiddlePriorityThreadCoreCoefficient][milvus][]set middle priority thread pool core coefficient: 5
I20250416 03:04:33.706454 56 Common.cpp:55] [SERVER][SetLowPriorityThreadCoreCoefficient][milvus][]set low priority thread pool core coefficient: 1
[2025/04/16 03:04:33.706 +00:00] [INFO] [querynodev2/server.go:235] ["set up knowhere build pool size"] [pool_size=5]
I20250416 03:04:33.706553 56 thread_pool.h:178] [KNOWHERE][InitGlobalBuildThreadPool][milvus] Init global build thread pool with size 5
I20250416 03:04:33.706599 56 Common.cpp:62] [SERVER][SetDefaultExecEvalExprBatchSize][milvus][]set default expr eval batch size: 8192
I20250416 03:04:33.706735 56 MinioChunkManager.cpp:161] [SERVER][InitSDKAPIDefault][milvus][]init aws with log level:fatal
I20250416 03:04:35.738305 56 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus][]start to precheck chunk manager with configuration: [address=my-release-minio:9000, bucket_name=milvus-b ucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000 , gcp_native_without_auth=false]
I20250416 03:04:35.747589 56 ChunkManager.cpp:112] [SERVER][AwsChunkManager][milvus][]init AwsChunkManager with parameter[endpoint=my-release-minio:9000][bucket_name=milvus-bucket][root_pa th=file][use_secure=false]
I20250416 03:04:35.748157 56 MmapChunkManager.cpp:320] [SERVER][MmapChunkManager][milvus][]Init MappChunkManager with: Path /var/lib/milvus/data/mmap/mmap_chunk_manager, MaxDiskSize 50682 MB, FixedFileSize 1 MB.
I20250416 03:04:35.748193 56 ChunkCache.h:42] [SERVER][ChunkCache][milvus][]Init ChunkCache with read_ahead_policy: willneed
I20250416 03:04:35.748203 56 MmapManager.h:74] [SERVER][Init][milvus][]Init MmapConfig with MmapConfig: [cache_read_ahead_policy=willneed, mmap_path=/var/lib/milvus/data/mmap/mmap_chunk_ma nager, disk_limit=50682MB, fix_file_size=1MB, growing_enable_mmap=false, scalar_index_enable_mmap=false]
I20250416 03:04:35.748277 56 Tracer.cpp:81] [SERVER][initTelemetry][milvus][]Empty Trace
I20250416 03:04:35.748442 56 FunctionFactory.cpp:59] [SERVER][RegisterAllFunctions][milvus][]2 functions registered
[2025/04/16 03:04:35.748 +00:00] [INFO] [querynodev2/server.go:385] ["query node init successfully"] [queryNodeID=1981] [Address=100.82.231.16:21123]
[2025/04/16 03:04:35.748 +00:00] [INFO] [querynodev2/server.go:413] ["query node start successfully"] [queryNodeID=1981] [Address=100.82.231.16:21123] [mmapEnabled=false] [growingmmapEnable=f alse] [mmapVectorIndex=false] [mmapVectorField=false] [mmapScalarIndex=false] [mmapScalarField=false] [mmapChunkCache=true]
[2025/04/16 03:04:35.748 +00:00] [INFO] [scheduler/concurrent_safe_scheduler.go:213] ["start execute loop"]
[2025/04/16 03:04:35.761 +00:00] [INFO] [sessionutil/session_util.go:494] ["put session key into etcd"] [key=by-dev/meta/session/querynode-1981] [value="{"ServerID":1981,"ServerName":"qu erynode","Address":"100.82.231.16:21123","TriggerKill":true,"Version":"2.5.1","IndexEngineVersion":{"CurrentIndexVersion":6},"LeaseID":1276654206192162661,"HostName":"my-r elease-milvus-querynode-7df5777fb7-97z6x"}"]
[2025/04/16 03:04:35.761 +00:00] [INFO] [sessionutil/session_util.go:504] ["Service registered successfully"] [ServerName=querynode] [serverID=1981]
[2025/04/16 03:04:35.764 +00:00] [INFO] [components/query_node.go:63] ["QueryNode successfully started"]
[2025/04/16 03:04:35.765 +00:00] [INFO] [tracer/tracer.go:50] ["Init tracer finished"] [Exporter=noop]
[2025/04/16 03:04:35.801 +00:00] [INFO] [sessionutil/session_util.go:944] ["register session success"] [role=querynode] [key=by-dev/meta/session/querynode-1981]
[2025/04/16 03:05:03.562 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=66]
[2025/04/16 03:05:33.563 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=66]
[2025/04/16 03:06:03.562 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=66]
[2025/04/16 03:06:15.934 +00:00] [INFO] [querynodev2/services.go:205] ["received watch channel request"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [channel= by-dev-rootcoord-dml_1_456275818148801950v1] [currentNodeID=1981] [version=1744772775931851176]
[2025/04/16 03:06:15.934 +00:00] [INFO] [segments/collection.go:98] ["put new collection"] [collectionID=456275818148801950] [schema="name:"qqxlmx1" description:"Test content search" fi elds:{name:"RowID" description:"row id" data_type:Int64} fields:{fieldID:1 name:"Timestamp" description:"time stamp" data_type:Int64} fields:{fieldID:100 name:"id" is_prima ry_key:true data_type:Int64} fields:{fieldID:101 name:"lib_id" data_type:VarChar type_params:{key:"max_length" value:"60"}} fields:{fieldID:102 name:"part_id" data_type:VarCh ar type_params:{key:"max_length" value:"60"}} fields:{fieldID:103 name:"doc_id" data_type:VarChar type_params:{key:"max_length" value:"60"}} fields:{fieldID:104 name:"cate gory" data_type:VarChar type_params:{key:"max_length" value:"100"}} fields:{fieldID:105 name:"user" data_type:VarChar type_params:{key:"max_length" value:"60"}} fields:{fi eldID:106 name:"source_content" data_type:VarChar type_params:{key:"max_length" value:"4096"}} fields:{fieldID:107 name:"versionList" data_type:Array type_params:{key:"max_ca pacity" value:"100"} type_params:{key:"max_length" value:"100"} element_type:Int16} fields:{fieldID:108 name:"labels" data_type:Array type_params:{key:"max_capacity" value :"100"} type_params:{key:"max_length" value:"100"} element_type:VarChar} fields:{fieldID:109 name:"embedding" data_type:FloatVector type_params:{key:"dim" value:"1024"}} fields:{fieldID:110 name:"$meta" description:"dynamic schema" data_type:JSON is_dynamic:true} enable_dynamic_field:true"]
[2025/04/16 03:06:15.934 +00:00] [INFO] [vecindexmgr/vector_index_mgr.go:124] ["init vector indexes with features : BINFLAT : 720897,BIN_FLAT : 720897,BIN_IVF_FLAT : 524289,DISKANN : 2097166, FLAT : 720910,HNSW : 1572910,HNSWLIB_DEPRECATED : 1572879,HNSW_PQ : 524334,HNSW_PRQ : 524334,HNSW_SQ : 524334,IVFBIN : 524289,IVFFLAT : 524302,IVFFLATCC : 14,IVFPQ : 524302,IVFSQ : 524302,IVF _FLAT : 524302,IVF_FLAT_CC : 14,IVF_PQ : 524302,IVF_SQ : 524302,IVF_SQ8 : 524302,IVF_SQ_CC : 14,SCANN : 524302,SPARSE_INVERTED_INDEX : 524304,SPARSE_INVERTED_INDEX_CC : 524304,SPARSE_WAND : 5 24304,SPARSE_WAND_CC : 524304,"]
I20250416 03:06:15.935715 56 Collection.cpp:64] [SERVER][parseIndexMeta][milvus][]index meta info: maxRowCount : {20609} FieldId : {109 IndexParams : { index_type : IVF_FLAT, metric_type : IP, nlist : 1024, }TypeParams : {dim : 1024, }}
[2025/04/16 03:06:15.935 +00:00] [INFO] [segments/collection.go:229] ["collection ref increment"] [nodeID=1981] [collectionID=456275818148801950] [refCount=1]
[2025/04/16 03:06:15.935 +00:00] [INFO] [delegator/delegator.go:885] ["Init delete cache with list delete buffer"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [replicaID=457252108674793478] [channel=by-dev-rootcoord-dml_1_456275818148801950v1] [version=1744772775931851176] [startTs=457381644842041348] [sizePerBlock=1048576] [startTime=2025/04/16 0 3:01:49.926 +00:00]
[2025/04/16 03:06:15.936 +00:00] [INFO] [delegator/delegator.go:890] ["shard delegator setup l0 forward policy"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [ replicaID=457252108674793478] [channel=by-dev-rootcoord-dml_1_456275818148801950v1] [version=1744772775931851176] [startTs=457381644842041348] [policy=FilterByBF]
[2025/04/16 03:06:15.936 +00:00] [INFO] [delegator/delegator.go:937] ["finish build new shardDelegator"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [replicaI D=457252108674793478] [channel=by-dev-rootcoord-dml_1_456275818148801950v1] [version=1744772775931851176] [startTs=457381644842041348]
[2025/04/16 03:06:15.936 +00:00] [INFO] [pipeline/manager.go:68] ["start create pipeine"] [collectionID=456275818148801950] [channel=by-dev-rootcoord-dml_1_456275818148801950v1]
[2025/04/16 03:06:15.936 +00:00] [INFO] [delegator/delegator_data.go:354] ["loading growing segments..."] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [channel =by-dev-rootcoord-dml_1_456275818148801950v1] [replicaID=457252108674793478] [segmentIDs="[]"]
[2025/04/16 03:06:15.936 +00:00] [INFO] [segments/segment_loader.go:231] ["no segment to load"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [segmentType=Growi ng]
[2025/04/16 03:06:15.936 +00:00] [INFO] [delegator/delegator_data.go:377] ["load growing segments done"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [channel= by-dev-rootcoord-dml_1_456275818148801950v1] [replicaID=457252108674793478] [segmentIDs="[]"]
[2025/04/16 03:06:15.936 +00:00] [INFO] [delegator/delegator_data.go:343] ["add growing segments to delegator"] [collectionID=456275818148801950] [channel=by-dev-rootcoord-dml_1_4562758181488 01950v1] [replicaID=457252108674793478] [segmentIDs="[]"]
[2025/04/16 03:06:15.936 +00:00] [INFO] [msgdispatcher/manager.go:66] ["create new dispatcherManager"] [role=querynode] [nodeID=1981] [pchannel=by-dev-rootcoord-dml_1]
[2025/04/16 03:06:15.936 +00:00] [INFO] [msgdispatcher/dispatcher.go:97] ["creating dispatcher..."] [pchannel=by-dev-rootcoord-dml_1] [subName=querynode-1981-by-dev-rootcoord-dml_1_4562758181 48801950v1-true] [isMain=true]
[2025/04/16 03:06:15.936 +00:00] [INFO] [msgdispatcher/manager.go:167] ["dispatcherManager is running..."] [role=querynode] [nodeID=1981] [pchannel=by-dev-rootcoord-dml_1]
[2025/04/16 03:06:15.937 +00:00] [INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true]
[2025/04/16 03:06:16.021 +00:00] [INFO] [msgstream/mq_msgstream.go:922] ["MsgStream begin to seek start msg: "] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [channel=by-dev-rootcoord-dml_1] [Me ssageID="CM4hEIWcAhgAIAAwAQ=="]
[2025/04/16 03:06:16.154 +00:00] [INFO] [msgstream/mq_msgstream.go:932] ["MsgStream seek finished"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [channel=by-dev-rootcoord-dml_1]
[2025/04/16 03:06:16.169 +00:00] [INFO] [msgdispatcher/dispatcher.go:126] ["seek successfully"] [pchannel=by-dev-rootcoord-dml_1] [subName=querynode-1981-by-dev-rootcoord-dml_1_45627581814880 1950v1-true] [isMain=true] [posTs=457381644842041348] [posTime=2025/04/16 03:01:49.926 +00:00] [tsLag=4m26.243846458s]
[2025/04/16 03:06:16.169 +00:00] [INFO] [msgdispatcher/dispatcher.go:162] ["add new target"] [vchannel=by-dev-rootcoord-dml_1_456275818148801950v1] [isMain=true]
[2025/04/16 03:06:16.169 +00:00] [INFO] [msgdispatcher/manager.go:114] ["add main dispatcher"] [role=querynode] [nodeID=1981] [vchannel=by-dev-rootcoord-dml_1_456275818148801950v1]
[2025/04/16 03:06:16.170 +00:00] [INFO] [msgdispatcher/dispatcher.go:190] ["get signal"] [pchannel=by-dev-rootcoord-dml_1] [signal=start] [isMain=true]
[2025/04/16 03:06:16.170 +00:00] [INFO] [msgdispatcher/dispatcher.go:213] ["handle signal done"] [pchannel=by-dev-rootcoord-dml_1] [signal=start] [isMain=true]
[2025/04/16 03:06:16.170 +00:00] [INFO] [msgdispatcher/client.go:103] ["register done"] [role=querynode] [nodeID=1981] [vchannel=by-dev-rootcoord-dml_1_456275818148801950v1]
[2025/04/16 03:06:16.170 +00:00] [INFO] [pipeline/stream_pipeline.go:133] ["stream pipeline seeks from position with msgDispatcher"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [pchannel=by-de v-rootcoord-dml_1_456275818148801950v1] [vchannel=by-dev-rootcoord-dml_1_456275818148801950v1] [checkpointTs=2025/04/16 03:01:49.926 +00:00] [tsLag=4m26.244046407s] [elapse=233.583722ms]
[2025/04/16 03:06:16.170 +00:00] [INFO] [querynodev2/services.go:341] ["watch dml channel success"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [channel=by-de v-rootcoord-dml_1_456275818148801950v1] [currentNodeID=1981]
[2025/04/16 03:06:16.170 +00:00] [INFO] [msgdispatcher/dispatcher.go:218] ["begin to work"] [pchannel=by-dev-rootcoord-dml_1] [isMain=true]
[2025/04/16 03:06:16.431 +00:00] [INFO] [querynodev2/services.go:417] ["received load segments request"] [traceID=39ceeafe2ec6438a01f8e842ffdba5ec] [collectionID=456275818148801950] [partitio nID=456275818148801951] [shard=by-dev-rootcoord-dml_1_456275818148801950v1] [segmentID=457204142627882600] [level=L1] [currentNodeID=1981] [dstNodeID=1978] [version=1744772776429185431] [need Transfer=true] [loadScope=Full]
[2025/04/16 03:06:16.431 +00:00] [INFO] [querynodev2/services.go:417] ["received load segments request"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [shard=by-dev-rootcoord-dml_1_456275818148801950v1] [segmentID=457204142627881297] [level=L1] [currentNodeID=1981] [dstNodeID=1981] [version=1744772776430590177] [need Transfer=true] [loadScope=Full]
[2025/04/16 03:06:16.431 +00:00] [INFO] [querynodev2/services.go:417] ["received load segments request"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [shard=by-dev-rootcoord-dml_1_456275818148801950v1] [segmentID=457204142627881297] [level=L1] [currentNodeID=1981] [dstNodeID=1981] [version=1744772776430590177] [need Transfer=false] [loadScope=Full]
[2025/04/16 03:06:16.431 +00:00] [INFO] [segments/collection.go:229] ["collection ref increment"] [nodeID=1981] [collectionID=456275818148801950] [refCount=2]
[2025/04/16 03:06:16.431 +00:00] [INFO] [querynodev2/services.go:489] ["start to load segments..."] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitionID=4 56275818148801951] [shard=by-dev-rootcoord-dml_1_456275818148801950v1] [segmentID=457204142627881297] [level=L1] [currentNodeID=1981] [dstNodeID=1981]
[2025/04/16 03:06:16.431 +00:00] [INFO] [segments/segment_loader.go:252] ["start loading..."] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [segmentType=Sealed] [requestSegments="[457204142627881297]"] [preparedSegments="[457204142627881297]"] [segmentNum=1] [afterFilter=1]
[2025/04/16 03:06:16.436 +00:00] [INFO] [segments/pool.go:108] ["init dynamicPool done"] [size=10]
[2025/04/16 03:06:16.438 +00:00] [INFO] [segments/segment_loader.go:1414] ["predict memory and disk usage while loading (in MiB)"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456 275818148801950] [maxSegmentSize(MB)=0.2816162109375] [committedMemSize(MB)=0] [memLimit(MB)=16384] [memUsage(MB)=56.57421875] [committedDiskSize(MB)=0] [diskUsage(MB)=0] [predictMemUsage(MB) =56.8558349609375] [predictDiskUsage(MB)=0] [mmapFieldCount=0]
[2025/04/16 03:06:16.438 +00:00] [INFO] [segments/segment_loader.go:482] ["request resource for loading segments (unit in MiB)"] [traceID=1c46417f55a81e976ca6025d2ec40028] [segmentIDs="[45720 4142627881297]"] [memory=0.2816162109375] [committedMemory=0.2816162109375] [disk=0] [committedDisk=0]
[2025/04/16 03:06:16.441 +00:00] [INFO] [segments/segment.go:337] ["create segment done"] [traceID=1c46417f55a81e976ca6025d2ec40028]
[2025/04/16 03:06:16.441 +00:00] [INFO] [segments/segment_loader.go:355] ["start to load segments in parallel"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [s egmentType=Sealed] [requestSegments="[457204142627881297]"] [preparedSegments="[457204142627881297]"] [segmentNum=1] [concurrencyLevel=1]
[2025/04/16 03:06:16.442 +00:00] [INFO] [segments/segment_loader.go:328] ["load segment..."] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [segmentType=Sealed] [requestSegments="[457204142627881297]"] [preparedSegments="[457204142627881297]"] [partitionID=456275818148801951] [segmentID=457204142627881297] [segmentType=L1]
[2025/04/16 03:06:16.442 +00:00] [INFO] [segments/segment_loader.go:842] ["start loading segment files"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [shard=by-dev-rootcoord-dml_1_456275818148801950v1] [segmentID=457204142627881297] [rowNum=15] [segmentType=Sealed]
[2025/04/16 03:06:16.442 +00:00] [INFO] [segments/pool.go:129] ["init loadPool done"] [size=50]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment.go:846] ["add field data info done"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitionID=456275 818148801951] [segmentID=457204142627881297] ["row count"=15]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment_loader.go:761] ["Start loading fields..."] [traceID=1c46417f55a81e976ca6025d2ec40028] [segmentID=457204142627881297] [indexedFields=" []"] ["indexed text fields"="[]"] ["unindexed text fields"="[]"]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment.go:773] ["start loading field data for field"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [segmentID=457204142627881297] [fieldID=108] [rowCount=15]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment.go:773] ["start loading field data for field"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [segmentID=457204142627881297] [fieldID=0] [rowCount=15]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment.go:773] ["start loading field data for field"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [segmentID=457204142627881297] [fieldID=102] [rowCount=15]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment.go:773] ["start loading field data for field"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [segmentID=457204142627881297] [fieldID=103] [rowCount=15]

[xiaofan-luan]

IVF_FLAT

cpu已经打满了，建议用hnsw索引

[jhyever]

CPU没打满啊，整机才用了30%左右 ，queryNode节点pod用了最多才8核，，，是queryNode有什么限流麽，为啥会出现波动呀

[xiaofan-luan]

波动时因为你的压力不够大吧。

很大概率你的查询都是batch，每次milvus会对batch进行一些并发。然后中间空的地方就是没有新的batch进来。
如果是看吞吐，那么你可以尝试并发更多的请求
如果是看延迟，那么要做的就是减小batch

[xiaofan-luan]

然后你看一下你这几台的机器的压力是不是平均，还是有一台特别高

[yhmo]

日志显示该表的向量维度1024， 有2个sealed segment，其中一个segment只有15行数据，另一个的行数没看到，不合理。
用webui观察segment:
http://[proxy_node_address]:9091/webui/data_component

客户端的压测方式可能有问题，nprobe的值不会是1吧，查询向量是随机的还是固定的

[jhyever]

1、这种java调用的

2、我是30路压测，每次请求是30个并发【备注，不是每秒，是每次】，不停的调用search的

[yhmo]

String expr = ExprUtils.buildExpr ......

这个expr比较可疑，看起来是个复杂的过滤表达式。对IVF索引来说，过滤表达式直接影响每个请求的latency，过滤量大的latency小，过滤量小的latency大。如果只有极少的数据符合条件，那么querynode的工作量就基本是在expr的解析，以及扫描符合条件的数据上。“扫描符合条件的数据” 这个事情并不像 “计算几万条向量的distance” 那么的容易并行，所以querynode的cpu使用率不会高。

为了验证，你可以把builder.withExpr(expr)这行注释掉，然后再测。

[jhyever]

去掉也还是一样呢

[yhmo]

你安装一个标准的监控看看：https://milvus.io/docs/monitor.md
然后在监控页面的QueryNode和Proxy里把跟search相关的指标都截图看一下。我比较怀疑压力没打满。

[jhyever]

已解决，是压测脚本的问题，感谢解答