version 2.4.27 query takes too long to run

[yongbin813]

The previously deployed version was 2.4.13. Due to the inability to load data after restarting, we upgraded to version 2.4.17. After the upgrade, the query was very slowly. There were only a few vectors in database , but the query took 7-10 seconds. Don't understand why！！！

this is deployment file:

apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
name: milvus-cluster
labels:
app: milvus
spec:
mode: cluster
components:
#image: milvusdb/milvusv:2.4.13-hotfix
image: milvusdb/milvus:v2.4.17
mixCoord:
replicas: 1
resources:
limits:
cpu: 2
memory: 4Gi
indexNode:
replicas: 3
resources:
limits:
cpu: 2
memory: 4Gi
proxy:
replicas: 1
resources:
limits:
cpu: 2
memory: 4Gi
config:
queryNode:
enableDisk: true
dataCoord:
segment:
maxSize: 2048
diskSegmentMaxSize: 2048
gc:
interval: 7200
missingTolerance: 43200
log:
level: warn
file:
maxAge: 1
maxBackups: 1
dependencies:
etcd:
inCluster:
values:
autoCompactionMode: revision
autoCompactionRetention: "1000"
extraEnvVars:
- name: ETCD_QUOTA_BACKEND_BYTES
value: "4294967296"
- name: ETCD_HEARTBEAT_INTERVAL
value: "500"
- name: ETCD_ELECTION_TIMEOUT
value: "25000"
- name: ETCD_SNAPSHOT_COUNT
value: "10000"
- name: ETCD_ENABLE_PPROF
value: "true"
persistence:
accessMode: ReadWriteOnce
enabled: true
size: 30Gi #SSD Required
storageClass: nfs
replicaCount: 2
resources:
limits:
cpu: 2
memory: 4Gi
requests:
cpu: 1
memory: 4Gi

pulsar:
  inCluster:
    values:
      proxy:
        replicaCount: 2
        configData:
          PULSAR_MEM: >
            -Xms4096M
            -Xmx4096M
            -XX:MaxDirectMemorySize=4096M
          httpNumThreads: "100"
      zookeeper:
        volumes:
          persistence: true
          data:
            name: data
            size: 20Gi   #SSD Required
            storageClassName: nfs
        resources:
          requests:
            memory: 2Gi
            cpu: 1
        configData:
          PULSAR_MEM: >
            -Xms2048M
            -Xmx2048M
      bookkeeper:
        volumes:
          journal:
            name: journal
            size: 200Gi
            storageClassName: nfs
          ledgers:
            name: ledgers
            size: 500Gi   #SSD Required
            storageClassName: nfs
        resources:
          requests:
            memory: 4Gi
            cpu: 2
        configData:
          PULSAR_MEM: >
            -Xms4096M
            -Xmx4096M
            -XX:MaxDirectMemorySize=4096M
      broker:
        component: broker
        podMonitor:
          enabled: false
        replicaCount: 2
        resources:
          requests:
            memory: 4Gi
            cpu: 2
        configData:
          PULSAR_MEM: >
            -Xms4096M
            -Xmx4096M
            -XX:MaxDirectMemorySize=4096M

storage:
  inCluster:
    values:
      mode: distributed
      resources:
        limits: 
          cpu: 2
          memory: 4Gi
      persistence:
        storageClass: nfs 
        accessMode: ReadWriteOnce
        size: 500Gi

[yhmo]

Try search with consistency_level="Eventually".
If it is fast with "Eventually", the possible reason is the internal time-tick is delay.
Follow this doc to fetch the full log for further investigation:
https://github.com/milvus-io/milvus/tree/master/deployments/export-log

[bhxiaoxinba]

time-tick is delay为什么会影响查询延迟？

[yhmo]

time-tick机制用于数据同步。假设你持续地调用insert插入数据，多个query node从消息队列拉取数据的进度不一样，有快有慢。于是rootcoord在消息队列中每隔200毫秒就广播一个time-tick，如果T时间点的time-tick被所有query node都接收到了，就说明T时间点以前的数据都已在query node中，对查询可见。所以最慢的那个query node决定了查询可见的数据是多少。

查询时设置consistency_level为"Eventually"，这种level无视time-tick，直接在已有的数据中查询
查询时设置consistency_level为"Strong"，这种level要求当前时间点之前的数据可见，如果某些query node还没同步到当前时间点最近的time-tick，该查询就要等，一直等到所有query node同步到当前时间点最近的time-tick才开始执行查询
查询时设置consistency_level为"Bounded"，这种level有一个容忍窗口期(默认是5秒)，如果所有query node同步到5秒之前的time-tick，就可以执行查询

所以，如果有些query node的time-tick落后很多，比如某个query node刚同步到30秒之前的time-tick，那么，"Strong level"的查询要等待30秒，"Bounded level"的查询要等待25秒，"Eventually level"的查询无视delay直接执行。

[xiaofan-luan]

If you are using standalone, usually it slow because rocksdb consume slow.
please offer detailed logs so we can investigate on it

[yongbin813]

milvus-log.tar.gz

This is the exported log. I have not analyzed any obvious errors. Please ask the professionals to find out what the problem is. thks!!!

[xiaofan-luan]

[xiaofan-luan]

From your log, the pod seems to be very slow.
Init a threadpool takes almost 10 minutes.
You probably need to check the host machine cpu usage.
The reason why milvus can not serve is due to the init is not done.

[yongbin813]

I see ,and where is the initialization threadpool configured? Maybe I can try making the threadpool smaller.

[xiaofan-luan]

since you only have 1-2 core I think adding threadpool won't help too much.
And it also seems under your case the pod is slow, maybe on your K8s there are some offline task consumes too much cpu that affect the performance

[xiaofan-luan]

probably you need to check with your devops team see

1. if the pod throttled by K8s
2. is there any heavy work load happens on your host machine

[yongbin813]

thks, Maybe this needs to be solved by the devops team