Sometimes Query latency was high. How to solve it?

[hjlee9182]

Do you know how to improve it in this situation? Regarding a separate collection, the query splatters when the writing comes in, so I checked it with otlp and the query segments splatter like below. How can I improve this part?

[yhmo]

Standalone or cluster? Which milvus version?
How many rows in the collection?
Which "consistency_level" is assigned?
How many rows are inserted per second?

[yhmo]

Assume the dimension is 768.
10 executors, each inserting 2,000 records at a time with a 0.15-second sleep in between.
2000 * 10 * 768 * 4bytes * (1/0.15) = 410MB is inserted per second

The data node will continually upload files to s3/minio.

If the "output_fields" parameter of query() contains the vector field name, it needs to read data from s3/minio since the IVF_SQ8 doesn't contain the original vectors.

Possible reasons:

1. Continually write operations on s3/minio affect read latency, while the query() interface needs to read data from s3/minio
2. If both two collections are loaded, the collection used for insertions will automatically/occasionally load new segments into querynode's memory, which could occupy the bandwidth(cpu resource, network resource) of the querynode, hence affecting the performance of query().

Options:
A. If the collection used for insertions is loaded, try releasing it and observe again
B. You can deploy a monitoring service to observe the cluster:
https://milvus.io/docs/monitor.md

Metrics of Proxty/DataCoordinator/QueryNode could reveal the root cause.

[hjlee9182]

If the "output_fields" parameter of query() contains the vector field name, it needs to read data from s3/minio since the IVF_SQ8 doesn't contain the original vectors.

Is IVF_FLAT contain original vectors? or Only FLAT contain original vectors?

[yhmo]

IVF_FLAT/FLAT/HNSW contains original vectors.

[hjlee9182]

Then, if I use IVF_FLAT, FLAT, or HNSW, will the issue not occur because the query won't fetch data from S3?

[hjlee9182]

And What is Delegator-waitTsafe?

[xiaofan-luan]

dis you specify vector as output field?

As yhmo said, the query seems to be slow.

try to modify this config and make sure the vector is loaded into cache and see if you still hit into the same issue
cache:
warmup: sync

[hjlee9182]

dis you specify vector as output field?

As yhmo said, the query seems to be slow.

We need to retrieve a vector with a specific ID and perform a search using it.
Is there a better way to do this?

try to modify this config and make sure the vector is loaded into cache and see if you still hit into the same issue
cache:
warmup: sync

what is the difference between async and sync??

[xiaofan-luan]

when load happened,
async skip loading raw vector onto disk, and create some background thread to load it.
So when the first search happened, there is no guarantee raw data is on local disk. And if the request falls into S3 it could much slower than local caching.

[hjlee9182]

Is querynode.cache used to store the original vector data from S3 onto the local disk of the querynode? Is it limited by querynode.memoryLimit?

Also, I understand that FLAT and IVF_FLAT store original vectors — is that the case for IVF_SQ8 only, or does IVF_FLAT also store the original vectors?

Does vector search happen entirely in memory on the querynode? If so, what role does the local disk (e.g., the querynode.cache) play in the search process?

Does the querynode have any type of cache for storing the results of query/get or search operations themselves, or is it limited to caching vector/index data?

[xiaofan-luan]

Is querynode.cache used to store the original vector data from S3 onto the local disk of the querynode? Is it limited by querynode.memoryLimit?

Also, I understand that FLAT and IVF_FLAT store original vectors — is that the case for IVF_SQ8 only, or does IVF_FLAT also store the original vectors?

Does vector search happen entirely in memory on the querynode? If so, what role does the local disk (e.g., the querynode.cache) play in the search process?

Does the querynode have any type of cache for storing the results of query/get or search operations themselves, or is it limited to caching vector/index data?

querynode.cache used to store the original vector data from S3 onto the local disk -> Yes
it takes only disk, not memory

Does vector search happen entirely in memory on the querynode -> maybe, maybe not, based on your index type. DiskANN use memory to search quantinized index and refine on disk.

es the querynode have any type of cache for storing the results of query/get or search operations -> no, for vector search, most of the queries are unique. That's why we only cache data

[hjlee9182]

After use this warmup, Delegator-waitTsafe is very slow. Do you know how to solve it?
I'm use eventually consistency when query,get, search

cache:
warmup: sync