Slow Retrieval Performance: 5 Minutes for 1000 Queries on FLAT Index (2.4M Docs, 113k Questions)

[Bhagyashreet20]

Hi Milvus team 👋,

I'm currently running a retrieval pipeline using Milvus and encountered some performance issues I'd like help with.

Setup Details:
Collection size: 2.4 million document chunks

Chunk size: 256 tokens

Embedding model: text-embedding-3-small (OpenAI)

Embedding dimension: 1536

Index type: FLAT

Query size: 113,000 questions total (tested with 1,000 for benchmarking)

Query batch size: 1,000

Deployment: standalone

Problem:
When I run a top-K retrieval (e.g., top-5 or top-10) using just 1000 questions, it takes approximately 4 minutes to fetch results using the FLAT index.

This seems unexpectedly slow, especially considering the relatively small batch size and document count (2.4M).

Questions:
Is this expected behavior with a FLAT index on this scale?

What is the typical or expected retrieval latency for 1000 queries in similar settings?

Would switching to an approximate index like IVF_FLAT, HNSW, or DISKANN help improve this latency?

Are there any tuning parameters I can set (e.g., nprobe, cache settings) to speed up FLAT search?

Any guidance or best practices to improve performance would be greatly appreciated! 🙏

I have attached the script for reference too

[Bhagyashreet20]

RAG.txt

[xiaofan-luan]

why we use FLAT index? HNSW should be 100 times faster than FLAT.
usually people pick Milvus because we offer effiecient nn search.
Unless you have strict recall requirement, it's highly recommend to use nn index.

Or if you have GPU card, maybe considering using GPU FLAT if you do need flat index