After deleteing Collection, the corresponding Index Build Tasks still exist and in running.

[zhongshanhao]

I create a index with HNSW_PQ(m=32, nbits=24) and insert Thirty million vector rows.

I found the Milvus OOM exit and the corresponding Index Build Tasks in InProgress state.

I guess the Kmeans consume a lot of memory in index build progress, then I drop the index, but the corresponding Index Build Tasks still exist and in running. The Milvus still exit due to OOM occur after restart Milvus.

1. Is this a bug? After deleteing Collection, the corresponding Index Build Tasks shuold stop automatically?
2. How should I stop the task?

Env:

• Milvus version: 2.5.5
• Deployment mode(standalone or cluster): standalone
• OS(Ubuntu or CentOS): Debian 6.3.0
• CPU/Memory: 64C/256G

[zhongshanhao]

@yhmo help me big bro!😊

[yhmo]

In index node, each index task is for one segment. When you call the drop_index() interface, the index node will cancel all the index tasks that are queued and not executed. The ongoing task cannot be canceled, index node will wait until it is finished.
So, after you drop the index and restart the milvus, the index node should be idle.

drop_index() must be called after release_collection(). So, after you drop the index and restart the milvus, the collection should not be loaded, the milvus should only consume a few memory.

[yhmo]

The log shows both the "exp_7_hnsw" and "exp_7_flat" are loaded.
After all the segments of the two collections are loaded, the memory usage is 223132MB(=223GB). There is only 30GB of memory space left.

{"log":"[2025/05/16 09:47:33.773 +00:00] [INFO] [segments/segment_loader.go:1440] [\"predict memory and disk usage while loading (in MiB)\"] [traceID=761e4b0d0d0cb368e1ac7d15c34b7266] [collectionID=456484240695966453] [maxSegmentSize(MB)=2561.1594944000244] [committedMemSize(MB)=222852.84903526306] [memLimit(MB)=255961.30859375] [memUsage(MB)=223132.90762901306] [committedDiskSize(MB)=44343.72051143646] [diskUsage(MB)=44343.72051143646] [predictMemUsage(MB)=225694.0671234131] [predictDiskUsage(MB)=44343.72051143646] [mmapFieldCount=0]\n","stream":"stdout","time":"2025-05-16T09:47:33.773537108Z"}

And then "exp_7_hnsw_pq_m32_nbits10" is also loaded, 30GB memory space is insufficient to load it, easily lead to OOM.

[zhongshanhao]

[2025/05/19 04:08:25.047 +00:00] [INFO] [segments/segment_loader.go:1440] ["predict memory and disk usage while loading (in MiB)"] [traceID=2b91fdb11b1507afd61087270ee6f5f5] [collectionID=456484240695966453] [maxSegmentSize(MB)=2583.5020933151245] [committedMemSize(MB)=202378.80245780945] [memLimit(MB)=255961.30859375] [memUsage(MB)=202590.00167655945] [committedDiskSize(MB)=0] [diskUsage(MB)=0] [predictMemUsage(MB)=205173.50376987457] [predictDiskUsage(MB)=0] [mmapFieldCount=0]

The newest log show memUsage(MB)=202590.00167655945, but the comand free -hm in pysical machine show 141GB of memory space left:

               total        used        free      shared  buff/cache   available
Mem:           249Gi       100Gi       147Gi       2.0Mi       2.1Gi       141Gi
Swap:             0B          0B          0B

Is segment_loader report a error's memUsage?

[yhmo]

In the source code, milvus is using the "gopsutil" lib to get memory info:

milvus/pkg/util/hardware/mem_info.go

Line 45 in 3d8629d

memInfo, err := proc.MemoryInfoEx()

The gopsutil repo already received 11k stars on github, which is a popular utility library in golang.
Milvus is using version v4.24.10 of gopsutil, which was released in November 2024. v4.24.10 is quite a new version.
https://github.com/shirou/gopsutil

You can use the command "docker stats" to show memory usage for all the containers. I believe the "MEM USAGE" of milvus standalone is almost equal to the "memUsage(MB)".

Note: the "memUsage(MB)" is a runtime value, each time a segment is ready to be loaded, milvus will call gopsutil interface to get the runtime memory usage of milvus process.

[zhongshanhao]

Only "exp_7_hnsw" and "exp_7_flat" are loaded.

[2025/05/19 08:00:05.730 +00:00] [INFO] [segments/segment_loader.go:1440] ["predict memory and disk usage while loading (in MiB)"] [traceID=14f5844de4a0f8114911105cb20d6291] [collectionID=456484240697312418] [maxSegmentSize(MB)=2367.168429374695] [committedMemSize(MB)=120945.19888973236] [memLimit(MB)=255961.30859375] [memUsage(MB)=214429.78092098236] [committedDiskSize(MB)=0] [diskUsage(MB)=0] [predictMemUsage(MB)=216796.94935035706] [predictDiskUsage(MB)=0] [mmapFieldCount=0]

The command "docker stats" show that:

CONTAINER ID   NAME                CPU %      MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
30da275159c8   milvus-standalone   2065.13%   99.08GiB / 250GiB   39.64%    144GB / 3.6GB    827kB / 431kB     242
49065d251691   milvus-minio        0.02%      437.7MiB / 250GiB   0.17%     429GB / 2.75TB   2.76TB / 426GB    74
68117a9796ee   milvus-etcd         0.44%      55.41MiB / 250GiB   0.02%     246MB / 208MB    2.32GB / 4.49GB   55

The mem usage show different in above two way.

Though I release Collection "exp_7_hnsw" and "exp_7_flat", "exp_7_hnsw_pq_m32_nbits10" unable to load and Milvus exit with log as above.

[yhmo]

Follow these steps to export a clean log:

1. Stop the milvus standalone by "docker compose stop"
2. Install the birdwatcher tool: https://milvus.io/docs/birdwatcher_install_guides.md, you can download the already-built binary by "wget -O birdwatcher.tar.gz
   https://github.com/milvus-io/birdwatcher/releases/download/v1.0.8/birdwatcher_Linux_x86_64.tar.gz"
3. Use the command "docker ps -a" to show all the containers, find out the ID of the Etcd container. And use the command "docker start [etcd container id]" to start the etcd container
4. Extract the birdwatcher_Linux_x86_64.tar.gz and run the already-built binary
5. Now you are in the command line mode of birdwatcher, type the following commands to force release all the collections:

connect --etcd 127.0.0.1:2379
force-release --all
exit

6. Use the command "docker compose restart" to restart the milvus standalone
7. Use the command "docker stats" to observe the MEM USAGE of milvus container, the usage should be less than 500MB
8. Load the collection "exp_7_hnsw_pq_m32_nbits10" by Python SDK

client.load_collection(collection_name="exp_7_hnsw_pq_m32_nbits10")

9. Observe the MEM USAGE of milvus container; the usage increases. Ideally, the usage should be less than 50GB.
10. If the milvus standalone is still OOM, use the command "docker logs [milvus_container_id] > oom.log" to export the log

[xiaofan-luan]

did you tune the segment size?

256GB memory should be anyway good enough for indexbuildings?

[zhongshanhao]

I did not.

[xiaofan-luan]

@zhongshanhao

I think 256G memory should be far more enough for 1536dim HNSWPQ.

You might hit into a bug in 2.5.5 which might cause unstoppable compaction and may consume a lot of resources.

Try to upgrade to 2.5.12 and see if it helps.

I'm assuming 64GB memory should be good