After upgrading Milvus from version 2.3.5 to 2.5.11, the CPU utilization of the IndexNode component became excessively high.

[waitwindy]

When I upgraded Milvus from version 2.3.5 to 2.5.11, I noticed that the CPU utilization of the IndexNode component consistently exceeded 90% for over a full day. Additionally, the logs repeatedly printed ["Get Index Job Stats"]. What causes this? How can I prevent such high CPU usage?
indexnode-1.log

[yhmo]

The log looks good, no problem.
In the v2.5.x, data is sorted by primary key in each segment. But in old versions 2.3/2.4, data is not sorted. So, after upgrading, the v2.5.11 needs to rewrite all the segments, and all the indexes are rebuilt. It might take a long time if the data is huge.

[yhmo]

If you have no Go environment, you can use curl to export the profile, and send the profile_snapshot.heap to us.

curl http://[IP_of_querycoord]:9091/debug/pprof/profile\?seconds\=60 > profile_snapshot.heap

[waitwindy]

The result of this request execution is 404

[yhmo]

Maybe you should forward port 9091 of the querycoord.

[waitwindy]

The same command can be executed in version 2.3.5, but in version 2.5.11 it generates error 404. Can other commands be used

[yhmo]

You can use kubectl exec to enter the pod and run the curl command(maybe you need to install curl inside the container).

[xiaofan-luan]

i'm a little bit confused. is this a querycoord cpu issue or index cpu issue?

if this indexnode and data is huge, then this might be expected because after to upgrade all the data need to compact and reordered.

think about adding more index node and datanode to make it compact faster.

If querycoord cpu is high, then there must be something went wrong

[waitwindy]

The issue of the earliest index coord node being too high has been automatically resolved, and now there is a query coord node with high CPU usage.

[xiaofan-luan]

for querycoord can you offer a pprof?

the pprof can be find if you access

curl http://querycoordip:9091/debug/pprof/profile?seconds=30

Or this might help to generate a flame graph
curl http://querycoordip:9091/debug/pprof/profile?seconds=30 > cpu.svg

[waitwindy]

Today I saw that the querycoord node has returned to normal。

[xiaofan-luan]

is there anything special on the cluster? Like how many collections, parititons or segments do you have?