The node was low on resource: ephemeral-storage

[jubingc]

Few query pods and a mixcoord pod reported "The node was low on resource: ephemeral-storage" even though an S3 bucket is used as external storage.

What could consume the disk space, logs, cache? Should I put a limit on resources.ephemeral-storage of the deployment? If yes, how much is recommended?

[yhmo]

Only datanode and index can persist data into S3.

Query node only reads data from S3, and never writes data to S3. Query node could download data to local disk for "cache" or "mmap", according to the queryNode.cache and queryNode.mmap configurations, by default the mmap disk usage should be less than 50%:

milvus/configs/milvus.yaml

Line 420 in 36a4fad

cache:

milvus/configs/milvus.yaml

Line 430 in 36a4fad

mmap:

Logs could be output to local disk if the log.file.rootPath is set to a valid path, the max size is limited by the configurations, by default no more than 3GB:

milvus/configs/milvus.yaml

Line 758 in 36a4fad

log:

You can check the disk usage of path "/var/lib/milvus/data/" inside the querynode.
How much disk space is available for the query node? I think you'd better assign at least 20GB ~ 30GB disk space to each query node.

The sizing tool can help you estimate resource requirement for each node: https://milvus.io/tools/sizing
For example, if the mmap is enabled, click the "With Scalar Fields" button, you can see the disk usage is estimated:

[bigsheeper]

@jubingc Could you please provide the following details to help us better understand the issue?

1. Milvus version you are using
2. Collection schema
3. Index type and parameters
4. Output fields specified in the query/search

[yhmo]

@jubingc
By default, the queryNode.mmap.chunkCache is true, did you change it to false?

queryNode:
  cache:
    memoryLimit: 2147483648 # 2 GB, 2 * 1024 *1024 *1024
    readAheadPolicy: willneed # The read ahead policy of chunk cache, options: `normal, random, sequential, willneed, dontneed`
    # options: async, sync, disable. 
    # Specifies the necessity for warming up the chunk cache. 
    # 1. If set to "sync" or "async" the original vector data will be synchronously/asynchronously loaded into the 
    # chunk cache during the load process. This approach has the potential to substantially reduce query/search latency
    # for a specific duration post-load, albeit accompanied by a concurrent increase in disk usage;
    # 2. If set to "disable" original vector data will only be loaded into the chunk cache during search/query.
    warmup: disable
  mmap:
    vectorField: false # Enable mmap for loading vector data
    vectorIndex: false # Enable mmap for loading vector index
    scalarField: false # Enable mmap for loading scalar data
    scalarIndex: false # Enable mmap for loading scalar index
    chunkCache: true # Enable mmap for chunk cache (raw vector retrieving).
    # Enable memory mapping (mmap) to optimize the handling of growing raw data. 
    # By activating this feature, the memory overhead associated with newly added or modified data will be significantly minimized. 
    # However, this optimization may come at the cost of a slight decrease in query latency for the affected data segments.
    growingMmapEnabled: false
    fixedFileSizeForMmapAlloc: 1 # tmp file size for mmap chunk manager
    maxDiskUsagePercentageForMmapAlloc: 50 # disk percentage used in mmap chunk manager

[jubingc]

Hi @bigsheeper

1. We are using milvusdb/milvus:v2.5.4 with helm chart milvus-4.2.38
2. 3. 4. The collections are defined by our internal users. Users have various collections with different schemas. If I understand what you are looking for and why, I might be able to look into those collections.

We have two clusters. I can only tell that one has 90M entities (65 GB), and another has 148M milvus_querynode_entity_num (225 GB milvus_querynode_entity_size) when this issue happened yesterday 2/24/2025. Each query node pod has 64 GB of memory. We have around 14 and 22 query pods for each cluster respectively, the numbers fluctuate due to collection load/release and horizontal pod autoscaler.

When the issue happened, I was configuring pod resources. Existing pods were replaced by new ones. Maybe it is another impact factor. It might be a transient error, the pods were killed and the cluster became stable after a while. But I still would like to know the reason and how to prevent this issue from happening again.

Hi @yhmo

When I said mmap was not enabled, I referred to I didn't enable it in helm values. And I realized that you may refer to the Milvus Configuration. And yes, mmap is always enabled. The default configurations are never changed.

According to our usage 65 GB + 225 GB, assuming the raw data size is 50% then the average entity size is 625 KB (150 GB / 240 M). The recommended memory 2.3 TB by the sizing tool is exactly what we deployed. The recommended local disk size is 291 GB. Is this 291 GB for individual pods or for all pods combined? If this is for individual pods, each pod requires 8 ~ 10 GB of disk space. So why did the query pod request 44 GB of disk space?

[yhmo]

queryNode.mmap.chunkCache is true, which means the mmap is enabled.
With mmap is enabled, if you have search/query with output_fields, the data files of the output_fields will be cached to local disk. With more and more search/query executed, more data files are downloaded to local disk, and the disk usage of each query node will increase.

291GB local disk is for each query node. Since the total raw data size is 2.15TB, each query node might download 2.15TB / 18 = 122GB data, we have a safety ratio = 2.x so that the sizing tool shows 291GB for each query node. The Total Local Disk should be displayed as 291GB * 18 = 5.23TB. It is a bug in the sizing tool that shows 291GB of the total local disk, we will fix it later.

[jubingc]

Hi @yhmo

Thanks for your helpful reply. Are there any metrics available to monitor the cache behavior on query nodes? I am looking at a few metrics

https://github.com/milvus-io/milvus/blob/master/internal/querynodev2/segments/metricsutil/observer.go#L182-L187

	DiskCacheLoadTotal                   prometheus.Counter
	DiskCacheLoadDuration                prometheus.Counter
	DiskCacheLoadBytes                   prometheus.Counter
	DiskCacheEvictTotal                  prometheus.Counter
	DiskCacheEvictBytes                  prometheus.Counter
	DiskCacheEvictDuration               prometheus.Counter

All these metrics have 0 values. Could you point me to the right metrics? Thanks

[xiaofan-luan]

Hi @yhmo

Thanks for your helpful reply. Are there any metrics available to monitor the cache behavior on query nodes? I am looking at a few metrics

https://github.com/milvus-io/milvus/blob/master/internal/querynodev2/segments/metricsutil/observer.go#L182-L187

	DiskCacheLoadTotal                   prometheus.Counter
	DiskCacheLoadDuration                prometheus.Counter
	DiskCacheLoadBytes                   prometheus.Counter
	DiskCacheEvictTotal                  prometheus.Counter
	DiskCacheEvictBytes                  prometheus.Counter
	DiskCacheEvictDuration               prometheus.Counter

All these metrics have 0 values. Could you point me to the right metrics? Thanks

right now the disk cache is not using since milvus is not evicting.

once eviction is done we should see those metrics