question abot mmap

[rere950303]

• milvus version: v2.4.20
• milvus mode: cluster

---

We are currently operating 5 collections and would like to enable mmap.
We want to apply mmap to all fields — including vectorField, vectorIndex, scalarField, and scalarIndex — as well as to all newly inserted data.

Based on the documentation (https://milvus.io/docs/v2.4.x/mmap.md), if we use the following, Will this apply mmap to all fields?

collection.set_properties({'mmap.enabled': True})

collection.alter_index(
    index_name="vector_index",  # Replace with your vector index name
    extra_params={"mmap.enabled": True}  # Enable memory mapping for index
)

---

Or, for already deployed Milvus instances, is it possible to apply mmap using the following method in the configuration file?

queryNode:
  mmap:
    vectorField: false  # Enable mmap for loading vector data
    vectorIndex: false  # Enable mmap for loading vector index
    scalarField: false  # Enable mmap for loading scalar data
    scalarIndex: false  # Enable mmap for loading scalar index

[rere950303]

@yhmo thanks for your works! Your detailed answers are always very helpful, and I really appreciate it.
could you check this issue plz 🙇 ?

[yhmo]

"mmap.enabled" is applied to both scalar field and vector field.
In the source code, it first checks whether the collection schema has "mmap.enabled" property, load mmap index/scalar if the "mmap.enabled" exists and is true. Otherwise, use the milvus.yaml queryNode.mmap configurations to determine:

milvus/internal/querynodev2/segments/utils.go

Line 265 in 679a145

enableMmap, exist := common.IsMmapIndexEnabled(indexInfo.IndexParams...)

milvus/internal/querynodev2/segments/utils.go

Line 283 in 679a145

enableMmap, exist := common.IsMmapDataEnabled(fieldSchema.GetTypeParams()...)

[rere950303]

@yhmo
When running collection.set_properties({'mmap.enabled': True}), does that mean the return value of common.IsMmapDataEnabled(fixedSchema.GetTypeParams()...) will be True for all fields?

The documentation says:

After 2.4.10, the memory mapping settings within a collection utilize the add_field method. Here, you can toggle mmap_enabled between True or False as needed.

This part is confusing to me.

[yhmo]

Three entries to configure mmap:

1. milvus.yaml

queryNode:
  mmap:
    vectorField: false # Enable mmap for loading vector data
    vectorIndex: false # Enable mmap for loading vector index
    scalarField: false # Enable mmap for loading scalar data
    scalarIndex: false # Enable mmap for loading scalar index

2. collection level
   collection.set_properties({'mmap.enabled': True})
   In this doc mentioned: "At the collection level, memory mapping is applied to all unindexed raw data within a collection, excluding primary keys, timestamps, and row IDs. "

3. index level
   schema.add_field(field_name="embedding", datatype=DataType.FLOAT_VECTOR, dim=768, mmap_enabled=True)
   The doc mentioned: "For index-level settings, memory mapping can be specifically applied to vector indexes without affecting other data types. "

So, collection.set_properties({'mmap.enabled': True}) has higher priority thanqueryNode.mmap.vectorField/scalarField/scalarIndex of milvus.yaml for a collection.
schema.add_field(field_name="embedding", datatype=DataType.FLOAT_VECTOR, dim=768, mmap_enabled=True) has higher priority than the queryNode.mmap.vectorIndex of milvus.yaml for a field.

[rere950303]

thanks!