channel not available: can't search after docker down and up

[miguelwon]

I'm working in a single machine with docker compose. I set it up, created the database, schema, collection, indexes and populated. Everything was working fine. I then took the Docker containers down (with docker-compose down) and brought them back up. The service runs, and I can retrieve the collection list. However, when performing a simple search, I get the following error:

2025-05-04 09:26:41,172 [ERROR][handler]: RPC error: [search], <MilvusException: (code=503, message=failed to search: segment lacks[segment=457772913211990950]: channel not available[channel=by-dev-rootcoord-dml_0_457772913194434681v0])>, <Time:{'RPC start': '2025-05-04 09:26:20.159896', 'RPC error': '2025-05-04 09:26:41.172557'}> (decorators.py:140)
2025-05-04 09:26:41,172 [ERROR][search]: Failed to search collection: dgsi_chunks (milvus_client.py:411)
Error in dense_emb search: <MilvusException: (code=503, message=failed to search: segment lacks[segment=457772913211990950]: channel not available[channel=by-dev-rootcoord-dml_0_457772913194434681v0])>

This is the docker-compose.yml I used:

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.18
    mem_limit: 1G
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    mem_limit: 2G
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.5.10
    mem_limit: 48G
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - /home/admin/milvus/milvus.yaml:/milvus/configs/milvus.yaml
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"

networks:
  default:
    name: milvus

[yhmo]

Please offer the full logs for investigation.
How many collections/partitions in the milvus?

You can release and load the collection again.

[miguelwon]

I have a single collection. I'm not able to release and load because I get another error related with OOM. I'm now trying to create the index in-disk but is not working either. The indexes creation is using a lot of memory, and I suspect is because of a sparse vector index that is using almost all my ram memory.

[yhmo]

Your milvus is a standalone, all components(queryNode, dataNode, indexNode) are running in the same process. All the tasks(compaction tasks, build index tasks, loading tasks) also use the same RAM. The memory usage peak could be very high and easily lead to OOM if the physical RAM is insufficient.

Disk-index is only for dense vectors. It sounds like your collection has multi-vector fields: one dense vector field and one sparse vector field. How many rows are in the collection?

There is a command-line tool named "birdwatcher" that can force release a collection.

The steps to force release the collection:

1. Install the birdwatcher tool: https://milvus.io/docs/birdwatcher_install_guides.md, you can download the already-built binary by "wget -O birdwatcher.tar.gz
   https://github.com/milvus-io/birdwatcher/releases/download/v1.0.8/birdwatcher_Linux_x86_64.tar.gz"
2. Stop your milvus service by command "docker compose stop"
3. Use the command "docker ps -a" to show all the containers, find out the ID of the Etcd container. And use the command "docker start [etcd container id]" to start the etcd container
4. Extract the birdwatcher.tar.gz and run the already-built binary
5. Now you are in the command line mode of birdwatcher, type the following commands to force release all the collections:

connect --etcd 127.0.0.1:2379
force-release --all
exit

6. Use the command "docker compose restart" to restart the milvus service

[miguelwon]

@yhmo thanks. I started everything from scratch, this time trying to index dense only. I'm limiting the RAM to 24GB, using float16 and diskann, and even so I'm still getting OOM when I'm reaching about 6M vectors (1024 dim each).
Can you tell me if it will be possible to fit in 24GB 10M vectors of 1024 dim, float16?

[xiaofan-luan]

M vectors (

I think 6M 1024dim can be fit into 24G machines.

can you share your logs so we can investigate on it?

[yhmo]

Different Client-side operations might have different memory usage.
Usually, we have two optional operations:
(1) create_collection ---> create_index ---> load_collection ---> insert 6M vectors
(2) create_collection ---> create_index ---> insert 6M vectors ---> wait index done ---> load_collection

The memory usage peak of (1) is much higher than (2), because index tasks, compaction tasks, and loading tasks are executed at the same time in one process.

An example of (1)

import random
import time

from pymilvus import (
    MilvusClient, DataType,
)

client = MilvusClient(uri="http://localhost:19530")
print(client.get_server_version())

collection_name = "AAA"
dim = 1024

def gen_embedding():
    return [random.random() for _ in range(dim)]

schema = MilvusClient.create_schema()
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=dim)

index_params = client.prepare_index_params()
index_params.add_index(
    field_name="vector",
    index_type="DISKANN",
    metric_type="L2",
)

client.drop_collection(collection_name=collection_name)
client.create_collection(
    collection_name=collection_name,
    schema=schema,
    index_params=index_params,
)
print(collection_name, "created")

print("loading...")
client.load_collection(collection_name=collection_name)

batch = 10000
for i in range(600):
    print("insert", i)
    data = [{"vector": gen_embedding()} for k in range(batch)]
    client.insert(collection_name=collection_name, data=data)

An example of (2)

import random
import time

from pymilvus import (
    MilvusClient, DataType,
)

client = MilvusClient(uri="http://localhost:19530")
print(client.get_server_version())

collection_name = "AAA"
dim = 1024

def gen_embedding():
    return [random.random() for _ in range(dim)]

schema = MilvusClient.create_schema()
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=dim)

index_params = client.prepare_index_params()
index_params.add_index(
    field_name="vector",
    index_type="DISKANN",
    metric_type="L2",
)

client.drop_collection(collection_name=collection_name)
client.create_collection(
    collection_name=collection_name,
    schema=schema,
    index_params=index_params,
)
print(collection_name, "created")

batch = 10000
for i in range(600):
    print("insert", i)
    data = [{"vector": gen_embedding()} for k in range(batch)]
    client.insert(collection_name=collection_name, data=data)

print("flushing...")
client.flush(collection_name=collection_name)

print("wait index finish...")
while True:
    time.sleep(3)
    desc = client.describe_index(collection_name=collection_name, index_name="vector")
    print(desc)
    if desc['indexed_rows'] >= desc['total_rows']:
        break

print("loading...")
client.load_collection(collection_name=collection_name)

I compare the two cases on my machine, with 600K vectors. The memory usage peak of (1) is 8.5 GB, (2) is 4.5 GB.
You can try (2) with 6M vectors.

[xiaofan-luan]

also did you tune any of the parameters like segment size?

[miguelwon]

@xiaofan-luan is there a method to get the logs or are you referring to the docker logs (they are big...)? I haven't tune. Used the default values.

@yhmo those values are quite larger than expected, no? For example HNSW memory estimation is (d * 4 + M * 2 * 4), which gives about 2.6GB, for 600k of 1024 and M=30 (I'm assuming is the default value). And this is for memory index. I was expecting that DiskANN would use much less memory.

I just did a test to index 10M of random vectors (fp16) and still having the same OOM. You can replicate with the following:

from pymilvus import MilvusClient, DataType
import numpy as np
from tqdm import tqdm

client = MilvusClient(
    uri="http://localhost:19530",
    token="root:Milvus"
)
client



schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=True,
)

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="dense_emb", datatype=DataType.FLOAT16_VECTOR, dim=1024)


index_params = client.prepare_index_params()
index_params.add_index(
    field_name="dense_emb",
    index_type="DISKANN",
    metric_type="IP",
)



schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=True,
)

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="dense_emb", datatype=DataType.FLOAT16_VECTOR, dim=1024)


index_params = client.prepare_index_params()
index_params.add_index(
    field_name="dense_emb",
    index_type="DISKANN",
    metric_type="IP",
)

client.drop_collection(collection_name="dense_index")
client.create_collection(
    collection_name="dense_index",
    schema=schema,
    index_params=index_params,
)

def gen_embedding():
    # Generate random vector of size 1024
    random_vector = np.random.randn(1024)

    # Normalize the vector (L2 normalization)
    normalized_vector = random_vector / np.linalg.norm(random_vector)

    # Convert to float16
    embedding = normalized_vector.astype(np.float16)
    return embedding



batch = 1000
for i in tqdm(range(10_000)):
    data = [{"dense_emb": gen_embedding()} for k in range(batch)]
    client.insert(collection_name="dense_index", data=data)

The docker compose:

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.18
    mem_limit: 1G
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    mem_limit: 2G
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.5.10
    mem_limit: 24G
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - /home/admin/milvus/milvus.yaml:/milvus/configs/milvus.yaml
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"

networks:
  default:
    name: milvus

and the milvus.yaml

[yhmo]

@miguelwon
Your milvus is standalone, you can use docker command "docker logs [milvus_container_ID] > 1.log" to export its log.

I tested your script on my machine(64GB RAM).
After inerted 5000+ batches, the memory usage already exceeds 25GB.

########################################################################
It is a long story about index/compaction/load.

First of all, if you called create_collection() with "index_params", the create_collection() will called load_collection() atuomatically, the source code is here.

client.create_collection(
    collection_name="dense_index",
    schema=schema,
    index_params=index_params,
)

With continual insertion, data is accumulated in a cache of datanode, Once the size of the cache exceeds 100+ MB, datanode will flush it to the storage to be a sealed segment.

As we know, in milvus, each segment size is expected to be 1GB. So, at the same time, datanode triggers compaction tasks to merge small segments into large segments. With continual insertion, each compaction task might merge 2 ~ 3 small segments(100+ MB) to be a 300+ MB segment. And with more and more 300+ MB segments generated, compaction tasks will merge 2 ~ 3 segments(300+ MB) to be a 1000+ MB segment. Since standalone only has one datanode, there might be hundreds of compaction tasks executed one by one over a very long time(maybe several hours).

At the same time, once a sealed segment is generated, indexnode will trigger a task to build index for this segment. So, there might be hundreds of tasks executed one by one over a very long time(maybe several hours).

For the segments that have no index, since the collection is loaded, querynode will load its raw data(original vector data) into memory. You will see memory usage continually increase.

The index tasks are much slower than insertion and compaction. Most of the segments are loaded with original vector data instead of index data. So, the memory usage is far more than index size.

Each compaction task will occupy memory, maybe 1GB.
For DISKANN index, each index task will occupy memory, maybe 2 ~ 3GB

The size of 5000 * 1000 float16 vectors is 1024 * 2 bytes * 5000 * 1000 = 10GB.
So, theoretically, there should be 13 ~ 15GB memory usage with your script after 5000+ batches are inserted. 25GB is far beyond expected, let me do more investigation.
########################################################################

To avoid it, you can call create_collection() without "index_params", and create_index() after all data is inserted.
You can try this script:

from pymilvus import MilvusClient, DataType
import numpy as np
import time
from tqdm import tqdm

client = MilvusClient(
    uri="http://localhost:19530",
    token="root:Milvus"
)


schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=True,
)

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="dense_emb", datatype=DataType.FLOAT16_VECTOR, dim=1024)


index_params = client.prepare_index_params()
index_params.add_index(
    field_name="dense_emb",
    index_type="DISKANN",
    metric_type="IP",
)



schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=True,
)

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="dense_emb", datatype=DataType.FLOAT16_VECTOR, dim=1024)


client.drop_collection(collection_name="dense_index")
client.create_collection(
    collection_name="dense_index",
    schema=schema,
)

def gen_embedding():
    # Generate random vector of size 1024
    random_vector = np.random.randn(1024)

    # Normalize the vector (L2 normalization)
    normalized_vector = random_vector / np.linalg.norm(random_vector)

    # Convert to float16
    embedding = normalized_vector.astype(np.float16)
    return embedding



batch = 1000
for i in tqdm(range(10_000)):
    data = [{"dense_emb": gen_embedding()} for k in range(batch)]
    client.insert(collection_name="dense_index", data=data)

print("flush...")
client.flush(collection_name="dense_index")

print("create index...")
index_params = client.prepare_index_params()
index_params.add_index(
    field_name="dense_emb",
    index_type="DISKANN",
    metric_type="IP",
)
client.create_index(collection_name="dense_index", index_params=index_params)

print("wait index finish...")
while True:
    time.sleep(3)
    desc = client.describe_index(collection_name="dense_index", index_name="dense_emb")
    print(desc)
    if desc['indexed_rows'] >= desc['total_rows']:
        break

print("loading...")
client.load_collection(collection_name="dense_index")

With this script, the memory usage is kept at a low level(< 1GB) during the entire insertion.

[xiaofan-luan]

@xiaofan-luan is there a method to get the logs or are you referring to the docker logs (they are big...)? I haven't tune. Used the default values.

@yhmo those values are quite larger than expected, no? For example HNSW memory estimation is (d * 4 + M * 2 * 4), which gives about 2.6GB, for 600k of 1024 and M=30 (I'm assuming is the default value). And this is for memory index. I was expecting that DiskANN would use much less memory.

I just did a test to index 10M of random vectors (fp16) and still having the same OOM. You can replicate with the following:

from pymilvus import MilvusClient, DataType
import numpy as np
from tqdm import tqdm

client = MilvusClient(
    uri="http://localhost:19530",
    token="root:Milvus"
)
client



schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=True,
)

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="dense_emb", datatype=DataType.FLOAT16_VECTOR, dim=1024)


index_params = client.prepare_index_params()
index_params.add_index(
    field_name="dense_emb",
    index_type="DISKANN",
    metric_type="IP",
)



schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=True,
)

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="dense_emb", datatype=DataType.FLOAT16_VECTOR, dim=1024)


index_params = client.prepare_index_params()
index_params.add_index(
    field_name="dense_emb",
    index_type="DISKANN",
    metric_type="IP",
)

client.drop_collection(collection_name="dense_index")
client.create_collection(
    collection_name="dense_index",
    schema=schema,
    index_params=index_params,
)

def gen_embedding():
    # Generate random vector of size 1024
    random_vector = np.random.randn(1024)

    # Normalize the vector (L2 normalization)
    normalized_vector = random_vector / np.linalg.norm(random_vector)

    # Convert to float16
    embedding = normalized_vector.astype(np.float16)
    return embedding



batch = 1000
for i in tqdm(range(10_000)):
    data = [{"dense_emb": gen_embedding()} for k in range(batch)]
    client.insert(collection_name="dense_index", data=data)

The docker compose:

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.18
    mem_limit: 1G
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    mem_limit: 2G
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.5.10
    mem_limit: 24G
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - /home/admin/milvus/milvus.yaml:/milvus/configs/milvus.yaml
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"

networks:
  default:
    name: milvus

and the milvus.yaml

can you give us some details number of how much memory consumes with HNSW and disk ANN on 10M fp16 data?

here might be some reason:

1. FP16 in opensource milvus is not a native data type, we still use fp32 to store all the data and might case extra memory usage.
2. Index node may take extra memory, especially under diskANN, index build itself could be much slower

We are expecting to see HNSW takes 30G memory for load only and DiskANN take 12G with querynode. But also indexnode and datanode will cause extra memory and especially under a standalone node, index build might not be able to catch up if you write too fast so it's highly likely data is still in memory and served as in memory.

Suggestions:

I would suggest to try at least 48G memory and see if it works.

If you are using diskann(make sure you are using nvme SSD), it should work. Maybe reduce the ingestion speed and see if indexnode can catch up index build speed

if memory is critical and performance is not important, you can try HNSW + mmap to reduce the memory cost.

If accuracy is not critical, try HNSW-SQ8 or IVF sq8 instead.

If you have scalar datas, don't forget to mmap it as well.

[miguelwon]

@yhmo and @xiaofan-luan , thanks for the support. It's working now with the @yhmo proposal of first inserting and only then creating the index. It uses about 11GB of RAM.
One thing that I'm particularly impressed is how fast the query is. I'm measuring 10ms of latency.

[yhmo]

DISKANN is slower than in-memory index(such as HNSW). Disk performance also affects DISKANN performance. So, NVMe disk is recommended.
Normally, for millions of data, with proper index parameters and search parameters, the search latency of HNSW is a few milliseconds, usually no more than 10ms. The latency of DISKANN could be 5x ~ 10x of HNSW.