Container milvus-standalone stop after i search database

[HoangSang1510]

Hello, i am using mivuls 2.4.23, when i query, container milvus-standalone has stoped. When call docker compose ps, etcd and minio still working but not found milvus-standalone. result log i got:

[2025-06-16 14:46:16,724] INFO in milvus: [MILVUS] Query Error: <MilvusException: (code=<bound method _MultiThreadedRendezvous.code of <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer", grpc_status:14, created_time:"2025-06-16T14:46:16.72299427+07:00"}"
>>, message=Retry timeout: 4s, message=failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer)>

full log is:
milvus.log

[yhmo]

The log shows it received "list database" and "describe database" requests, and the requests were successfully completed:

milvus-standalone  | [2025/06/16 07:46:12.223 +00:00] [INFO] [rootcoord/root_coord.go:950] ["received request to list databases"] [traceID=29f7043751f55936e3e68784fe87464b] [msgID=0]
milvus-standalone  | [2025/06/16 07:46:12.223 +00:00] [INFO] [rootcoord/root_coord.go:976] ["done to list databases"] [traceID=29f7043751f55936e3e68784fe87464b] [msgID=0] ["num of databases"=1]
milvus-standalone  | [2025/06/16 07:46:12.224 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=29f7043751f55936e3e68784fe87464b] [sdk_type=Python] [sdk_version=2.4.4] [local_time="2025-06-16 14:46:12.219447"] [user=] [host=ims5-server] [identifier=458767721358163972] [last_active_time=2025/06/16 07:46:12.224 +00:00]
milvus-standalone  | [2025/06/16 07:46:12.240 +00:00] [INFO] [rootcoord/root_coord.go:3147] ["received request to describe database "] [traceID=1a4382d3533b949edd19411a4fe281e5] [dbName=default]
milvus-standalone  | [2025/06/16 07:46:12.240 +00:00] [INFO] [rootcoord/root_coord.go:3171] ["done to describe database"] [traceID=1a4382d3533b949edd19411a4fe281e5] [dbName=default] [ts=458767721358163981]

I didn't see any critical errors in the log, seems the standalone works well.

The error "Connection reset by peer" could be a network problem, not milvus error.

[yhmo]

No crash callstacks in the log.log.
I think it has nothing to do with avx. Do you have a script to reproduce the crash?

[HoangSang1510]

# -*- Milvus 2.4 -*-
import logging
import pickle
logging.basicConfig(
    level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
from pymilvus import MilvusClient


logging.info(">>> BEGIN SCRIPT TEST CONNECTION <<<")
URI = "http://localhost:19530"

client = MilvusClient(uri=URI, timeout=5.0)
collection_name = 'test_v4'

logging.info("[Load DB Success]")
logging.info("[Client] " + str(client))

result = client.query(
    collection_name=collection_name,
    filter="id > 0",
    output_fields=["id"],
    consistency_level='Strong',
    timeout=5
)
# Working
logging.info('[QUERY Result] ' + str(result))

file = open('image_test/face_vector.pkl', 'rb')
vector = pickle.load(file)
logging.info("Lenght vector: " + str(len(vector)))

search_params = {
    "metric_type": "COSINE",
    "params": { "radius": 0.4, "range_filter": 1.2}}

res_search = client.search(
    collection_name=collection_name,
    data=[vector],
    anns_field='vector',
    output_fields=['id'],
    search_params=search_params,
    timeout=5)
# Crash here
logging.info('[SEARCH Result] ' + str(res_search))

logging.info(">>> RUN SCRIPT TEST DB CONNECTION DONE <<<")

logging i got:

2025-06-16 18:54:59,420 - INFO - >>> BEGIN SCRIPT TEST CONNECTION <<<
2025-06-16 18:54:59,442 - DEBUG - Created new connection using: 096ef25b423a46919b5a02fe4aec7c9a
2025-06-16 18:54:59,442 - INFO - [Load DB Success]
2025-06-16 18:54:59,442 - INFO - [Client] <pymilvus.milvus_client.milvus_client.MilvusClient object at 0x7f4c6454a7d0>
2025-06-16 18:54:59,748 - INFO - [QUERY Result] data: ["{'id': 999999}"] , extra_info: {'cost': 0}
2025-06-16 18:54:59,750 - INFO - Lenght vector: 512
2025-06-16 18:55:00,482 - WARNING - [search] retry:4, cost: 0.27s, reason: <_InactiveRpcError: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer>
2025-06-16 18:55:00,753 - WARNING - [search] retry:5, cost: 0.81s, reason: <_InactiveRpcError: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer>
2025-06-16 18:55:01,565 - WARNING - [search] retry:6, cost: 2.43s, reason: <_InactiveRpcError: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer>
2025-06-16 18:55:03,997 - WARNING - [search] retry:7, cost: 3.00s, reason: <_InactiveRpcError: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer>
2025-06-16 18:55:06,999 - ERROR - RPC error: [search], <MilvusException: (code=<bound method _InactiveRpcError.code of <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer", grpc_status:14, created_time:"2025-06-16T18:55:06.999326511+07:00"}"
>>, message=Retry timeout: 5s, message=failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer)>, <Time:{'RPC start': '2025-06-16 18:54:59.750320', 'RPC error': '2025-06-16 18:55:06.999754'}>
2025-06-16 18:55:07,000 - ERROR - Failed to search collection: test_v4
Traceback (most recent call last):
  File "/home/ims/code/ims_id_server/error.py", line 36, in <module>
    res_search = client.search(
                 ^^^^^^^^^^^^^^
  File "/home/ims/.pyenv/versions/ims_id/lib/python3.11/site-packages/pymilvus/milvus_client/milvus_client.py", line 331, in search
    raise ex from ex
  File "/home/ims/.pyenv/versions/ims_id/lib/python3.11/site-packages/pymilvus/milvus_client/milvus_client.py", line 317, in search
    res = conn.search(
          ^^^^^^^^^^^^
  File "/home/ims/.pyenv/versions/ims_id/lib/python3.11/site-packages/pymilvus/decorators.py", line 147, in handler
    raise e from e
  File "/home/ims/.pyenv/versions/ims_id/lib/python3.11/site-packages/pymilvus/decorators.py", line 143, in handler
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ims/.pyenv/versions/ims_id/lib/python3.11/site-packages/pymilvus/decorators.py", line 182, in handler
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ims/.pyenv/versions/ims_id/lib/python3.11/site-packages/pymilvus/decorators.py", line 93, in handler
    raise MilvusException(e.code, f"{to_msg}, message={e.details()}") from e
pymilvus.exceptions.MilvusException: <MilvusException: (code=<bound method _InactiveRpcError.code of <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer", grpc_status:14, created_time:"2025-06-16T18:55:06.999326511+07:00"}"
>>, message=Retry timeout: 5s, message=failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:19530: recvmsg:Connection reset by peer)>

after this log, container of milvus standalone is disappeared when i using docker compose ps.

log file i have:
milvus.log

[yhmo]

Your script works well on my local.
You can replace this line to a random vector to see if it still crash:

vector = pickle.load(file)

replace with:

vector = [1.0 for _ in range(dim)]

Seems the list_databases() is not called in the script.

[HoangSang1510]

Thanks you, I think the crash is due to my Ubuntu device lacking AVX-512 support. The same code runs normally on devices that support AVX-512.