Can there be a data loss during Milvus node failure?

[divya2661]

I am evaluating Milvus for some of our use cases and have a few questions about its capabilities. According to this thread (#16651), Milvus segments remain in memory until they are flushed by default. Does this mean there is a possibility of data loss in such cases? How does Milvus handle data node failures. To give an example, is the following sequence of events possible?

1. Insert data into milvus
2. Milvus returns success
3. Query the data
4. Milvus returns the result (as query node can access growing un-flushed segments )
5. Data node goes down and the in memory segments are not available
6. query milvus again for the same data
7. Milvus doesn't return anything given the data is lost

[yhmo]

A milvus cluster has these components: https://milvus.io/docs/main_components.md
Responsibility of each component:

• Pulsar/Kafka acts as a message queue to store WAL(write-ahead-log) for insert/delete requests, and time-tick message for data consistency. All the inserted data will be stored in Pulsar/Kafka before consumed by datanode/querynode.
• MinIO/S3 is the major data storage of Milvus, all the original data and index data is stored here.
• Etcd stores the system metadata of Milvus, including schema of collections, runtime address of each component, paths of data/index of each segment, etc.
• Proxy receives requests from clients and dispatches requests to other components.
• Query node loads segments into memory for search, including growing segments and sealed segments.
• Data node persists data into segments, merges small segments into large segments.
• Index node builds indexes for sealed segments.
• Root coordinator maintains data consistency and the entire system state. Milvus manages data consistency by time-tick machinery: https://milvus.io/docs/time_sync.md#Time-Synchronization. The root coordinator sends a time-tick for each 200ms, and each "data-channel" will receive the time-tick messages. Once a time-tick message is accepted by all the datanodes and querynodes, we can say the data before this time point is visible and searchable.
• Data coordinator manages data nodes, and assigns tasks to data nodes.
• Query coordinator manages query nodes and index nodes(in old versions, index coordinator manages index nodes, in new version, index coordinator is removed)

When you insert some data by insert():

1. The data is received by proxy node, proxy node splits the data into pieces by shards_num(which is defined in collection schema, default is 1), and sends the pieces(as messages) into Pulsar/Kafka. The insert() returns success to the client once the data is persisted into Pulsar/Kafka.
2. For each growing segment, two "data channels" are created, one is between querynode and Pulsar/Kafka, the other is between datanode and Pulsar/Kafka.
3. Querynode and datanode subscribe messages asynchronously from Pulsar/Kafka, the data pieces are consumed by querynode and datanode.
4. For each growing segment, datanode keeps a buffer to accept new data. Once the buffer size reaches 16MB or the buffer lifetime hits the limit(10 minutes), the buffer is persisted to be a chunk into S3/MinIO.
5. If the collection is loaded, all segments are kept in query nodes. for each growing segment, querynode keeps a buffer to accept new data, so that search requests can search the data. New data is consumed by querynode from Pulsar/Kafka, only when the new data is already consumed by querynode, the data is visible&searchable by search()/query().
6. With more and more chunks persisted, the size of the growing segment reaches a threshold(100+MB), the growing segment is converted to be sealed segment by data coordinator. No more data is allowed to be inserted into this segment. Query node and data node will create a new buffer for the next growing segment.
7. Once a new sealed segment is generated, index node will receive a task to build index for this seal segment.
8. Once an index is finished, the query node will load this index into memory to replace the original segment data.

When you call load(), the collection's data is loaded into the memory of query nodes. Milvus automatically balances the segments across query nodes.

When you call search() on the client side:

1. A search request is sent to a proxy node of Milvus
2. The proxy node passes the request to a "leader query node" which is leading a copy, the query node knows the distribution of all the segments of this copy. Some segments might be loaded in other query nodes. The "leader" passes the request to other query nodes, gets the results from other query nodes, combine the results to a final result and returns the the proxy.
3. The proxy returns the final result to the client

[yhmo]

Since the Pulsar is like a WAL component for Milvus, the data is safe and cannot lost.

Consistency level controls the data visibility for search requests:
https://milvus.io/docs/consistency.md#Consistency