Does the Milvus Distributed Version support high-performance distributed file systems?

[xiyan18]

When deploying Milvus and MinIO in a private cloud environment, Milvus primarily relies on object storage for persistent. storage Deploying MinIO directly on bare disks would face issues such as disk failure maintenance, so it is usually necessary to deploy MinIO on top of a distributed file system or block storage. However, MinIO itself supports redundancy mechanisms such as replication or Erasure Coding (EC), and the underlying storage may also adopt similar redundancy strategies, which can significantly increase storage costs. Additionally, the deployment path for MinIO is typically Milvus node -> MinIO node -> Storage node. Compared to the architecture where Milvus directly connects to a distributed file system (Milvus node -> Storage node), this adds an extra network hop, which not only increases system complexity but may also lead to performance degradation.
Currently, most private cloud storage vendors do not support all-flash S3 storage, while public clouds generally do. Private cloud vendors in China are more inclined to provide high-performance distributed file systems and all-flash block storage to meet the needs of high-performance computing scenarios such as AI training. Moreover, standard S3 storage does not support RDMA technology, and its performance is significantly lower compared to distributed file systems, which also limits the performance of Milvus in private cloud environments.

[yhmo]

So far, milvus supports these storage types:

milvus/configs/milvus.yaml

Line 146 in 9bbaaac

cloudProvider: aws

  # Cloud Provider of S3. Supports: "aws", "gcp", "aliyun".
  # Cloud Provider of Google Cloud Storage. Supports: "gcpnative".
  # You can use "aws" for other cloud provider supports S3 API with signature v4, e.g.: minio
  # You can use "gcp" for other cloud provider supports S3 API with signature v2
  # You can use "aliyun" for other cloud provider uses virtual host style bucket
  # You can use "gcpnative" for the Google Cloud Platform provider. Uses service account credentials

For "support high-performance distributed file systems" -------------------------------- Wait @xiaofan-luan comment

[xiaofan-luan]

Adding support for storage like HDFS or NFS is not very difficult for Milvus.

1. We'd like to learn more about what storage you guys are using
2. We need high performance storage with both Golang and Cpp SDK, or S3 compatible Storage
3. Any contribution is welcomed

[xiyan18]

Adding support for storage like HDFS or NFS is not very difficult for Milvus.

We'd like to learn more about what storage you guys are using
We need high performance storage with both Golang and Cpp SDK, or S3 compatible Storage
Any contribution is welcomed

Thanks @xiaofan-luan .
It’s great to see that Milvus can be easily integrated with POSIX systems. In the field of AI storage, Lustre, GPFS, NFS, and BeeGFS have become mainstream solutions due to their high performance, scalability, and crucial POSIX interface for AI software ecosystems. Lustre holds a dominant market share of 46%, followed by GPFS at 22%, NFS at 8%, and BeeGFS at 6%. These solutions are widely used for large-scale model training. In the private cloud sector, solutions like DDN's Lustre, IBM's GPFS, HPE's Lustre, and NetApp's BeeGFS are extensively adopted. In the public cloud domain, cloud AI storage solutions such as AWS, Azure, and Oracle's Lustre, as well as Alibaba Cloud's GPFS, have become the default choices for GPU clusters. These systems not only support full-path RDMA, TB-level bandwidth, and tens of millions of IOPS but also integrate with S3 storage to enable intelligent data movement, thereby effectively enhancing the efficiency of AI pipelines and reducing storage costs.

ES, Qdrant, pgvector, and other popular open-source vector databases all support POSIX file systems. In contrast, Milvus currently does not support POSIX file systems. This limitation may impact Milvus's ecosystem expansion in the private cloud domain, especially when integration with storage systems that rely on POSIX interfaces is required.

[xiaofan-luan]

Using local file system, milvus should already support any filesystem with POSIX support. This is something really hard for milvus.

The goal of milvus is a cloud native, distributed vectordb with full compuate stoarge disaggreation.

For Zilliz, our customers is fully on public cloud so S3, GCS and OSS is always our first recommendation.

As an opensource community, we definitely welcome all kinds of contribution and it should be a simple code to do so.

[xiaofan-luan]

check localstorage and this is all you need for support a distributed file system

Mean time I may more interested with HDFS

[xiyan18]

OK, we'll test it.