Migrating data to another bucket

[artesby]

Hello
I want to migrate data to a new s3 bucket. My plan is:

1. stop milvus
2. rclone data to new bucket (preserving old rootPath: "/")
3. update values.yaml with new external s3 configurations
4. run milvus

Question is - does milvus rely on bucket name (and store it etcd)? Are these steps enough to migrate data to new s3 bucket, or i have to do backup + restore?

[yhmo]

MinIO/S3 is the major data storage of Milvus, all the original data files and index data files are stored here.
Etcd stores the system metadata of Milvus, including the schema of collections, the runtime address of each component, the paths of data/index of each segment, etc.

Before any compaction/index/load tasks, Milvus reads the paths of segments from etcd, to know where to get the data files from minio/s3. So, the data in etcd and the data files in minio/s3 must be strongly consistent, otherwise, milvus will run into trouble.
In runtime, milvus might occasionally persist data into minio/s3 and etcd. It is difficult to keep the consistency by manually copying the data from minio and etcd respectively.
The milvus-backup is the recommended tool to backup/restore data of milvus.

[artesby]

Thanks for your reply
I understand the purpose of s3 and etcd, what am i asking is - does etcd contain absolute paths of files in s3 (including s3 host and s3 bucket name) or relative paths (only starting from rootPath). If i stop milvus, then rsync two buckets, then start milvus pointing to new bucket (relative paths won't change) -- will milvus be able to read the data from new source?

[yhmo]

Just confirmed, in earlier versions(2.2/2.3), there were relative paths stored in etcd. Now, ids of collection/partition/segment/field/log_file are stored in etcd, milvus reads the ids from etcd and combines the path in memory. So, theoretically, it is ok that "stop milvus, then rsync two buckets, then start milvus pointing to new bucket".