mc mirror takes long time when target bucket contains a lot of files

[archsh]

Expected behavior

Using mc mirror to mirror local folder to a target bucket, it should be exit soon if local folder is empty or few files.

Actual behavior

It took a long time to start transfer files if the target bucket contains a lot of objects.

Steps to reproduce the behavior

mc mirror --json ./localfolder target/bucket/path

mc --version

• (paste output of mc --version)
  mc version RELEASE.2024-01-05T05-04-32Z (commit-id=59eca9fea8984adec1e8e7a1c95d0ea23107ceff)

System information

CentOS release 6.9 (Final) x64

[zveinn]

Yes, mc mirror does a full source to target comparison when it's ran. This is due to the --remove flag. However, it might be possible to speed up the mirror given that the --remove flag is not present. I will look into this in the coming weeks, no promises on delivery date though since my schedule is rather full.

[PrisedRabbit]

Same here. It takes a lot of time to mirror folder with 20 000+ files ( using aws cli solved the problem )

[zveinn]

I will be looking into this next week, if there is any additional information you want to share, then now is the time :)

[karsapeng]

helloHello, I wrote a script to use “mc mirror --remove --overwrite --limit-download $LIMIT_DOWNLOAD --limit-upload $LIMIT_UPLOAD minio-master minio-slave” to synchronize the data of the master node. Even if the data of the two minio nodes are consistent, if the master node is shut down and restarted due to a fault, I will change the direction of the synchronization and transfer the data to the slave node. Then the master will run “mc mirror --remove --overwrite --limit-download $LIMIT_DOWNLOAD --limit-upload $LIMIT_UPLOAD minio-slave minio-master”for synchronization. However, even if the data of the two minio nodes is consistent, minio-master will perform a full synchronization, which will take up a lot of io and cause the server to crash. My mc version is mc:RELEASE.2024-08-26T10-49-58Z

[karsapeng]

Hello, has this problem been solved?

[zveinn]

Hey, I have not had time to look into this yet. Just as I was about to start, something else came up. I will see if someone else can take it off my hands.

[karsapeng]

hello，Is there any progress on this issue？I'm a little troubled with this mirror function :)

[klauspost]

There is no reason to add more clutter to the issue. This is a low priority item, and will be done when there is bandwidth for it.

[fherenius]

One thing to add, I’m using mc mirror to move a cluster. The target cluster already has several million objects in it.

mc mirror will do a full synchronization of the target, even for buckets that don’t exist on the source. (Visible when running with —debug). It would make it a lot faster if it would filter buckets on name first, instead of content.

Or maybe, —exclude-bucket should also count for the target synchronization.

[zveinn]

@fherenius @karsapeng @PrisedRabbit @archsh
Sorry how long this took, minio is growing fast and we've been super busy during the last few months.

#5116

This allows mc to exit early when the local bucket has synced when --remove is not used.

Please try it out and let me know if this improves things.

[zveinn]

NOTE: my implementation is not going to do the job. I'll be amending the PR tomorrow