feat: migrate tools support pika v3.5.0 #2984

pro-spild · 2024-12-24T04:55:12Z

feature

pika-migrate-tools支持3.5.0。

Test

写入SET/HSET/LPUSH/SADD/ZADD 各10000000条，然后运行migrate-tool。在迁移期间写入/DEL/HDEL/LPOP/SREM/ZREM各5000000条。在sourceDB和targetDB对操作的key进行一致性测试，结果均一致。

coderabbitai · 2024-12-24T04:55:21Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

src/pika_repl_bgworker.cc

wangshao1 · 2025-01-09T06:24:57Z

src/pika_sender.cc

+          s = cli_->Recv(&resp);
+          if (resp[0] == "OK") {
+          } else {
+            LOG(FATAL) << "Connect to redis(" << ip_ << ":" << port_ << ") Invalid password";


FATAL级别的日志会直接abort进程，所以我理解这个地方打印个ERROR的日志加重试吧。

以修改为ERROR

wangshao1 · 2025-01-09T06:26:30Z

src/pika_sender.cc

+          s = cli_->Recv(&resp);
+          if (s.ok()) {
+            if (resp[0] == "NOAUTH Authentication required.") {
+              LOG(FATAL) << "Ping redis(" << ip_ << ":" << port_ << ") NOAUTH Authentication required";


同上，FATAL级别日志会导致进程直接退出

wangshao1 · 2025-01-09T06:28:34Z

src/pika_sender.cc

+    elements_--;
+    LoadKey(key);
+    cli_->Close();
+    LOG(INFO) <<  s.ToString().data();


LOG（INFO）<< s.ToString()就可以了.
另外，这里应该得判断下返回值，如果返回结果不符合预期，我理解需要打印出具体失败的key和类型，而且是需要一直重试直到成功的，要不就丢数据了。

这里失败会调用LoadKey将key重新存到缓冲区，等待下次重新发送

wangshao1 · 2025-01-09T07:05:15Z

src/pika_sender.cc

+    if (!s.ok()) {
+      delete cli_;
+      cli_ = NULL;
+      LOG(WARNING) << "Can not connect to " << ip_ << ":" << port_ << ", status: " << s.ToString();


如果是connect失败需要一直重试的话，我理解重试connect即可，感觉不需要重新new对象。

done
将connectRedis的逻辑修改为重新connect，cli_初始化放在pikaSender构造函数中

wangshao1 · 2025-01-09T07:07:32Z

src/pika_sender.cc

+        pstd::Status s = cli_->Send(&cmd);
+
+        if (s.ok()) {
+          s = cli_->Recv(&resp);


auth返回值是不是需要判断下，如果是穿的密码是错的，就不用重试了。

wangshao1 · 2025-01-09T07:20:23Z

src/pika_sender.cc

+  LOG(INFO) << "Start sender thread...";
+
+  if (cli_ == NULL) {
+    ConnectRedis();


如果ConnectRedis返回时，cli不能保证一定初始化成功，那我理解你在外面就需要考虑cli是空指针的问题。或者你就直接让进程退出，比如auth 命令返回密码错误，或者服务端需要auth但migrate启动的时候没有设置密码，这种直接就进程退出。

wangshao1 · 2025-01-09T07:25:12Z

src/redis_sender.cc

+      return 0;
+    }
+
+    cli_->Close();


delete cli_

chenbt-hz · 2025-02-28T02:06:10Z

src/pika_conf.cc

+
+    // pika-migrate-tool only support 1 db
+    if (databases_ != 1) {
+      LOG(FATAL) << "pika-migrate-tool only support 1 db";


这块会导致CI过不了
还有个问题：现在pika默认配置就是3db、一般线上都是6db，这种情况怎么处理合适？

wangshao1 · 2025-02-28T09:02:02Z

刚发现，你的改动好像是在pika/src或者是pika/include下，是不应该修改 pika/tools/migrate_tool下呢？

Issues-translate-bot · 2025-02-28T09:02:17Z

Bot detected the issue body's language is not English, translate it automatically.

I just found out that your changes seem to be under pika/src or pika/include. Shouldn't you modify pika/tools/migrate_tool?

pro-spild · 2025-02-28T09:30:05Z

刚发现，你的改动好像是在pika/src或者是pika/include下，是不应该修改 pika/tools/migrate_tool下呢？
3.5当时直接在src下面改了，我不知道重新commit会不会导致你以前的review记录没了，所以想着review结束了再和4.0一样放到tools下面。当时unstable分支下面有这个工具，3.5没有，我就直接在src下面改了。

Issues-translate-bot · 2025-02-28T09:30:18Z

Bot detected the issue body's language is not English, translate it automatically.

I just found out that your changes seem to be under pika/src or pika/include. Shouldn't you modify pika/tools/migrate_tool?
3.5 was changed directly under src at that time. I don’t know if recommit will cause your previous review record to be lost, so I thought that after the review ended, I put it under tools like 4.0. At that time, there was this tool under the unstable branch, but it didn't have it in 3.5, so I changed it directly under src.

guozhihao-224 · 2025-03-03T04:02:59Z

这里我尝试使用该工具进行迁移，发现其仅将sst文件下载到migrate-tool本地，但是没有将命令转发给下游

Issues-translate-bot · 2025-03-03T04:03:10Z

Bot detected the issue body's language is not English, translate it automatically.

Here I tried to use the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream

pro-spild · 2025-03-04T06:50:17Z

这里我尝试使用该工具进行迁移，发现其仅将sst文件下载到migrate-tool本地，但是没有将命令转发给下游

@guozhihao-224 可以贴一下pika_migrate用的conf吗？还有使用场景是啥样的呢？

Issues-translate-bot · 2025-03-04T06:50:30Z

Bot detected the issue body's language is not English, translate it automatically.

Here I tried using the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream

@guozhihao-224 Can you post the conf used for pika_migrate? What are the usage scenarios?

guozhihao-224 · 2025-03-04T09:46:34Z

这里我尝试使用该工具进行迁移，发现其仅将 sst 文件下载到 migrate-tool 本地，但是没有将命令转发给下游

@guozhihao-224可以贴一下pika_migrate用的conf吗？还有使用场景是啥样的呢？
是pika3.5.5 迁移到 3.5.5版本，100GB量级的数据

这里测试后，发现有将命令往下转发给下游，但是我发现，设置redis-sender-num大于8的话，就会在转发一部分数据后卡住不动了，设置成其他值会的比如5 — 8 之间，也会突然出现卡住不执行的状态。有试过设置为2能够跑完迁移任务，但是设置为2后，迁移时间太长了，100GB的数据的迁移1个多小时

是否有死锁的情况出现
对应的conf：
pika-migrate.txt

Issues-translate-bot · 2025-03-04T09:46:48Z

Bot detected the issue body's language is not English, translate it automatically.

Here I tried using the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream

@guozhihao-224 Can you post the conf used by pika_migrate? What are the usage scenarios?
After testing here, I found that the command is forwarded downstream to the downstream, but I found that if the redis-sender-num is greater than 8, it will be stuck after forwarding part of the data, but there is no problem with setting 8. This may be due to the influence of that direction setting.

Corresponding conf:
pika-migrate.txt

pro-spild · 2025-04-03T05:30:23Z

这里我尝试使用该工具进行迁移，发现其仅将 sst 文件下载到 migrate-tool 本地，但是没有将命令转发给下游

@guozhihao-224可以贴一下pika_migrate用的conf吗？还有使用场景是啥样的呢？
是pika3.5.5 迁移到 3.5.5版本，100GB量级的数据

这里测试后，发现有将命令往下转发给下游，但是我发现，设置redis-sender-num大于8的话，就会在转发一部分数据后卡住不动了，设置成其他值会的比如5 — 8 之间，也会突然出现卡住不执行的状态。有试过设置为2能够跑完迁移任务，但是设置为2后，迁移时间太长了，100GB的数据的迁移1个多小时

是否有死锁的情况出现对应的conf： pika-migrate.txt

这边可能还需要提供一下pika_migrate的日志，我这边测试5-8之间同量级的数据均没有出现卡住的情况。

Issues-translate-bot · 2025-04-03T05:30:34Z

Bot detected the issue body's language is not English, translate it automatically.

Here I tried using the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream

@guozhihao-224 Can you post the conf used by pika_migrate? What are the usage scenarios?
is pika3.5.5 migrated to 3.5.5 version, 100GB data

After testing here, I found that the command is forwarded downstream to the downstream, but I found that if the redis-sender-num is greater than 8, it will be stuck after forwarding part of the data. If the setting to other values, such as between 5 and 8, it will suddenly become stuck and not executed. I have tried setting it to 2 to complete the migration task, but after setting it to 2, the migration time is too long, and the migration of 100GB of data will take more than an hour.

Whether there is a deadlock? Corresponding conf: pika-migrate.txt

Here, you may also need to provide the log of pika_migrate. I tested that the data of the same magnitude between 5-8 did not get stuck.

chejinge · 2025-08-28T06:49:01Z

创建了新的分支，pr就不会有这么多文件修改和冲突，功能完成之后编译出二进制再放入migrate_tool，已经新建分支

Issues-translate-bot · 2025-08-28T06:49:10Z

Bot detected the issue body's language is not English, translate it automatically.

After creating a new branch, there will be no so many file modifications and conflicts in pr. After the function is completed, compile the binary and put it in migrate_tool. A new branch has been created.

… synchronization exceptions

github-actions bot added Invalid PR Title ✏️ Feature New feature or request 📒 Documentation Improvements or additions to documentation labels Dec 24, 2024

pro-spild changed the title ~~feature: migrate tools support pika v3.5.0~~ feat: migrate tools support pika v3.5.0 Dec 24, 2024

github-actions bot removed the Invalid PR Title label Dec 24, 2024

wangshao1 requested review from chenbt-hz and wangshao1 January 3, 2025 12:09

pro-spild force-pushed the pika-migrate-v3.5.0 branch from 188566a to e744786 Compare January 6, 2025 09:03

wangshao1 reviewed Jan 9, 2025

View reviewed changes

chenbt-hz reviewed Feb 28, 2025

View reviewed changes

wangshao1 requested a review from baixin01 March 7, 2025 10:10

chejinge changed the base branch from 3.5 to pika-migrate-v3.5 August 28, 2025 06:45

pro-spild force-pushed the pika-migrate-v3.5.0 branch from 5c1cb48 to 8b2fca9 Compare September 8, 2025 07:09

feature: migrate tools support pika v3.5.0

406777f

pro-spild force-pushed the pika-migrate-v3.5.0 branch from 8b2fca9 to 406777f Compare September 8, 2025 08:11

fix: different versions of Cmd::ToRedisProtocol may cause incremental…

cfd83b2

… synchronization exceptions

chejinge merged commit bd84cf7 into OpenAtomFoundation:pika-migrate-v3.5 Sep 21, 2025
2 checks passed

feat: migrate tools support pika v3.5.0 #2984

feat: migrate tools support pika v3.5.0 #2984

Uh oh!

Conversation

pro-spild commented Dec 24, 2024

feature

Test

Uh oh!

coderabbitai bot commented Dec 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangshao1 commented Feb 28, 2025

Uh oh!

Issues-translate-bot commented Feb 28, 2025

Uh oh!

pro-spild commented Feb 28, 2025

Uh oh!

Issues-translate-bot commented Feb 28, 2025

Uh oh!

guozhihao-224 commented Mar 3, 2025

Uh oh!

Issues-translate-bot commented Mar 3, 2025

Uh oh!

pro-spild commented Mar 4, 2025

Uh oh!

Issues-translate-bot commented Mar 4, 2025

Uh oh!

guozhihao-224 commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Issues-translate-bot commented Mar 4, 2025

Uh oh!

pro-spild commented Apr 3, 2025

Uh oh!

Issues-translate-bot commented Apr 3, 2025

Uh oh!

chejinge commented Aug 28, 2025

Uh oh!

Issues-translate-bot commented Aug 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

coderabbitai bot commented Dec 24, 2024 •

edited

Loading

guozhihao-224 commented Mar 4, 2025 •

edited

Loading