[Feature] Support resending fuse requests when restoring nydusd. #1740

lsc2001 · 2025-08-16T04:38:14Z

Relevant Issue

Details

Support resending the fuse request when nydusd undergoes a hot upgrade or during a fault recovery process.
Requests can be resend by fuse device and sysfs.

Types of changes

What types of changes does your PullRequest introduce? Put an x in all the boxes that apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation Update (if none of the other choices apply)

codecov · 2025-08-16T04:43:05Z

Codecov Report

❌ Patch coverage is 9.33333% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.48%. Comparing base (0c53e00) to head (76a3f7a).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
service/src/fusedev.rs	0.00%	62 Missing ⚠️
service/src/upgrade.rs	58.33%	5 Missing ⚠️
src/bin/nydusd/main.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1740      +/-   ##
==========================================
- Coverage   55.54%   55.48%   -0.06%     
==========================================
  Files         197      197              
  Lines       53122    53188      +66     
  Branches    44443    44509      +66     
==========================================
+ Hits        29507    29512       +5     
- Misses      22144    22205      +61     
  Partials     1471     1471

Files with missing lines	Coverage Δ
service/src/lib.rs	`79.03% <ø> (ø)`
src/bin/nydusd/main.rs	`0.00% <0.00%> (ø)`
service/src/upgrade.rs	`65.31% <58.33%> (+0.86%)`	⬆️
service/src/fusedev.rs	`0.00% <0.00%> (ø)`

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Zephyrcf · 2025-08-20T16:01:51Z

service/src/fusedev.rs

+
+        let fd = file.as_raw_fd();
+        let mut buf = vec![0u8; session.bufsize()];
+        let writer: FuseDevWriter = FuseDevWriter::new(fd, &mut buf).unwrap();


Suggested change

let writer: FuseDevWriter = FuseDevWriter::new(fd, &mut buf).unwrap();

let writer: FuseDevWriter = FuseDevWriter::new(fd, &mut buf)?;

It is more appropriate to use the ? operator.

Thanks for this suggestion. I'll fix it.

se.with_writer(|writer| {
self.server
.notify_resend(writer)
.unwrap_or_else(|e| println!("failed to send resend notification {}", e));
});

I think we have expose a with_writer on session to make it easy to acquire writer from session

se.with_writer(|writer| { self.server .notify_resend(writer) .unwrap_or_else(|e| println!("failed to send resend notification {}", e)); });

I think we have expose a with_writer on session to make it easy to acquire writer from session

I noticed with_writer, but I need the error returned by notify_resend.

cloud-hypervisor/fuse-backend-rs#216
I make a PR to update the with_writer to try_with_writer to handle error properly, let me know if that satisfy your expectation

cloud-hypervisor/fuse-backend-rs#216 I make a PR to update the with_writer to try_with_writer to handle error properly, let me know if that satisfy your expectation

Yes, I need it. I'll update my code after fuse-backend-rs version of nydusd getting updated.

service/src/fusedev.rs

imeoer · 2025-08-21T03:29:46Z

src/bin/nydusd/main.rs

        Arg::new("failover-policy")
            .long("failover-policy")
-            .default_value("resend")
+            .default_value("none")


Maybe keep resend as default value.

OK, I'll fix it.

Any updates?

service/src/upgrade.rs

service/src/lib.rs

aftersnow · 2025-08-21T07:46:42Z

service/src/fusedev.rs

+            .ok_or(NydusError::FuseFileNotFound)?;
+
+        let fd = file.as_raw_fd();
+        let mut buf = vec![0u8; session.bufsize()];


Maybe it's better to put buf allocation out of the lock

OK, I'll fix it.

Maybe it's better to put buf allocation out of the lock

But we maybe have to lock session before we get the bufsize.

aftersnow · 2025-08-21T07:49:02Z

service/src/fusedev.rs

+    fn notify_resend_by_sysfs(&self) -> NydusResult<()> {
+        // Notify the kernel to resend the request by writing to `/sys/fs/fuse/connections/<cid>/resend`
+        let path = format!(
+            "/sys/fs/fuse/connections/{}/resend",


try to avoid fusectl path hardcode: /sys/fs/fuse/connections/{}/resend

OK, I'll fix it.

aftersnow · 2025-08-21T07:49:12Z

service/src/fusedev.rs

+    fn notify_flush_by_sysfs(&self) -> NydusResult<()> {
+        // Notify the kernel to flush the request by writing to `/sys/fs/fuse/connections/<cid>/flush`
+        let path = format!(
+            "/sys/fs/fuse/connections/{}/flush",


OK, I'll fix it.

imeoer · 2025-09-11T02:01:22Z

src/bin/nydusd/main.rs

        Arg::new("failover-policy")
            .long("failover-policy")
-            .default_value("resend")
+            .default_value("none")


Any updates?

imeoer · 2025-09-11T02:11:32Z

service/src/fusedev.rs

+            }
+        }
+
+        Err(open_error.unwrap())


How about:

for (idx, path) in paths.iter().enumerate() { match self.try_notify_with_path(base_path, event) { Ok(()) => return Ok(()), Err(e) => { if !matches!(e, NydusError::SysfsOpenError(_)) || idx == paths.len() - 1 { return Err(e); } } } } Ok(())

OK, this looks better.

Resend the fuse request when nydusd undergoes a hot upgrade or during a fault recovery process. Support resend requests by fuse device and sysfs. Signed-off-by: Sicheng Liu <lsc2001@outlook.com>

imeoer

Thanks, LGTM!

lsc2001 requested a review from a team as a code owner August 16, 2025 04:38

lsc2001 requested review from hsiangkao, jiangliu and power-more and removed request for a team August 16, 2025 04:38

Zephyrcf reviewed Aug 20, 2025

View reviewed changes

imeoer reviewed Aug 21, 2025

View reviewed changes

service/src/fusedev.rs Outdated Show resolved Hide resolved

imeoer reviewed Aug 21, 2025

View reviewed changes

service/src/upgrade.rs Outdated Show resolved Hide resolved

imeoer reviewed Aug 21, 2025

View reviewed changes

service/src/lib.rs Outdated Show resolved Hide resolved

aftersnow reviewed Aug 21, 2025

View reviewed changes

imeoer mentioned this pull request Sep 10, 2025

fix: https://github.com/dragonflyoss/nydus/issues/1697 #1754

Closed

10 tasks

lsc2001 force-pushed the fuse-resend branch from 01d85b5 to b85ccb1 Compare September 10, 2025 16:46

imeoer reviewed Sep 11, 2025

View reviewed changes

lsc2001 force-pushed the fuse-resend branch from b85ccb1 to f57e4ec Compare September 13, 2025 03:30

Support resend fuse requests when restoring nydusd.

76a3f7a

Resend the fuse request when nydusd undergoes a hot upgrade or during a fault recovery process. Support resend requests by fuse device and sysfs. Signed-off-by: Sicheng Liu <lsc2001@outlook.com>

lsc2001 force-pushed the fuse-resend branch from f57e4ec to 76a3f7a Compare September 13, 2025 04:11

imeoer approved these changes Sep 15, 2025

View reviewed changes

imeoer merged commit 637a28a into dragonflyoss:master Sep 15, 2025
24 of 25 checks passed

imeoer mentioned this pull request Oct 14, 2025

support fuse resend API for live upgrade and failover #1697

Closed

	let writer: FuseDevWriter = FuseDevWriter::new(fd, &mut buf).unwrap();
	let writer: FuseDevWriter = FuseDevWriter::new(fd, &mut buf)?;

[Feature] Support resending fuse requests when restoring nydusd. #1740

[Feature] Support resending fuse requests when restoring nydusd. #1740

Uh oh!

Conversation

lsc2001 commented Aug 16, 2025

Relevant Issue

Details

Types of changes

Uh oh!

codecov bot commented Aug 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

imeoer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Aug 16, 2025 •

edited

Loading