Upstream status

[Fantu]

@SergeiShtepa Hi, thanks for your works on blksnap kernel module, I think will be very useful for improve backup on linux.
I'm curious about the status of its possible upstream integration, from 85bad8f seems you prepared a patches serie for upstream (for send a mail with it) but I not found it posted searching on lkml.org and google. I saw also other series of months ago on other branches but I not found them on upstream mailing list and found only older your works of 1 year ago or more.
Thanks for any reply and sorry for my bad english.

[SergeiShtepa]

Thank you for creating the issue. So it's important to someone.

Yes, I haven't tried to offer blksnap in the upstream yet.
The problem is that I can't find anyone who could do a code review. And I don 't have enough experience with upstream.
In addition, the module is not a regular driver for some device. I suggest adding a filter to the block layer. Mantainers may not like it.
It would be great if I could attract other developers of backup tools to work on the module. But I think that while the module is not in the upstream, no one will want to invest their time in this project.
Therefore, I doubt the success of this venture.

But I intend to try to do it soon. It remains to double-check something.
I hope that someone will give good advice.
I will add a link to the patch discussion in upstream to this issue.

Any feedback is welcome.

[Fantu]

Thanks for reply, I did/do small contributions on some open source projects but I never did on linux kernel, if I remember good I gave only an help in testing one xen part preparing to upstream many years ago.
To know if can be acceptable your work and receive review of expert kernel developers (who can give you their opinions on it and their advices to improve where needed) post the patches is really important, from a fast look I found this doc that explain how to do it: https://www.kernel.org/doc/html/latest/process/submitting-patches.html (probably you already saw it)
and from a fast look to your latest patches serie ( 85bad8f ) seems you already did a good work on major requirements.
I found this project in a search done after I found one blk-snap patches serie you posted 2 year ago so I think posting patches is also a way let know it to any other interested developers and users (as well as receiving upstream developers review)

[SergeiShtepa]

seems you already did a good work on major requirements.

Thanks. Indeed, it is the good quality of the code that is an important component of success.

a way let know it to any other interested developers and users

Of course, that's why I'll be offering a patch.

[andymend]

I agree that your code represents a potentially important improvement for backup of linux systems, especially as new file systems proliferate. Blksnap is a distinct improvement over earlier such attempts like dattobd and elastio-snap. I have been lightly testing blksnap on a computer I routinely use where there are multiple file systems being used. Yours is a worthwhile project that will hopefully get merged into the mainstream kernel at some point.

[SergeiShtepa]

Thank you @andymend .
Thank you all for your support.
I just sent a patch.
If it passes the robot check, we will see it in the list soon.

[SergeiShtepa]

patchwork.kernel.org
lore.kernel.org

[Fantu]

for what I see here 2 kernel developers reviewed and replied and there is also 2 replied from an automatic test bot (reporting some warnings), for now, no one has objected or criticized the implementation method but only recommended improvements for a v2.

from a quick search I did not find other proposals for implementing "temporary" snapshots for better backup integrity and that supports many filesystems

unfortunately I have not yet had time to try blksnap but from what I have seen from experience this functionality integrated into the kernel is very useful and as I have already seen several backup solutions have made their own modules for similar things but no one who has proposed its implementation in the kernel

I have used several backup solutions over the years, most with file level backup using rsync, however even in such cases I think it would be useful because either the services stop during the whole backup or you risk not having it integrated and the larger the part to back up and the more time it takes, the greater the risk of integrity being compromised.
For example, if you have to back up a server of hundreds of gb or over 1 tb (in most cases via the network the times increase), it takes hours and during the operations to numerous files in various interconnected parts and rsync buckups of the parts before such them, others during and others after would be problematic for its integrity on restore while with a snapshot and rsync backup on it you would have a backup with minimal integrity risk (reduced to the fraction of snapshot creation time FWIK) and restore you would know that you have a fairly healthy situation at the time of the snapshot and not a high risk of integrity problems for the time it took to make the backup. And without such a snapshot, the risks can only be mitigated by stopping the services (as mentioned) or doing it outside working hours.

Another thing I tried at works in latest years is to use btrfs with integrated snapshot for backups (I also tried zfs years ago) but I use only in some cases because for example on virtualization systems, the COW it has a significant impact on performance and there are also some space issues keeping snapshots on the same production filesystem. this could be a solution to make a backup with minimum downtime stopping services to take the snapshot (with blksnap) and maximum integrity probability, or at least making the snapshot to active services like what a COW filesystem like btrfs and zfs can do but without having to use such filesystems.

I hope these examples based on my experience give an idea of how useful a system like blksnap is
When there is a calm period where I am not too busy I will start testing it
in the meantime thanks again for your work SergeiShtepa

sorry for my bad english

[SergeiShtepa]

Thanks @Fantu.

I expected a very calm reaction to the patch.
I plan to:

• correct a flaws, indicated by the reviewers
• move some of the logic into the kernel module, simplifying the main kernel code even more
• fix bugs (QA found something).

So, the next patch will be better.

Alas, the performance of the blksnap module should not be better than DM snapshots (LVM2).
When handling a write request, it is required to read a chunk of data, and then write this chunk of data to a new location. There are no magic here. When writing, the load on the drive increases threefold.

Theoretically, snapshots on Btrfs should be better. But for BTRFS, we cannot read the entire block device. We have to synchronize the source and target file systems. The Veeam Agent for Linux implements btrfs backup support by its native means. The performance was not the best.

The presence of a change tracker and the ease of allocating space for the difference storage are the main advantages of blksnap, compared to "classic" snapshots.

[Fantu]

thanks for reply and your works
about btrfs I know that is better where there is no "limit" or performance issue (for example I use on some backup storage and fileservers), but for example in "low cost" virtualization servers there was significant performance issue, disabling COW partially solves the issue because using snapshot will require it and ignore the previous options to disable cow, so still using ext4 and do temp. snapshot only on backup time, avoid the downtime or having it shortly, having high integrity probability and lower performance only when backup with snapshot is running still seems to me a good idea.
however I can only be helpful enough when I start using it

[SergeiShtepa]

Hello everyone.
I am very glad that the first patch was noticed and I received quite useful feedback.
A lot will have to be rewritten. And that's great.

[Fantu]

hi, is there any news about v2 patch serie for upstream?
I suppose anyway it would not be reviewed in time for the kernel 6.0 :(
edit: merge window for 6.0 is already closed, I hope for the 6.1

[SergeiShtepa]

Hi. That's what I really want to do.
But I had to switch to more urgent tasks. I hope to be able to reduce my task list soon.

[SergeiShtepa]

Hi!

This looks like a workable version of the patch for linux kernel 6.1-rc1 .
It contains corrections in accordance with the comments that were made by the maintainers to the first version of the patch.
I plan to test this for two or three days.
I invite everyone to join the testing.

---

Feedback is welcome.