BTRFS snapshots and grub-btrfs/snapper behaviour

[onelittlehope]

Hi

Awesome job at creating Spiral Linux. In the past I had looked inside the tar archives of Gecko Linux and borrowed configs I wanted to import into my openSUSE TW install and I'm cheekily doing this again with Spiral Linux now that I have moved from openSUSE to Debian Testing. (your work is very much appreciated!)

Specifically, I was interested in how you managed to get snapper / grub-btrfs working in Debian. I don't think anyone appreciates how convuluted it all is. You should have blogged about your trials / tribulations when creating Spiral Linux as it would have been interesting reading.

I understand that by installing snapper, you are only making use of its functionality to create/delete/list/diff/manage snapshots. i.e. you are not using its rollback functionality. That job is done by snapper-rollback.

Full system rollback via grub is an important feature to me since with Debian Testing, I can expect that at times, I might break the system with an apt upgrade.

So I've been testing how snapper/grub-btrfs and snapper-rollback work in a Spiral Linux KDE Plasma VM in order to borrow the config and apply to my Debian Testing install.

In Spiral Linux, when creating snapshots (by installing pkgs via apt-get) and performing a snapper-rollback, I noticed that there was no way to find out which snapshot id is active after the reboot.

Running snapper ls does not show a star (*) next to the snapshot id in the table. e.g. output from openSUSE:

 # | Type   | Pre # | Date                     | User | Used Space | Cleanup | Description           | Userdata     
---+--------+-------+--------------------------+------+------------+---------+-----------------------+--------------
0  | single |       |                          | root |            |         | current               |              
1* | single |       | Fri Jul  8 18:28:28 2022 | root |  79.38 MiB |         | first root filesystem |              
2  | single |       | Fri Jul  8 18:44:38 2022 | root |  30.70 MiB | number  | after installation    | important=yes
3  | pre    |       | Sun Jul 10 21:38:08 2022 | root |   1.02 MiB | number  | zypp(zypper)          | important=no 
4  | post   |     3 | Sun Jul 10 21:38:12 2022 | root |   3.06 MiB | number  |                       | important=no 

In Spiral Linux, what I see is:

 # | Type   | Pre # | Date                         | User | Cleanup | Description | Userdata
---+--------+-------+------------------------------+------+---------+-------------+---------
0  | single |       |                              | root |         | current     |         
1  | pre    |       | Wed 20 Jul 2022 18:54:49 BST | root | number  | apt         |         
2  | post   |     1 | Wed 20 Jul 2022 18:55:54 BST | root | number  | apt         |         
3  | pre    |       | Thu 21 Jul 2022 18:34:09 BST | root | number  | apt         |         
4  | post   |     3 | Thu 21 Jul 2022 18:34:12 BST | root | number  | apt         |         
5  | pre    |       | Thu 21 Jul 2022 18:34:20 BST | root | number  | apt         |         
6  | post   |     5 | Thu 21 Jul 2022 18:34:21 BST | root | number  | apt         |         

Query 1: Is it possible to figure which snapshot is the default or is active ? The output of btrfs subvolume list -ta / or cat /proc/mounts or btrfs subvolume get-default / don't seem to help answer this (unless I'm overlooking something). Update: I came across this PR which indicates that snapper can be compiled with a no-rollback compile option and was wondering if that is what needs to happen in this case when we aren't planning to use snapper's rollback functionality. You can find what compile options are set via snapper debug.

I saw the Github issues you raised in snapper-rollback and grub-btrfs. I was likewise testing what snapshots are visible once a system rollback is done and it seems I can only see the snapshots created before the snapshot id I rolled back too in grub. For example, in the above snapper ls output, I did a snapper-rollback 5 and the only snapshots I could see in grub were: 4, 3, 2 and 1.

Query 2: Should I be able to see snapshots 5 and 6 in grub ? I ask since they are visible and present in snapper ls output. In openSUSE for example, I am able to see all the snapshots except the one that is the active one.

regards,
JC

[geckolinux]

Hi there, thanks a lot for writing. I'm really glad to hear you find the projects useful.

---

Regarding the first question, when booted into a read-only snapshot it does show which one is booted with the - after the number, in this case I booted into snapshot 3:

sudo snapper list
 # | Type   | Pre # | Date                            | User | Cleanup | Description | Userdata
---+--------+-------+---------------------------------+------+---------+-------------+---------
0  | single |       |                                 | root |         | current     |         
1  | pre    |       | Fri 22 Jul 2022 09:31:15 AM EDT | root | number  | apt         |         
2  | post   |     1 | Fri 22 Jul 2022 09:31:17 AM EDT | root | number  | apt         |         
3- | pre    |       | Fri 22 Jul 2022 09:31:36 AM EDT | root | number  | apt         |         
4  | post   |     3 | Fri 22 Jul 2022 09:31:38 AM EDT | root | number  | apt         |         
5  | pre    |       | Fri 22 Jul 2022 09:31:49 AM EDT | root | number  | apt         |         
6  | post   |     5 | Fri 22 Jul 2022 09:31:50 AM EDT | root | number  | apt         |         

After performing sudo snapper-rollback 3 and rebooting into the RW system the subvolumes look like this:

sudo btrfs sub list /
ID 257 gen 90 top level 5 path @2022-07-22T09:39
ID 258 gen 89 top level 5 path @snapshots
ID 259 gen 105 top level 5 path @home
ID 260 gen 105 top level 5 path @root
ID 261 gen 105 top level 5 path @var@log
ID 262 gen 105 top level 5 path @var@lib@AccountsService
ID 263 gen 105 top level 5 path @var@lib@blueman
ID 264 gen 105 top level 5 path @tmp
ID 275 gen 80 top level 258 path @snapshots/1/snapshot
ID 276 gen 82 top level 258 path @snapshots/2/snapshot
ID 277 gen 99 top level 258 path @snapshots/3/snapshot
ID 278 gen 85 top level 258 path @snapshots/4/snapshot
ID 279 gen 87 top level 258 path @snapshots/5/snapshot
ID 280 gen 88 top level 258 path @snapshots/6/snapshot
ID 283 gen 105 top level 5 path @

sudo btrfs subvolume get-default /   
ID 283 gen 118 top level 5 path @

sudo cat /boot/grub/grub.cfg ##Filtered just the relevant line
linux	/@/boot/vmlinuz-5.18.0-0.bpo.1-amd64 root=UUID=2dc99819-da0d-47c6-bf77-c92a5deb6f7a ro rootflags=subvol=@  quiet loglevel=0 splash

This correctly mounts the restored system because the snapper-rollback script simply renames the old (let's imagine it's broken) @ subvolume with the date and time stamp and then copies the requested rollback snapshot and renames it to @. The /etc/fstab also mounts / with the option subvol=/@

cat /etc/fstab ##Filtered for brevity
UUID=2dc99819-da0d-47c6-bf77-c92a5deb6f7a /              btrfs   subvol=/@,defaults,noatime,space_cache=v2,compress=zstd 0 0

Compare this with the initial SpiralLinux installation before snapshots or rollbacks:

sudo btrfs sub list /
ID 257 gen 75 top level 5 path @
ID 258 gen 55 top level 5 path @snapshots
ID 259 gen 76 top level 5 path @home
ID 260 gen 74 top level 5 path @root
ID 261 gen 77 top level 5 path @var@log
ID 262 gen 68 top level 5 path @var@lib@AccountsService
ID 263 gen 68 top level 5 path @var@lib@blueman
ID 264 gen 74 top level 5 path @tmp

sudo btrfs subvolume get-default /
ID 5 (FS_TREE)

sudo cat /boot/grub/grub.cfg ##Filtered just the relevant line
linux	/@/boot/vmlinuz-5.18.0-0.bpo.1-amd64 root=UUID=2dc99819-da0d-47c6-bf77-c92a5deb6f7a ro rootflags=subvol=@  quiet loglevel=0 splash

This is all quite different from what GeckoLinux does (following the same basic principles of vanilla openSUSE):

cat /etc/fstab  ##Filtered for brevity
UUID=4887f89a-a1b5-47e4-8fe7-8f1e9c46fb19 /              btrfs   defaults,noatime,space_cache=v2,autodefrag,compress=zstd 0 0

sudo btrfs sub list /
ID 257 gen 112 top level 5 path home
ID 258 gen 113 top level 5 path root
ID 259 gen 115 top level 5 path var
ID 260 gen 14 top level 5 path boot/grub2/x86_64-efi
ID 261 gen 25 top level 5 path boot/grub2/i386-pc
ID 262 gen 104 top level 5 path .snapshots
ID 263 gen 105 top level 257 path home/.snapshots

sudo cat /boot/grub2/grub.cfg ##Filtered just the relevant line
linux	/boot/vmlinuz-5.15.6-1-default root=UUID=4887f89a-a1b5-47e4-8fe7-8f1e9c46fb19  quiet splash

And then after booting RO snapshot 4 in GeckoLinux, then snapper rollback and then rebooting into the RW system:

cat /etc/fstab  ##Filtered for brevity
UUID=4887f89a-a1b5-47e4-8fe7-8f1e9c46fb19 /              btrfs   defaults,noatime,space_cache=v2,autodefrag,compress=zstd 0 0

sudo btrfs sub list /
[sudo] password for root: 
ID 257 gen 146 top level 5 path home
ID 258 gen 113 top level 5 path root
ID 259 gen 150 top level 5 path var
ID 260 gen 14 top level 5 path boot/grub2/x86_64-efi
ID 261 gen 25 top level 5 path boot/grub2/i386-pc
ID 262 gen 135 top level 5 path .snapshots
ID 263 gen 147 top level 257 path home/.snapshots
ID 264 gen 134 top level 262 path .snapshots/1/snapshot
ID 265 gen 125 top level 262 path .snapshots/2/snapshot
ID 266 gen 133 top level 262 path .snapshots/3/snapshot
ID 267 gen 134 top level 262 path .snapshots/4/snapshot

sudo cat /boot/grub2/grub.cfg ##Filtered just the relevant line
linux	/.snapshots/4/snapshot/boot/vmlinuz-5.15.6-1-default root=UUID=4887f89a-a1b5-47e4-8fe7-8f1e9c46fb19 rootflags=subvol=.snapshots/4/snapshot  quiet splash

So as you can see, the openSUSE GRUB configuration makes it boot the snapshot that was selected for rollback directly from the /.snapshots/ directory. On the other hand, the SpiralLinux configuration never changes its GRUB parameters for the root of the kernel and initrd, it always looks for them in /@/boot/ with rootflags=subvol=@ . Both configurations work in practice, but the SpiralLinux implementation via snapper-rollback feels more technically correct and clean. Here's the reasoning behind why this method is preferable when compared to Snapper's native rollback strategy:

• https://wiki.archlinux.org/title/snapper#Suggested_filesystem_layout
• https://wiki.archlinux.org/title/snapper#Restoring_/_to_its_previous_snapshot
• https://bbs.archlinux.org/viewtopic.php?id=194491

So basically, SpiralLinux uses Snapper because it's the best solution for automated snapshots with automatic cleanup and integration with apt, but Snapper's rollback functionality is more convoluted and only works with the openSUSE GRUB patches.

I came across this openSUSE/snapper#443 which indicates that snapper can be compiled with a no-rollback compile option and was wondering if that is what needs to happen in this case when we aren't planning to use snapper's rollback functionality

That definitely won't be possible for SpiralLinux, because a major goal of the project is to not fork or recompile any Debian packages, so we're stuck with whatever Snapper package Debian chooses to ship. But I'm not concerned about that, because first of all Snapper's native snapper rollback throws a confusing error under SpiralLinux about a missing --ambit which most users won't know how to get past. And if they do (via sudo snapper --ambit classic rollback) it won't change anything after they reboot to the RW system because the SpiralLinux GRUB scripts and the /etc/fstab will still make it boot the correct @ subvolume.

---

Regarding the second question, you're seeing the expected behavior, because immediately after the rollback it also reverts to the previous contents of /boot/grub/ at the time of that snapshot, so the GRUB menus are out of date. Once booted into the restored system you can simply run the GRUB Customizer GUI or sudo update-grub or make package changes via apt or Synaptic and it will regenerate the GRUB menus.

The issue that you found that I reported was different; the problem there was that even after rebooting into the restored system and trying to update the GRUB menus it would find the new snapshots and add them to the correct files in /boot/grub/ but the actual GRUB screen wasn't using those updated menu files because it was still retrieving them from the toplevel snapshot it was originally installed onto. The result was that any new snapshots created after the rollback would never appear in the grub-btrfs menu. So it was necessary to run sudo grub-install /dev/sdX to make GRUB look for the menus inside the new current default snapshot, which wasn't ideal for ease of use. This is a very obscure and unknown quirk of GRUB that I learned about in this post.

Hope this helps!

[onelittlehope]

Both configurations work in practice, but the SpiralLinux implementation via snapper-rollback feels more technically correct and clean.

I agree. I find understanding the rollback process easier with the SpiralLinux config.

So basically, SpiralLinux uses Snapper because it's the best solution for automated snapshots with automatic cleanup and integration with apt

The integration of a snapshot creation tool with apt is pretty straightforward. Snapper does it via the /etc/apt/apt.conf.d/80snapper config file (in SpiralLinux) which has:

# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=770938

  DPkg::Pre-Invoke  { "if [ -e /etc/default/snapper ]; then . /etc/default/snapper; fi; if [ -x /usr/bin/snapper ] && [ ! x$DISABLE_APT_SNAPSHOT = 'xyes' ] && [ -e /etc/snapper/configs/root ]; then rm -f /var/tmp/snapper-apt || true ; snapper create -d apt -c number -t pre -p > /var/tmp/snapper-apt || true ; snapper cleanup number || true ; fi"; };

  DPkg::Post-Invoke { "if [ -e /etc/default/snapper ]; then . /etc/default/snapper; fi; if [ -x /usr/bin/snapper ] && [ ! x$DISABLE_APT_SNAPSHOT = 'xyes' ] && [ -e /var/tmp/snapper-apt ]; then snapper create -d apt -c number -t post --pre-number=`cat /var/tmp/snapper-apt` || true ; snapper cleanup number || true ; fi"; };

So in a similar file like the above, I can replace snapper with something else which will create a snapshot.

For creation + management of snapshots, I'm investigating whether either of the following two tools will suffice as a replacement for snapper:

• digint/btrbk
• NicholasBHubbard/yabsm

For handling rollbacks, I've not yet come across anything other than snapper-rollback. I hope one of the above two tools will support rollbacks in the future but for now am investigating amending snapper-rollback script to make it work without needing snapshots in snapper format.

Lastly, I think grub-btrfs already supports non timesync or snapper format snapshots but I need to install / configure and check this.

I'm against using snapper due to its openSUSE origin and the possibility that they will change it in some way in the future that breaks its usage in other distributions. I moved away from that distribution due to the stance of a small number of vocal and toxic male Karen contributors who view non contributors as freeloaders. I've come across several helpful knowledgable people in openSUSE but a few bad apples does spoil the whole considering they are leading the direction that ALP is going in. baah sorry for the rant sneaking in but there wasn't an easy way to explain why I don't want to use snapper.

Regarding the second question, you're seeing the expected behavior

Ah ok. Thanks for confirming.

If it allows me, I'll mark this thread as resolved. If not, please feel free to do so.

[geckolinux]

For handling rollbacks, I've not yet come across anything other than snapper-rollback.

Another option is actually Timeshift. It even has the advantage of being completely GUI based even for the rollback process. I confirmed that it works well with grub-btrfs. The reason I didn't choose it was because its options for managing separate frequency and retention profiles of multiple snapshot configurations (such as / and /home) aren't as robust as Snapper. And the APT integration hook had to be manually installed (no Debian package for it), and the hook cluttered up the apt output in the terminal with lots of noise. So I didn't choose it for SpiralLinux, but then again my goals and considerations for what gets included in the project are somewhat different. For your personal usage scenario Timeshift might actually be the best option, and another advantage is that it's in the Debian repos.

I moved away from that distribution due to the stance of a small number of vocal and toxic male Karen contributors who view non contributors as freeloaders. I've come across several helpful knowledgable people in openSUSE but a few bad apples does spoil the whole considering they are leading the direction that ALP is going in.

I totally agree, that was extremely offensive and it does indeed ruin the entire experience.

[onelittlehope]

Another option is actually Timeshift.

I had discounted it since I thought it was a GUI only solution but now after looking into it futher, it looks like it does have a cli interface based on what's shown in its man page: linuxmint/timeshift/blob/master/man/timeshift.1.gz

I think I was worried I wouldn't be able to use Timeshift to recover from a rescue cd or when Xorg doesn't load on a broken system.

Thanks for mentioning it and for the link to APT integration hook. I'll test using Timeshift today.

[geckolinux]

Cool, please do let us know about your experience with it.

[onelittlehope]

I played with Timeshift the whole day in a VM and I do like it.

It has some nice features:

• I don't need to setup a snapshot subvolume. It will set one up by itself. My subvolume layout was very simple: @, @VarLog. I keep my /home on an xfs filesystem mainly due to not wanting to worry about cow downsides for user data and because I already have a seperate backup process for it based on restic.
• It dynamically mounts and unmounts the snapshot subvolume so that tools like du when run as du -hs / report accurate disk usage (i.e. it doesn't count the contents of the snapshots) Its also a minor security improvement in that old snapshots are not accessible by the non root users when the subvolume is not mounted.
• It doesn't change the btrfs default subvolume. (i.e. btrfs subvolume get-default /) like snapper does.

Some downsides:

• It doesn't have an ability to show a diff between snapshots like snapper. I found workarounds 1, 2 but the output is not as simple to understand as the snapper equivalent.
• There seems to be a restriction of only one timeshift process running at the same time. So if the gui is in use, no cron/apt initiated snapshots will occur. This was surprising.
• Can't delete a range of snapshots. Its either one by one or delete all
• I wish the debian maintainer had split the timeshift package into two. A gui one and a cli one. The GUI is not useful.

As long as I understand and accept that Timeshift is only to be used for recovery in situations of a healthy disk with a healthy filesystem and is not a backup solution it serves its purpose well. I'll augment it with something else to create and export a backup to another machine down the line to protect against a filesystem and hardware failures.

[alexrelis]

I currently have a system running Debian with encrypted btrfs and @ and @home subvolumes, but I have always wanted to replicate OpenSUSE's easy btrfs rollback feature. Later on I find out about the awesome Spiral Linux project which basically has exactly what I'm looking for. At the same time, I don't want to go through the effort of reinstalling and I don't find much use for any of Spiral Linux's other additions.

So, does anyone know how I can get this functionality on a regular Debian system? I've tried to think about how to implement but there are a lot of moving parts and I am not sure where to start. Any advice?

[CicadaSeventeen]

SpiralLinux is not quite away from Debian. Maybe the major thing is grub and vbox.
Installing vanilla Debian with btrfs subvols as you say here is possible but annoying. I do not say difficult here because it is not such hard but just... very ... make me upset.
And I think make that change without a re-installation is not so safe.

[onelittlehope]

I currently have a system running Debian with encrypted btrfs and @ and @home subvolumes, but I have always wanted to replicate OpenSUSE's easy btrfs rollback feature.

Your current subvolume layout is perfect for using TimeShift. The rollback process is very easy and you can do things via the CLI and not need the GUI at all. I'd say stick with your current simple subvolume layout and use TimeShift. The openSUSE subvolume layout is convoluted and relies on manipulating the default subvolume (via btrfs subvolume set-default) and that causes a whole bunch of issues with grub which they have added patches for in Fedora / openSUSE. I investigated how it all worked in order to reproduce it in Debian Testing and I really dislked how it all was put together.

[alexrelis]

Thanks for reaching out guys. So I'm trying Timeshift. It's neat but the last thing that I need is the ability to create btrfs snapshots before modifying packages with apt. I don't really need my system subvolume backed up intervallically since it only really changes when I upgrade or install packages. Ultimately I don't really need Timeshift at all, although it is a nifty GUI.

Update: I just used Timeshift and it runs my file manager as root. And when I close it, now Plasma apps are running in root. Uninstalling.

[geckolinux]

Hi there @alexrelis , it's definitely possible to configure a fresh vanilla Debian installation for Btrfs with bootable GRUB snapshots and clean rollbacks using and adapting some things from these references:

• https://github.com/Antynea/grub-btrfs
• https://github.com/jrabinow/snapper-rollback
• https://www.kali.org/docs/installation/btrfs/
• https://wiki.archlinux.org/title/snapper#Suggested_filesystem_layout
• https://github.com/Deebble/arch-btrfs-install-guide#snapper

As for converting an existing installation, yes, it's possible, but I honestly couldn't explain to you how to do it. ;-)

[markie83]

I went to post asking why rolling back with snapper was not working like openSUSE and found this page via search. This really needs pinned or a brief explanation on a welcome window for new users since it does not align with all the common snapper syntax we find online.

[geckolinux]

Hmm, sorry it's not very obvious, and there is no welcome window in SpiralLinux. It is however in the wiki:
https://github.com/SpiralLinux/SpiralLinux-project/wiki#working-with-btrfs-snapshots-and-rollbacks