Update: My feedback article is now available.
I wanted to install openSUSE Tumbleweed (the rolling release version of openSUSE) on one of my computers to see how it looked outside a VM (and thus to try to use it daily). I am thinking about switching to this distribution to avoid major updates of Fedora. They happen every 6 months and I have multiple machines to update, so I find it cumbersome to update them all (and I like to test new stuff too ^^).
Why openSUSE? Because it looks stable, well tested, it provides a graphical installer (I already installed Arch and Gentoo in the past and wasn't enthusiastic about doing this again) and has some nice defaut with BTRFS (snapshots and booting from snapshots without having to configure it manually). I also know a contributor and a long time user of openSUSE and Tumbleweed who's very happy about the system.
In this article, I'll explain how I installed it next to my Fedora (which I don't want to loose it until I commit to making the switch). Some things are specific to my setup, others are more generic and should be reusable (more or less easily). If you have any question or remark, please leave a comment. I'll assume you have some knowledge of BTRFS.
I had fedora installed with a BTRFS top level volume and 3 child volumes root for /, var for /var and home for /home. Since I use docker with the BTRFS driver I also had many subvolumes of /var for docker.
I also installed Tumbleweed in a VM to see how it behaves. I created another VM with Fedora installed just like on my computer so I could try installing Tumbleweed next to it without breaking my current setup.
I removed the docker containers and images as well as all docker related subvolumes that remained. I don't know whether this could have an impact and didn't want to risk it (nor did I wish to migrate them).
The two most useful commands you can use at anytime to check something are:
- btrfs subvolume list PATH to list all subvolumes in the supplied filesystem.
- btrfs subvolume show PATH to see which subvolume is mounted under the supplied path.
Disable SELinux: it's not used in openSUSE so I think some files will be mislabeled which may prevent Fedora to work properly (or maybe just prevent me to access some files, I don't know and I don't want to test). Let's simply as much as possible and remove it from the equation.
Since my subvolumes are named var, root and home, the Tumbleweed installer will mount them under /var, /root and /home. I didn't find a way to prevent this. For /home it's the behavior I want since I want to keep my data. For the other two, it's not. You cannot rename subvolumes directly but you can rely on snapshots to achieve it. To do so, I (as root):
Mounted the top level BTRFS volume (which is not accessible by default) with: mount -t btrfs -o subvolid=5 /dev/vda3 /mnt/toplevel/ (most of my commands are copied from when I tested in a VM, hence the vda, adapt this part to your setup).
Created a snapshot for the root subvolume: btrfs subvolume snapshot /mnt/toplevel/root/ /mnt/toplevel/fedoraroot/
Emptied /mnt/toplevel/fedoraroot/var to remove the directory with rmdir /mnt/toplevel/fedoraroot/var
Created another snapshot for var (I decided to make it a subvolume of fedoraroot because I think it makes sense to have one top volume for Fedora): btrfs subvolume snapshot /mnt/toplevel/var/ /mnt/toplevel/fedoraroot/
Migrated the nested subvolumes for /var:
[root@baley toplevel]# rmdir /mnt/toplevel/fedoraroot/var/lib/machines/ [root@baley toplevel]# btrfs subvolume snapshot /mnt/toplevel/var/lib/machines/ /mnt/toplevel/fedoraroot/var/lib/ [root@baley toplevel]# rmdir /mnt/toplevel/fedoraroot/var/lib/portables/ [root@baley toplevel]# btrfs subvolume snapshot /mnt/toplevel/var/lib/portables/ /mnt/toplevel/fedoraroot/var/lib/
Identified each subvolume so I could know which ones where mounted as my main filesystem:
[root@localhost ~]# touch /mnt/toplevel/root/stdroot [root@localhost ~]# touch /mnt/toplevel/fedoraroot/fedoraroot [root@baley toplevel]# touch fedoraroot/var/fedoravar [root@baley toplevel]# touch ./var/stdvar
Updated fstab in the snapshot (that's in fedoraroot) since that what we want to use from now on. To do so, I replaced subvol=root by subvol=fedoraroot```and ``subvol=var by subvol=fedoraroot/var.
Edited the grub line from the menu before booting from it. I updated the subvol parameter of the kernel to be subvol=fedoraroot. This way grub mounted the proper subvolume during boot.
Checked that everything was correct:
[root@localhost ~]# btrfs subvolume show / fedoraroot Name: fedoraroot UUID: 1fc8ffa2-76f5-a343-aae9-4b499073d805 Parent UUID: 5effda61-179e-9b48-900f-b7ad85d5f9d9 Received UUID: - Creation time: 2020-02-22 16:14:19 +0100 Subvolume ID: 268 Generation: 4554 Gen at creation: 4447 Parent ID: 5 Top level ID: 5 Flags: - Snapshot(s): [root@localhost ~]# ls /fedoraroot /fedoraroot
Remounted the top level volume.
Deleted the nested subvolumes: btrfs subvolume delete /mnt/toplevel/root/var/lib/machines/ (same for portables).
Deleted root and var:
btrfs subvolume delete /mnt/toplevel/root/ btrfs subvolume delete /mnt/toplevel/var/
Updated grub: grub2-mkconfig -o /boot/grub2/grub.cfg # or /boot/efi/EFI/fedora/grub.cfg if you use EFI
Checked the generated grub configuration:
[root@localhost ~]# grep -R subvol /boot/grub2/grub.cfg # The subvolume parameter is correct. set default_kernelopts="root=UUID=ce254a08-22e0-481e-b027-498aa059cc1a ro rootflags=subvol=fedoraroot resume=UUID=163891ab-ac20-4e58-8731-306810f99c9c rhgb quiet "
Rebooted. All should still be working.
Just do the normal installation process until you reach the disk setup. Here you'll have to choose Expert partition and Start with existing partitions. Then, on the disk that contains you BTRFS volume:
- Mount the swap
- Mount the EFI partition under /boot/efi (if you use EFI).
- Mount the main BTRFS volume / and be sure to skip the formatting of the volume. /boot must be under / in the BTRFS volume even if you use luks. See below for why.
- Check installation summary: should not remove anything but should create a lot of new subvolume. None should conflict with fedora's.
That should be it. Inspect in the summary the operations done. You should see that the installer:
- Will create all the subvolumes it needs.
- Won't delete anything.
- Will mount the fedora subvolumes. Not a big deal, we will remove that from the fstab once the installation is done.
You can launch the installation.
Finishing the setup of Tumbleweed
If you use luks, you will have to enter you luks passphrase in Grub with a US keyboard. You will then have to enter you passphrase normally as part of the boot process.
By default Tumbleweed will create a subvolume named @ and create its other subvolumes as subvolumes of it. Here however, it didn't (I don't know why). Since I still want a main subvolume for each system (having home as a direct subvolume of the top level volume sounds good to me since it is independent of any other system). So, it's time to move subvolumes again. I won't detail much this time, I think you know the drill (do this from fedora, the directories are in use in Tumbleweed):
mount -t btrfs -o subvolid=5 /dev/vda3 /mnt/toplevel/ cd /mnt/toplevel/ btrfs subvolume list . # Repeat for each subvolume. rmdir /mnt/toplevel/@/var/ btrfs subvolume snapshot /mnt/toplevel/var /mnt/toplevel/@ # Check. Don't remove subvolume while in use, the deletion will come later. btrfs subvolume list /mnt/toplevel btrfs subvolume list .
Let's edit the fstab. We are just appending @/ to each subvol arguments like this: subvol=@/var.
We also need to update /boot/grub2/x86_64-efi/load.cfg so grub mounts the correct subvolume during the boot process:
set btrfs_relative_path='y' cryptomount -u bf63369674824969a7726fda0cb46537 btrfs-mount-subvol ($root) /boot/grub2/x86_64-efi @/boot/grub2/x86_64-efi
I didn't need this step in my VM and given the content of the file, I think it is specific to luks.
We can now reboot. Everything should go well. Boot on the other system if not.
We can now delete the old subvolumes from the installation:
mount -t btrfs -o subvolid=5 /dev/vda3 /mnt/toplevel/ cd /mnt/toplevel/ # Repeat for each subvolume. btrfs subvolume delete ./var
We can also enable snapper now to have snapshot. It's done automatically during a normal installation but not here (because we did weird things I guess). We can do this with snapper -c root create-config /. And we are done!
Fedora uses subvol in grub.cfg but not Tumbleweed. Tumbleweed changes the default subvolume of the top level volume. It's not a problem with this setup but if you want to install two systems that do this, you will have to make at least one explicitly use subvol everywhere.
It required a lot of preparation and tests but I'm glad I made it. I find Tumbleweed fast (faster than fedora at boot and when installing package) and reliable (from what I used until now). I'll need to do more testing to confirm that (the system is very fresh after all). I also learned a lot about BTRFS, subvolumes and snapshots along the way. I still have some issues though:
- The nextcloud client ask me to reconnect each time I launch it. It may come from a configuration shared with Fedora. I'll fix this later once I know which system I want to keep.
- The grub menu to choose the kernel (or the snapshot) is sometimes not displayed.
- Grub takes some time to start once the key is entered.
- If I fail to enter my passphrase for grub of the first try, it fails and I have to do a hard reboot.
- I decided to switch from Gnome to KDE and I think I hit a bug with the online accounts that prevent me to setup my gmail account. I'll have to dig into this later (I'll try to update this post if I find the solution). I also spent a lot of time to configure KMail (which has a lot of options by the way).
Enter luks passphrase only once
If you use luks, you have to enter your luks passphrase twice: once to unlock /boot so grub can start the system and then so the system can access the disk. Sadly, they can't communicate (yet?).
You could put /boot on an unencrypted filesystem (that's what fedora does), but according to this thread you won't be able to boot from a BTRFS snapshot (which is one cool thing Tumbleweed enables by default).
You can also fellow these steps so you'll only have to enter the passphrase for grub. Check also this link to update the initramfs in Tumbleweed (don't forget to launch dracut --force). For reference, my /etc/dracut.conf contains:
# PUT YOUR CONFIG IN separate files # in /etc/dracut.conf.d named "<name>.conf" # SEE man dracut.conf(5) for options install_items+="/etc/luks-keys/root.key"
If you need to add many files, separate them with spaces like this: install_items+="/etc/luks-keys/root.key /etc/luks-keys/swap.key"
Don't add key-slot=1 immediately, if you did a mistake you will be locked out of the system because luks will try to match the key against this slot. Don't forget to adapt the slot if needed. You may also choose not to add it since I found it doesn't really change the boot time anyway.
Tumbleweed default install
localhost:~ # btrfs subvolume list / ID 256 gen 32 top level 5 path @ ID 258 gen 4192 top level 256 path @/var ID 259 gen 4125 top level 256 path @/usr/local ID 260 gen 4192 top level 256 path @/tmp ID 261 gen 3045 top level 256 path @/srv ID 262 gen 4166 top level 256 path @/root ID 263 gen 3620 top level 256 path @/opt ID 264 gen 4192 top level 256 path @/home ID 265 gen 26 top level 256 path @/boot/grub2/x86_64-efi ID 266 gen 3550 top level 256 path @/boot/grub2/i386-pc ID 267 gen 4118 top level 256 path @/.snapshots ID 268 gen 4191 top level 267 path @/.snapshots/1/snapshot ID 274 gen 197 top level 267 path @/.snapshots/2/snapshot ID 275 gen 262 top level 267 path @/.snapshots/3/snapshot ID 278 gen 485 top level 267 path @/.snapshots/4/snapshot ID 282 gen 1614 top level 267 path @/.snapshots/5/snapshot ID 283 gen 2100 top level 267 path @/.snapshots/6/snapshot ID 290 gen 3024 top level 267 path @/.snapshots/11/snapshot ID 291 gen 3066 top level 267 path @/.snapshots/12/snapshot ID 292 gen 3233 top level 267 path @/.snapshots/13/snapshot ID 295 gen 3551 top level 267 path @/.snapshots/14/snapshot ID 296 gen 3593 top level 267 path @/.snapshots/15/snapshot ID 297 gen 3613 top level 267 path @/.snapshots/16/snapshot ID 301 gen 3631 top level 267 path @/.snapshots/17/snapshot ID 305 gen 4005 top level 267 path @/.snapshots/18/snapshot ID 306 gen 4051 top level 267 path @/.snapshots/19/snapshot ID 307 gen 4113 top level 267 path @/.snapshots/20/snapshot localhost:/mnt/toplevel # cat /etc/fstab UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 / btrfs defaults 0 0 UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 /var btrfs subvol=/@/var 0 0 UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 /usr/local btrfs subvol=/@/usr/local 0 0 UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 /tmp btrfs subvol=/@/tmp 0 0 UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 /srv btrfs subvol=/@/srv 0 0 UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 /root btrfs subvol=/@/root 0 0 UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 /opt btrfs subvol=/@/opt 0 0 UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 /home btrfs subvol=/@/home 0 0 UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 /boot/grub2/x86_64-efi btrfs subvol=/@/boot/grub2/x86_64-efi 0 0 UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 /boot/grub2/i386-pc btrfs subvol=/@/boot/grub2/i386-pc 0 0 UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1 /.snapshots btrfs subvol=/@/.snapshots 0 0 UUID=6f2fce23-6460-4d69-9e69-a5b6b5e4456c swap swap defaults 0 0