Installing openSUSE next to Fedora with BTRFS

Posted on 2020-02-23 in Blog

Update: My feedback article is now available.

I wanted to install openSUSE Tumbleweed (the rolling release version of openSUSE) on one of my computers to see how it looked outside a VM (and thus to try to use it daily). I am thinking about switching to this distribution to avoid major updates of Fedora. They happen every 6 months and I have multiple machines to update, so I find it cumbersome to update them all (and I like to test new stuff too ^^).

Why openSUSE? Because it looks stable, well tested, it provides a graphical installer (I already installed Arch and Gentoo in the past and wasn't enthusiastic about doing this again) and has some nice defaut with BTRFS (snapshots and booting from snapshots without having to configure it manually). I also know a contributor and a long time user of openSUSE and Tumbleweed who's very happy about the system.

In this article, I'll explain how I installed it next to my Fedora (which I don't want to loose it until I commit to making the switch). Some things are specific to my setup, others are more generic and should be reusable (more or less easily). If you have any question or remark, please leave a comment. I'll assume you have some knowledge of BTRFS.

Starting point

I had fedora installed with a BTRFS top level volume and 3 child volumes root for /, var for /var and home for /home. Since I use docker with the BTRFS driver I also had many subvolumes of /var for docker.

I also installed Tumbleweed in a VM to see how it behaves. I created another VM with Fedora installed just like on my computer so I could try installing Tumbleweed next to it without breaking my current setup.

Preparing Fedora

I removed the docker containers and images as well as all docker related subvolumes that remained. I don't know whether this could have an impact and didn't want to risk it (nor did I wish to migrate them).
The two most useful commands you can use at anytime to check something are:
- btrfs subvolume list PATH to list all subvolumes in the supplied filesystem.
- btrfs subvolume show PATH to see which subvolume is mounted under the supplied path.
Disable SELinux: it's not used in openSUSE so I think some files will be mislabeled which may prevent Fedora to work properly (or maybe just prevent me to access some files, I don't know and I don't want to test). Let's simply as much as possible and remove it from the equation.
Since my subvolumes are named var, root and home, the Tumbleweed installer will mount them under /var, /root and /home. I didn't find a way to prevent this. For /home it's the behavior I want since I want to keep my data. For the other two, it's not. You cannot rename subvolumes directly but you can rely on snapshots to achieve it. To do so, I (as root):
1. Mounted the top level BTRFS volume (which is not accessible by default) with: mount -t btrfs -o subvolid=5 /dev/vda3 /mnt/toplevel/ (most of my commands are copied from when I tested in a VM, hence the vda, adapt this part to your setup).
2. Created a snapshot for the root subvolume: btrfs subvolume snapshot /mnt/toplevel/root/ /mnt/toplevel/fedoraroot/
3. Emptied /mnt/toplevel/fedoraroot/var to remove the directory with rmdir /mnt/toplevel/fedoraroot/var
4. Created another snapshot for var (I decided to make it a subvolume of fedoraroot because I think it makes sense to have one top volume for Fedora): btrfs subvolume snapshot /mnt/toplevel/var/ /mnt/toplevel/fedoraroot/
5. Migrated the nested subvolumes for /var:
```
[root@baley toplevel]# rmdir /mnt/toplevel/fedoraroot/var/lib/machines/
[root@baley toplevel]# btrfs subvolume snapshot /mnt/toplevel/var/lib/machines/ /mnt/toplevel/fedoraroot/var/lib/
[root@baley toplevel]# rmdir /mnt/toplevel/fedoraroot/var/lib/portables/
[root@baley toplevel]# btrfs subvolume snapshot /mnt/toplevel/var/lib/portables/ /mnt/toplevel/fedoraroot/var/lib/
```
6. Identified each subvolume so I could know which ones where mounted as my main filesystem:
```
[root@localhost ~]# touch /mnt/toplevel/root/stdroot
[root@localhost ~]# touch /mnt/toplevel/fedoraroot/fedoraroot
[root@baley toplevel]# touch fedoraroot/var/fedoravar
[root@baley toplevel]# touch ./var/stdvar
```
7. Updated fstab in the snapshot (that's in fedoraroot) since that what we want to use from now on. To do so, I replaced subvol=root by subvol=fedoraroot```and ``subvol=var by subvol=fedoraroot/var.
8. Rebooted.
9. Edited the grub line from the menu before booting from it. I updated the subvol parameter of the kernel to be subvol=fedoraroot. This way grub mounted the proper subvolume during boot.
10. Checked that everything was correct:
```
[root@localhost ~]# btrfs subvolume show /
fedoraroot
        Name:                   fedoraroot
        UUID:                   1fc8ffa2-76f5-a343-aae9-4b499073d805
        Parent UUID:            5effda61-179e-9b48-900f-b7ad85d5f9d9
        Received UUID:          -
        Creation time:          2020-02-22 16:14:19 +0100
        Subvolume ID:           268
        Generation:             4554
        Gen at creation:        4447
        Parent ID:              5
        Top level ID:           5
        Flags:                  -
        Snapshot(s):
[root@localhost ~]# ls /fedoraroot
/fedoraroot
```
11. Remounted the top level volume.
12. Deleted the nested subvolumes: btrfs subvolume delete /mnt/toplevel/root/var/lib/machines/ (same for portables).
13. Deleted root and var:
```
btrfs subvolume delete /mnt/toplevel/root/
btrfs subvolume delete /mnt/toplevel/var/
```
14. Updated grub: grub2-mkconfig -o /boot/grub2/grub.cfg # or /boot/efi/EFI/fedora/grub.cfg if you use EFI
15. Checked the generated grub configuration:
```
[root@localhost ~]# grep -R subvol /boot/grub2/grub.cfg  # The subvolume parameter is correct.
set default_kernelopts="root=UUID=ce254a08-22e0-481e-b027-498aa059cc1a ro rootflags=subvol=fedoraroot resume=UUID=163891ab-ac20-4e58-8731-306810f99c9c rhgb quiet "
```
16. Rebooted. All should still be working.

Installing Tumbleweed

Just do the normal installation process until you reach the disk setup. Here you'll have to choose Expert partition and Start with existing partitions. Then, on the disk that contains you BTRFS volume:

Mount the swap
Mount the EFI partition under /boot/efi (if you use EFI).
Mount the main BTRFS volume / and be sure to skip the formatting of the volume. /boot must be under / in the BTRFS volume even if you use luks. See below for why.
Check installation summary: should not remove anything but should create a lot of new subvolume. None should conflict with fedora's.

That should be it. Inspect in the summary the operations done. You should see that the installer:

Will create all the subvolumes it needs.
Won't delete anything.
Will mount the fedora subvolumes. Not a big deal, we will remove that from the fstab once the installation is done.

You can launch the installation.

Finishing the setup of Tumbleweed

If you use luks, you will have to enter you luks passphrase in Grub with a US keyboard. You will then have to enter you passphrase normally as part of the boot process.

By default Tumbleweed will create a subvolume named @ and create its other subvolumes as subvolumes of it. Here however, it didn't (I don't know why). Since I still want a main subvolume for each system (having home as a direct subvolume of the top level volume sounds good to me since it is independent of any other system). So, it's time to move subvolumes again. I won't detail much this time, I think you know the drill (do this from fedora, the directories are in use in Tumbleweed):

mount -t btrfs -o subvolid=5 /dev/vda3 /mnt/toplevel/
cd /mnt/toplevel/
btrfs subvolume list .

# Repeat for each subvolume.
rmdir /mnt/toplevel/@/var/
btrfs subvolume snapshot /mnt/toplevel/var /mnt/toplevel/@

# Check. Don't remove subvolume while in use, the deletion will come later.
btrfs subvolume list /mnt/toplevel
btrfs subvolume list .

Let's edit the fstab. We are just appending @/ to each subvol arguments like this: subvol=@/var.

We also need to update /boot/grub2/x86_64-efi/load.cfg so grub mounts the correct subvolume during the boot process:

set btrfs_relative_path='y'
cryptomount -u bf63369674824969a7726fda0cb46537
btrfs-mount-subvol ($root) /boot/grub2/x86_64-efi @/boot/grub2/x86_64-efi

Note

I didn't need this step in my VM and given the content of the file, I think it is specific to luks.

We can now reboot. Everything should go well. Boot on the other system if not.

We can now delete the old subvolumes from the installation:

mount -t btrfs -o subvolid=5 /dev/vda3 /mnt/toplevel/
cd /mnt/toplevel/
# Repeat for each subvolume.
btrfs subvolume delete ./var

We can also enable snapper now to have snapshot. It's done automatically during a normal installation but not here (because we did weird things I guess). We can do this with snapper -c root create-config /. And we are done!

Avis

Fedora uses subvol in grub.cfg but not Tumbleweed. Tumbleweed changes the default subvolume of the top level volume. It's not a problem with this setup but if you want to install two systems that do this, you will have to make at least one explicitly use subvol everywhere.

Wrapping up

It required a lot of preparation and tests but I'm glad I made it. I find Tumbleweed fast (faster than fedora at boot and when installing package) and reliable (from what I used until now). I'll need to do more testing to confirm that (the system is very fresh after all). I also learned a lot about BTRFS, subvolumes and snapshots along the way. I still have some issues though:

The nextcloud client ask me to reconnect each time I launch it. It may come from a configuration shared with Fedora. I'll fix this later once I know which system I want to keep.
The grub menu to choose the kernel (or the snapshot) is sometimes not displayed.
Grub takes some time to start once the key is entered.
If I fail to enter my passphrase for grub of the first try, it fails and I have to do a hard reboot.
I decided to switch from Gnome to KDE and I think I hit a bug with the online accounts that prevent me to setup my gmail account. I'll have to dig into this later (I'll try to update this post if I find the solution). I also spent a lot of time to configure KMail (which has a lot of options by the way).

Bonus

Enter luks passphrase only once

If you use luks, you have to enter your luks passphrase twice: once to unlock /boot so grub can start the system and then so the system can access the disk. Sadly, they can't communicate (yet?).

You could put /boot on an unencrypted filesystem (that's what fedora does), but according to this thread you won't be able to boot from a BTRFS snapshot (which is one cool thing Tumbleweed enables by default).

You can also fellow these steps so you'll only have to enter the passphrase for grub. Check also this link to update the initramfs in Tumbleweed (don't forget to launch dracut --force). For reference, my /etc/dracut.conf contains:

# PUT YOUR CONFIG IN separate files
# in /etc/dracut.conf.d named "<name>.conf"
# SEE man dracut.conf(5) for options
install_items+="/etc/luks-keys/root.key"

Note

If you need to add many files, separate them with spaces like this: install_items+="/etc/luks-keys/root.key /etc/luks-keys/swap.key"

Astuce

Don't add key-slot=1 immediately, if you did a mistake you will be locked out of the system because luks will try to match the key against this slot. Don't forget to adapt the slot if needed. You may also choose not to add it since I found it doesn't really change the boot time anyway.

Tumbleweed default install

For reference:

localhost:~ # btrfs subvolume list /
ID 256 gen 32 top level 5 path @
ID 258 gen 4192 top level 256 path @/var
ID 259 gen 4125 top level 256 path @/usr/local
ID 260 gen 4192 top level 256 path @/tmp
ID 261 gen 3045 top level 256 path @/srv
ID 262 gen 4166 top level 256 path @/root
ID 263 gen 3620 top level 256 path @/opt
ID 264 gen 4192 top level 256 path @/home
ID 265 gen 26 top level 256 path @/boot/grub2/x86_64-efi
ID 266 gen 3550 top level 256 path @/boot/grub2/i386-pc
ID 267 gen 4118 top level 256 path @/.snapshots
ID 268 gen 4191 top level 267 path @/.snapshots/1/snapshot
ID 274 gen 197 top level 267 path @/.snapshots/2/snapshot
ID 275 gen 262 top level 267 path @/.snapshots/3/snapshot
ID 278 gen 485 top level 267 path @/.snapshots/4/snapshot
ID 282 gen 1614 top level 267 path @/.snapshots/5/snapshot
ID 283 gen 2100 top level 267 path @/.snapshots/6/snapshot
ID 290 gen 3024 top level 267 path @/.snapshots/11/snapshot
ID 291 gen 3066 top level 267 path @/.snapshots/12/snapshot
ID 292 gen 3233 top level 267 path @/.snapshots/13/snapshot
ID 295 gen 3551 top level 267 path @/.snapshots/14/snapshot
ID 296 gen 3593 top level 267 path @/.snapshots/15/snapshot
ID 297 gen 3613 top level 267 path @/.snapshots/16/snapshot
ID 301 gen 3631 top level 267 path @/.snapshots/17/snapshot
ID 305 gen 4005 top level 267 path @/.snapshots/18/snapshot
ID 306 gen 4051 top level 267 path @/.snapshots/19/snapshot
ID 307 gen 4113 top level 267 path @/.snapshots/20/snapshot

localhost:/mnt/toplevel # cat /etc/fstab
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /                       btrfs  defaults                      0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /var                    btrfs  subvol=/@/var                 0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /usr/local              btrfs  subvol=/@/usr/local           0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /tmp                    btrfs  subvol=/@/tmp                 0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /srv                    btrfs  subvol=/@/srv                 0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /root                   btrfs  subvol=/@/root                0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /opt                    btrfs  subvol=/@/opt                 0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /home                   btrfs  subvol=/@/home                0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /boot/grub2/x86_64-efi  btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /boot/grub2/i386-pc     btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=e5a3ef21-d865-4bb6-bd11-6056ff4a60b1  /.snapshots             btrfs  subvol=/@/.snapshots          0  0
UUID=6f2fce23-6460-4d69-9e69-a5b6b5e4456c  swap                    swap   defaults                      0  0