ZFS snapshots in action
I recently had my laptop running Xubuntu 19.10 reach its end of life for security updates. I needed to upgrade to a newer version of Xubuntu to continue receiving the important updates. Luckily when I originally put Xubuntu 19.10 on this laptop, I installed the OS using ZFS as the filesystem – a feature new to the Ubuntu installer at that time. Thankfully ZFS proved itself as a great safety net for when the upgrade failed midway through the Xubuntu upgrade to 20.04 (the laptop abruptly turned off). But first, some background on ZFS snapshots.
ZFS Snapshots
One of the features not discussed in my previous article on ZFS was the powerful snapshotting features available to use. For any ZFS dataset (synonymous with a filesystem), snapshots can be created to mark a moment in time for all data stored within the dataset. These snapshots can be created or removed at any time, and will take up more storage space over time as files are added and removed after a snapshot has been taken. With a snapshot created, at any point in the future it’s possible to rollback to this snapshot, or go and read the data within it. Rolling back to a snapshot effectively erases anything that happened after that snapshot was taken. There are more advanced uses for snapshots that can be discovered in this great resource.
Right after I installed 19.10 a year ago, I created a snapshot to mark a clean install of Xubuntu in case I messed something up and needed to revert to a fresh new install. I haven’t yet needed to use this at all. Next up I’ll walk through my experience upgrading to 20.04 and using ZFS snapshots.
Taking ZFS Snapshots
Xubuntu 19.10 recently stopped receiving security updates, and therefore I needed to upgrade. The 20.04 version is Ubuntu’s long-term support (LTS) release, which provides a number of years of support and security updates – far greater than the non-LTS releases such as 19.10. Going into the upgrade I made sure to make a snapshot of all of the different datasets before performing the upgrade. From the Ars Technica article referenced eariler, the following command takes a recursive snapshot of all datasets that are part of the rpool:
$ zfs snapshot -r rpool@2020-upgrade
No output means that the command was successful. The following command then shows all of the different datasets that were snapshotted in the pool named rpool. If you’re following along, this may look a bit different for you. Ubuntu’s installer creates many different datasets for different directories, and two pools, one named rpool, and the other named bpool (not important for this article).
$ zfs list -rt snap rpool | grep 2020-upgrade
rpool@2020-upgrade 0B - 96K -
rpool/ROOT@2020-upgrade 0B - 96K -
rpool/ROOT/ubuntu_191r26@2020-upgrade 1.98G - 6.49G -
rpool/ROOT/ubuntu_191r26/srv@2020-upgrade 0B - 96K -
rpool/ROOT/ubuntu_191r26/usr@2020-upgrade 0B - 96K -
rpool/ROOT/ubuntu_191r26/usr/local@2020-upgrade 72K - 112K -
rpool/ROOT/ubuntu_191r26/var@2020-upgrade 0B - 96K -
rpool/ROOT/ubuntu_191r26/var/games@2020-upgrade 0B - 96K -
rpool/ROOT/ubuntu_191r26/var/lib@2020-upgrade 35.8M - 1.32G -
rpool/ROOT/ubuntu_191r26/var/lib/AccountServices@2020-upgrade 0B - 96K -
rpool/ROOT/ubuntu_191r26/var/lib/NetworkManager@2020-upgrade 156K - 284K -
rpool/ROOT/ubuntu_191r26/var/lib/apt@2020-upgrade 6.41M - 88.6M -
rpool/ROOT/ubuntu_191r26/var/lib/dpkg@2020-upgrade 18.8M - 40.8M -
rpool/ROOT/ubuntu_191r26/var/log@2020-upgrade 23.0M - 1011M -
rpool/ROOT/ubuntu_191r26/var/mail@2020-upgrade 0B - 96K -
rpool/ROOT/ubuntu_191r26/var/snap@2020-upgrade 8K - 160K -
rpool/ROOT/ubuntu_191r26/var/spool@2020-upgrade 72K - 112K -
rpool/ROOT/ubuntu_191r26/var/www@2020-upgrade 0B - 96K -
rpool/USERDATA@2020-upgrade 0B - 96K -
rpool/USERDATA/jon_ip6jrn@2020-upgrade 396M - 17.8G -
rpool/USERDATA/root_ip6jrn@2020-upgrade 404K - 1.87M -
Now that a snapshot was created, I could make any change to the system and be able to rollback to this snapshot, undoing any changes that were made after the snapshot.
Upgrading to 20.04
To perform the OS upgrade to 20.04, a sudo do-release-upgrade was entered, initiating the upgrade. Things were progressing well until the laptop’s battery unexpectedly ran out. Plugging in the power and starting the laptop back up, the login screen wasn’t showing up. Great. Thankfully there’s the little-known virtual tty consoles available a keyboard combo away for cases where you need a terminal but aren’t able to use the graphical window manager.
Now that I have a terminal on the laptop, poking around has shown that the upgrade was definitely interrupted midway through. Only a handful of packages were installed and many more needed to be installed and configured.
Instead of going on and trying to manually fix the failed upgrade, why not roll back to the ZFS snapshot taken just before the upgrade and restart the upgrade from this fresh state? This is what is shown next. With the open terminal, executing this command rolled back the system to the state taken at the 2020-upgrade snapshot.
$ sudo zfs list -rt snap rpool | grep 2020-upgrade | awk '{print $1}' | xargs -I% sudo zfs rollback -r %
$ reboot now
Performing a reboot right after executing this series of commands makes sure that the system is properly initialized from the 2020-upgrade snapshot’s state.
To get a better idea of what the above series of commands does, refer to this Ars Technica article.
And it Worked
After the reboot, the system came back up looking like it was exactly where the snapshot had been taken. I was able to proceed again with the upgrade to 20.04, this time leaving the laptop plugged in.
The safety net of ZFS snapshots proved itself during this experience. It can feel scary knowing that your data is on the line if things go wrong. Having a strong understanding of how ZFS and related systems work helped me get through this without making any unrecoverable mistakes. If you haven’t read it already, my previous article on ZFS takeaways includes many of the references I used to build a strong understanding of ZFS.