ZFS snapshots in action

I recently had my laptop running Xubuntu 19.10 reach its end of life for security updates. I needed to upgrade to a newer version of Xubuntu to continue receiving the important updates. Luckily when I originally put Xubuntu 19.10 on this laptop, I installed the OS using ZFS as the filesystem – a feature new to the Ubuntu installer at that time. Thankfully ZFS proved itself as a great safety net for when the upgrade failed midway through the Xubuntu upgrade to 20.04 (the laptop abruptly turned off). But first, some background on ZFS snapshots.

ZFS Snapshots

One of the features not discussed in my previous article on ZFS was the powerful snapshotting features available to use. For any ZFS dataset (synonymous with a filesystem), snapshots can be created to mark a moment in time for all data stored within the dataset. These snapshots can be created or removed at any time, and will take up more storage space over time as files are added and removed after a snapshot has been taken. With a snapshot created, at any point in the future it’s possible to rollback to this snapshot, or go and read the data within it. Rolling back to a snapshot effectively erases anything that happened after that snapshot was taken. There are more advanced uses for snapshots that can be discovered in this great resource.

Right after I installed 19.10 a year ago, I created a snapshot to mark a clean install of Xubuntu in case I messed something up and needed to revert to a fresh new install. I haven’t yet needed to use this at all. Next up I’ll walk through my experience upgrading to 20.04 and using ZFS snapshots.

Taking ZFS Snapshots

Xubuntu 19.10 recently stopped receiving security updates, and therefore I needed to upgrade. The 20.04 version is Ubuntu’s long-term support (LTS) release, which provides a number of years of support and security updates – far greater than the non-LTS releases such as 19.10. Going into the upgrade I made sure to make a snapshot of all of the different datasets before performing the upgrade. From the Ars Technica article referenced eariler, the following command takes a recursive snapshot of all datasets that are part of the rpool:

$ zfs snapshot -r rpool@2020-upgrade

No output means that the command was successful. The following command then shows all of the different datasets that were snapshotted in the pool named rpool. If you’re following along, this may look a bit different for you. Ubuntu’s installer creates many different datasets for different directories, and two pools, one named rpool, and the other named bpool (not important for this article).

$ zfs list -rt snap rpool | grep 2020-upgrade
rpool@2020-upgrade                                                0B      -       96K  -
rpool/ROOT@2020-upgrade                                           0B      -       96K  -
rpool/ROOT/ubuntu_191r26@2020-upgrade                          1.98G      -     6.49G  -
rpool/ROOT/ubuntu_191r26/srv@2020-upgrade                         0B      -       96K  -
rpool/ROOT/ubuntu_191r26/usr@2020-upgrade                         0B      -       96K  -
rpool/ROOT/ubuntu_191r26/usr/local@2020-upgrade                  72K      -      112K  -
rpool/ROOT/ubuntu_191r26/var@2020-upgrade                         0B      -       96K  -
rpool/ROOT/ubuntu_191r26/var/games@2020-upgrade                   0B      -       96K  -
rpool/ROOT/ubuntu_191r26/var/lib@2020-upgrade                  35.8M      -     1.32G  -
rpool/ROOT/ubuntu_191r26/var/lib/AccountServices@2020-upgrade     0B      -       96K  -
rpool/ROOT/ubuntu_191r26/var/lib/NetworkManager@2020-upgrade    156K      -      284K  -
rpool/ROOT/ubuntu_191r26/var/lib/apt@2020-upgrade              6.41M      -     88.6M  -
rpool/ROOT/ubuntu_191r26/var/lib/dpkg@2020-upgrade             18.8M      -     40.8M  -
rpool/ROOT/ubuntu_191r26/var/log@2020-upgrade                  23.0M      -     1011M  -
rpool/ROOT/ubuntu_191r26/var/mail@2020-upgrade                    0B      -       96K  -
rpool/ROOT/ubuntu_191r26/var/snap@2020-upgrade                    8K      -      160K  -
rpool/ROOT/ubuntu_191r26/var/spool@2020-upgrade                  72K      -      112K  -
rpool/ROOT/ubuntu_191r26/var/www@2020-upgrade                     0B      -       96K  -
rpool/USERDATA@2020-upgrade                                       0B      -       96K  -
rpool/USERDATA/jon_ip6jrn@2020-upgrade                          396M      -     17.8G  -
rpool/USERDATA/root_ip6jrn@2020-upgrade                         404K      -     1.87M  -

Now that a snapshot was created, I could make any change to the system and be able to rollback to this snapshot, undoing any changes that were made after the snapshot.

Upgrading to 20.04

To perform the OS upgrade to 20.04, a sudo do-release-upgrade was entered, initiating the upgrade. Things were progressing well until the laptop’s battery unexpectedly ran out. Plugging in the power and starting the laptop back up, the login screen wasn’t showing up. Great. Thankfully there’s the little-known virtual tty consoles available a keyboard combo away for cases where you need a terminal but aren’t able to use the graphical window manager.

Now that I have a terminal on the laptop, poking around has shown that the upgrade was definitely interrupted midway through. Only a handful of packages were installed and many more needed to be installed and configured.

Instead of going on and trying to manually fix the failed upgrade, why not roll back to the ZFS snapshot taken just before the upgrade and restart the upgrade from this fresh state? This is what is shown next. With the open terminal, executing this command rolled back the system to the state taken at the 2020-upgrade snapshot.

$ sudo zfs list -rt snap rpool | grep 2020-upgrade | awk '{print $1}' | xargs -I% sudo zfs rollback -r %
$ reboot now

Performing a reboot right after executing this series of commands makes sure that the system is properly initialized from the 2020-upgrade snapshot’s state.

To get a better idea of what the above series of commands does, refer to this Ars Technica article.

And it Worked

After the reboot, the system came back up looking like it was exactly where the snapshot had been taken. I was able to proceed again with the upgrade to 20.04, this time leaving the laptop plugged in.

The safety net of ZFS snapshots proved itself during this experience. It can feel scary knowing that your data is on the line if things go wrong. Having a strong understanding of how ZFS and related systems work helped me get through this without making any unrecoverable mistakes. If you haven’t read it already, my previous article on ZFS takeaways includes many of the references I used to build a strong understanding of ZFS.

ZFS takeaways

ZFS is quite a feature rich filesystem. If you’re managing a number of hard drives or SSDs, the benefits of using ZFS are numerous. For example, ZFS offers a more powerful software-based RAID than normal hardware-based RAID. Snapshotting is a powerful feature for versioning and replicating data. Data consistency is built in and automatically makes sure all data continues to stay readable over time. The pool of drives can grow or shrink while being transparent to the filesystem above it. These are only a few of the powerful features gained from using ZFS. With this power comes a fair amount of initial overhead learning about how ZFS works. Though it’s worth it if you value flexibility and your data. Here’s a number of resources and tips I found useful as I learned and used ZFS over the past few months.

For a general overview of ZFS and more of its benefits, see this great ZFS 101 article on Ars Technica.

Testing out ZFS by using raw files

Throughout the Ars Technica article above, the author uses files on the filesystem instead of physical devices to test out different pool configurations. This is very handy to build up experience with using the different zpool and zfs commands. For example, I used this to get a feel for using the different types of vdevs and using the zpool remove command. A quick example is as follows:

$ for n in {1..4}; do truncate -s 1G /tmp/$n.raw; done
$ ls -lh /tmp/*.raw
-rw-rw-r-- 1 jon jon 1.0G Dec 21 17:09 /tmp/1.raw
-rw-rw-r-- 1 jon jon 1.0G Dec 21 17:09 /tmp/2.raw
-rw-rw-r-- 1 jon jon 1.0G Dec 21 17:09 /tmp/3.raw
-rw-rw-r-- 1 jon jon 1.0G Dec 21 17:09 /tmp/4.raw
$ sudo zpool create test mirror /tmp/1.raw /tmp/2.raw mirror /tmp/3.raw /tmp/4.raw
$ zpool status test
  pool: test
 state: ONLINE
  scan: none requested
config:

	NAME            STATE     READ WRITE CKSUM
	test            ONLINE       0     0     0
	  mirror-0      ONLINE       0     0     0
	    /tmp/1.raw  ONLINE       0     0     0
	    /tmp/2.raw  ONLINE       0     0     0
	  mirror-1      ONLINE       0     0     0
	    /tmp/3.raw  ONLINE       0     0     0
	    /tmp/4.raw  ONLINE       0     0     0

errors: No known data errors
$ sudo zpool remove test mirror-0
$ zpool status test
  pool: test
 state: ONLINE
  scan: none requested
remove: Removal of vdev 0 copied 38.5K in 0h0m, completed on Mon Dec 21 17:39:09 2020
    96 memory used for removed device mappings
config:

	NAME            STATE     READ WRITE CKSUM
	test            ONLINE       0     0     0
	  mirror-1      ONLINE       0     0     0
	    /tmp/3.raw  ONLINE       0     0     0
	    /tmp/4.raw  ONLINE       0     0     0

errors: No known data errors
$ sudo zpool destroy test

Here I use truncate to create four 1 GB sized empty files, then created a zpool named test with two mirror vdevs, using those four raw files. Then mirror-0 is removed, moving any blocks over to mirror-1. The pool is then finally destroyed.

Really understanding vdevs

Vdevs are a foundational part of using ZFS, and knowing what each vdev type accomplishes, and their strengths and weaknesses helps build confidence in keeping your data safe. This Reddit post on the ZFS subreddit goes into detail about many of these considerations. Again, in advance of making changes to a production ZFS pool, dry-running the changes on a test pool can provide more confidence for the changes to be made.

Adding and removing disks

One of the newer features allows removing only certain types of vdevs from a pool via the zpool remove command. This Reddit post and answer goes into some of the different potential scenarios. I did some thorough testing with a test pool of raw files before making any changes to my production pool. The zpool manpages mention the following about what can and can’t be removed:

Removes the specified device from the pool. This command supports removing hot spare, cache, log, and both mirrored and non-redundant primary top-level vdevs, including dedup and special vdevs. When the primary pool storage includes a top-level raidz vdev only hot spare, cache, and log devices can be removed.

I was amazed at the ability of being able to remove a vdev from a pool and have all data transparently moved over to the rest of the pool with the pool still online. One command moved terabytes of data and verified its integrity before removing the vdev.

Use different /dev/ references when creating pools

A small, but important tip when it comes to which /dev/ to use when adding devices to a production pool is to stick to using the device symlinks provided by /dev/disk/by-id/ or /dev/disk/by-path/ as they are less likely to change. Referencing drives directly like /dev/sdc can be more risky as these identifiers can change whenever hardware is added or removed from the system. The OpenZFS docs provide a great rundown on why this is recommended.

Other helpful resources

These were just a handful of my biggest takeaways of using ZFS over the past couple of months. A number of useful resources I’ve found along the way can be found here:

Building a homelab – local DNS

This is a second post in a series of my experiences while Building a Homelab. The first post focusing on the history, hardware, and OS can be found here.

Having a number of networked devices at home presents some management overhead. You may find yourself asking, what was the IP address of that one laptop? or just getting plain old tired of looking at IP addresses. One method people often use to manage their network is to assign Domain Name System (DNS) names to their devices. Instead of constantly typing in 192.168.1.1 you could instead assign it the domain name router.home. Entering router.home into your browser then transparently brings you to the same webpage as 192.168.1.1. This not only works for browsing the internet, services such as SSH, FTP, and other places where an IP address would normally be used can likely use the friendlier domain name instead.

So how can this be done? It’s actually quite simple given you have an always-on computer on the same network as the rest of your devices, a router with DNS serving capabilities, or even a DNS provider such as Cloudflare. This article will focus on the DIY solution of running a DNS server on an always on computer.

Before we get to how to set this up, let’s first explain what DNS is and how it works. Feel free to skip over this section if you’re already knowledgeable.

What is DNS?

DNS is a technology used to translate human-friendly domain names to IP addresses. For example, we can ask a DNS server what is the IP address for the domain google.com? The DNS server would then respond with the IP address for google.com: 172.217.1.174. DNS is used for almost every request your computer, phone, smart lightbulbs, and more when it communicates with the internet.

Anyone who runs a website is using DNS whether they know it or not. Usually the basic premise is that each domain name (eg. mysite.com) will have a DNS record which points to an IP address. The IP address is the actual computer on the internet which traffic for mysite.com will be sent to.

An example of DNS being used can be for jonsimpson.ca. This site is hosted on a server that I pay for at DigitalOcean. That server has an IP address of 1.2.3.4 (a fictitious example). I use Cloudflare as the DNS provider for jonsimpson.ca. Anytime a user’s browser wants to go to jonsimpson.ca, it uses DNS to figure out that jonsimpson.ca is located at 1.2.3.4, then the user’s browser opens up a connection with the server at 1.2.3.4 to load this site.

This is quite a simplified definition of DNS as the system is distributed across the world, hierarchical, and involves hundreds of thousands, if not millions, of different entities. Cloudflare provides a more detailed explanation as to how DNS works, and Wikipedia has comprehensive coverage of multiple concerns relating to DNS. But what was explained earlier will provide enough context for this article.

Running a local DNS server

If there’s an always-on computer – whether that’s a spare computer or Raspberry Pi – a DNS server can run on it and provide DNS capabilities for the local network. Dnsmasq is a lightweight but powerful DNS server that has been around for a long time. Many hobbyists use Dnsmasq for their home environments since it’s quite simple to configure and get going. One minimal text file is all that’s needed for configuring a functional DNS service.

I chose to run Dnsmasq on my always-on server in a Docker container. When configuring Dnsmasq, for each device that I wanted to provide a domain name for, I added a line in the configuration mapping its IP address to the name I wanted to give it. For example, my router which lives at 192.168.1.1 was assigned router.home.mysite.com, and my server which lives at 192.168.1.2 was assigned server.home.mysite.com.

I then configured my router’s DHCP to tell all clients to use the DNS provided by the server (contact 192.168.1.2 for DNS), and configure some manually networked devices to explicitly use the DNS provided by the server. Now on all of my devices I can type in server.home.mysite.com anywhere I would type 192.168.1.2 – so much nicer compared to having to type in an entire IP address.

nslookup and dig are both common command line tools to query the Domain Name System. They are often found already available on many Linux and Unix operating systems, or a straightforward install away. Using these tools can help with inspecting and debugging DNS setups. Here’s an example query using nslookup to find google.com:

$ nslookup google.com
Server:          192.168.1.2
Address:        192.168.1.2#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.1.174

The first Server and Address denote the DNS server that was used to find the IP address for google.com. In this case, it was the Dnsmasq DNS server running on my home server. Name and Address at the bottom signify the actual response we’re interested in. In this case, 172.217.1.174 is the IP address I get whenever I go to google.com.

The configuration

I use Docker as a way to simplify the configuration and running of different services. Specifically, I use docker-compose to define the Dnsmasq Docker image to use, which ports should be opened, and where to find its configuration. Here’s the docker-compose.yml file I use:

The docker-compose file defines one dns service that uses the base image of strm/dnsmasq, as its one of the more popular Dnsmasq images available on hub.docker.com. The volume option specifies that we map a config file located alongside the docker-compose.yml file at config/dnsmasq.conf into the container’s filesystem at /etc/dnsmasq.conf. This is done to allow the container to be recreated at any time while keeping the same configuration. Networking-wise, TCP and UDP port 53 are exposed (yes, DNS operates over TCP sometimes). The network-mode is set to the host’s network (Dnsmasq just doesn’t work without this). And lastly, the NET_ADMIN capability so that we can use privileged ports below 1024. The last option restart, (one of my favourite features of docker-compose) is to keep the container running even when the host reboots or the container dies.

All of these docker-compose.yml options can be understood in more detail in Docker’s reference docs.

More importantly, here’s the dnsmasq.conf file I use to actually configure Dnsmasq’s DNS capabilities:

A lot of these settings were based off of the following blog post. Many of these options can be looked up online in the official documentation, therefore I will focus on the ones relevant to this article.

I have my Ubiquity router handle providing DHCP for my network, therefore the no-dhcp-interface=eno1 is set here to not provide any DHCP services to the local network, as eno1 is the interface my server uses to connect to the network.

When Dnsmasq needs to find the DNS record for something that it doesn’t know, it performs a request to an upstream DNS server. server is used for this and can be specified multiple times to provide redundancy in case one of these DNS servers are down. I’ve specified both the Google and Cloudflare DNS servers. In addition to this, the all-servers option results in all defined server entries being queried simultaneously. This has the benefit that one DNS server may respond quicker than the others, resulting a net-faster response to the DNS query.

The most important part of this dnsmasq.conf configuration file are the last lines defined in the file that start with address=. This is Dnsmasq’s way to declare DNS mappings. For example, any device on my network performing a request for server.home.mysite.com will have 192.168.1.2 returned.

The really cool thing with DNS is that subdomains for any of these records return the same IP, unless declared explicitly otherwise. An example of this is blog.apps.home.mysite.com doesn’t exist in the configuration file, but performing a DNS request for it will return 192.168.1.2. This has the effect that “multiple services” can each have its own domain name, but all be served by the same IP address.

Conclusion

Hopefully this article gives a background about what DNS is, how it can be useful in a home environment, and how to setup and operate a Dnsmasq DNS server. A future post will build on top of the DNS functionality that has been setup here to provide multiple HTTP services running on separate domain names, all served by the same server, for the home network to use.

Twenty Six

2020 is the year I turn 26. October 1st is that day! As always with every year for the past seven years now I reflect back on the past year by sharing some accomplishments and progress with life.

Travel

Given Covid-19, things have been different but manageable. Before the craziness started I travelled to Nashville in November for RubyConf (a conference for developers) with a number of teammates. We had a great time exploring the city and sampling the different foods over the few days we were there. I never realized how much of a party city Nashville is. The live music and bar scene wants me to go back again with friends. I’ll have to grab a cowboy hat next time I’m there.

PEI lighthouse

In the new year there was a work trip to Montreal for a couple days. Great times were spent getting to know new colleagues from our greater team, and having fun with existing. The city never gets boring. This trip is my best celebrity claim to fame: at a speakeasy in old Montreal, Harrison Ford (of Bladerunner, Star Wars) showed up with a few people and had a discreet time. They didn’t want any attention, therefore the group I was with and myself weren’t able to actually meet him. Oh well, at least he walked around a bit so that we could try to remember him a whole lot better.

Right before things started shutting down in Ottawa at the end of February, my mom and sister gave me a surprise visit. The highlights were hitting up the town with my friends and going to some new restaurants. Recommendations are for the Duelling Pianos event at the Sens House Saturday nights, and breakfast at the Manx on Elgin.

Isolating in Nova Scotia

Part way through the pandemic, I had the opportunity to travel out east to Nova Scotia with a few friends out to their family’s beach house. We had to quarantine for two weeks while out there but that wasn’t too difficult when the weather was hot, the beach was there, and light beer was plentiful. It was so great the first time that we decided to go back a second time (including a second quarantine) for an entire month. During these two trips I attended one of my best buddy’s wedding, his bachelor party, made apple cider, experienced the east coast cultural norms, and rekindled my love of rocks. Home base was Merigomish, Nova Scotia, but we also stayed in Fredericton, Moncton, Charlottetown, and French River, PEI. It was surreal to experience the relaxed Covid restrictions out in the eastern bubble. I’m thankful that I was able to work remotely while out there and it having no impact on my work. I’m glad to have been able to travel during the pandemic.

Fitness

To stay fit throughout the winter I invested in a smart trainer for my bike. Tied with the virtual cycling app, Zwift, I started crushing seasons of Brooklyn Nine-Nine while keeping my fitness up.

With Fridays being days off during the summer months at Shopify, this gave my friends and I a lot of great summer days to get up to trouble. Many of the days were spent cycling around Ottawa and going to different beaches in the area. We frequented Aylmer beach on the Quebec side. It was a scenic hour long cycle and great sandy beach, perfect for very hot days. One time we went out to Sablon Beach for hanging out at the large beach and camping over for a night. With all this travel and sun, it’s been satisfying to work on getting a nice, even tan.

Many cycling trips into Gatineau Park

A few friends and I signed up to run the Ottawa Race Weekend 10k. The last time I did this was in 2017! The official race was cancelled, but could still be run any time over the summer, and anywhere to still get ranked. On the last day to submit the results I was running a 5k and decided to see how much further I could go. Pacing myself and keeping good enough form allowed me to run the 10k successfully! and in a fair time, albeit with some breaks.

Work

At work my team and I have transitioned to working in a different problem space. It’s quite a refreshing feeling being immersed in a little-known area as it keeps me on my toes. There was a several month period where I was leading two teams of developers on two different projects – one being the old team and project that is wrapping up, the other being the new team and project. This was challenging as my time was split between the two teams. Prioritizing, delegating, and providing the right nudges to influence the projects were critical throughout this period, especially when one team was coming up to a big launch, and the other team was trying to get off the ground. At the end of the day, the launch was wildly successful, and the new team is just about to ship its first version of the service we’re building.

Earlier in 2020, our greater team moved from the Support org into the Trust org. We’re still solving problems for Support, but our scope has expanded to accelerate the rest of the business. This is a great opportunity for us that hasn’t fully come to fruition just yet. A lot of our services can be leveraged by other teams. The mindset we have now is building services that provide a whole lot of leverage and speed to other teams. Every year things change a whole lot on my team for the better. Three years in now and it’s still a wild ride.

Numbers

As always, here’s some numbers on what I’ve accomplished over the year:

  • 110 km of running, 11 hours total
  • 1,039 km of cycling, 47 hours total
  • 1,010 Github contributions from work and personal projects
  • 3 books read, 5 on the go
  • 6 posts published on this blog, 5 unpublished
  • 2,373,358 steps, 1,642.99 km, 868,256 calories recorded via my Fitbit

Looking back, my running and cycling rival my 2017 numbers. It’s been a great time outside this year!

🍻 to another year!

Building a homelab – a walk through history and investing in new hardware

This is the first post in a series of my experiences while Building a Homelab. The second post focuses on setting up a local DNS server and can be found here.

I’ve had a particular interest in home computers and servers for a long time now. One of my experiences was wiring my childhood home up with CAT-5 ethernet to the rooms with TVs or computers and having them all connected to a 24 port 100 Mbps switch in the crawlspace. This was part of a master plan to provide different computers in the house with internet connection (when WiFi wasn’t as good as it is today), TVs with smart media boxes (think Apple TV, Roku, and the like but 10 years ago), and to tie it all together a home server for serving media storing files.

The magazine Maximum PC was a major source for this inspiration as they had a number of captivating DIY articles for running your own home server, media streaming devices, and home networking. The memory is a bit rough around the edges, but these projects happened around the same time and on my own dollar – all for the satisfaction of having a bleeding edge entertainment system.

Around this time Windows had a product out for a year called Windows Home Server. It was a OS which catered towards consumers and their home needs. Some of the features it had was network file shares for storing files, computer backup and restore, media sharing, and a number of extensions available from the community. I built a $400 box to run this OS and store two hard drives. The network switch in the crawlspace was a perfect place to put this headless server. Over many years this server was successfully used for computer backups, file storage, network bandwidth monitoring, and media serving to a number of PCs and media streaming boxes attached to TVs.

Two of the TVs in the house had these Western Digital TV Live boxes for playing media off of the network. These devices were quite basic at the time where only Youtube, Flickr, and a handful of other services were available – lacking Netflix and the other now popular Internet streaming services. Instead, they were primarily built for streaming media off of the local network – in this case off of the home server file share. My family and I were able to watch movies and TV shows from the comfort of our couch, and on-demand. This was crazy cool at the time as most people were still using physical media (DVD/Blu-ray) and streaming media had not taken off yet. I also vaguely remember hacking one of the boxes to put on a community-built firmware.

Windows Home Server was great at the time since it offered all of this functionality out of the box with simple configuration. I remember playing with BSD-based FreeNAS on old computers and being overwhelmed at all of the extra configuration needed to achieve something that you get out of the box with Windows Home Server. Additionally, the overhead of having to administer FreeNAS while only having a vague knowledge of Linux and BSD at the time wasn’t a selling point.

Now back to current times. I’m in the profession of software development, have been using various Linux distros for personal use on laptops and servers, and would now consider myself a sysadmin enthusiast. Living in my own place, I’ve been using my own Ubuntu-based laptop to run a Plex media server and stream content to my Roku Streaming Stick+ attached to my TV. The laptop’s 1 TB hard drive was filling up. It was also inconvenient to have this laptop constantly on for serving content.

Browsing Reddit, I came across r/homelab, a community of people interested in owning and managing servers for their own fun. Everything from datacenter server hardware to Raspberry PIs, networking, virtualization, operating systems, and applications. This subreddit gave me the idea of purchasing some decommissioned server hardware from eBay. I sat on the idea for a few months. Covid-19 eventually happened and with all my spare time I gave in to buying some hardware.

After a bunch of research on r/homelab about which servers are quiet, energy efficient, extendable, and will last a number of years, I settled on a Dell R520 with 2 x 6 cores at 2.4 Ghz, 48 GB DDR3 RAM, 2 x 1 Gbit NICs and 8 x 3.5″ hard drive bays. I bought a 1 TB SSD as the boot drive and a refurbished 10 TB hard drive for storing data.

The front of the Dell R520, showing the 8 3.5″ drive bays and some of the internals.

Since I intended on running the ZFS filesystem on the data drive, many people gave the heads up that the Host Bus Adaptor (HBA) card (a piece of hardware which connects the SAS/SATA hard drives and SSDs to the motherboard) comes with the default Dell firmware. This default firmware caters towards always running some sort of hardware-based RAID setup, thus hiding the SMART status of all drives. With ZFS, accessing the SMART data for each drive is paramount for data integrity. To get around this limitation with the included HBA card, the homelab community has some unofficial firmware for it which exposes IT mode, basically a way to pass through each drive to the OS – completely bypassing any hardware RAID functionality. Some breath holding later and the HBA card now had the new firmware.

I bought a separate HBA card with the knowledge at the time that the one that comes with the Dell R520 didn’t have any IT mode firmware from the community. I ended up being wrong after a whole lot of investigation. Thankfully I should be able to flash new firmware on this card as well and sell it back on eBay.

A Dell Perc H310 Mini Mono HBA (Host Bus Adaptor) used in Dell servers for interfacing between the motherboard and SAS/SATA drives.

As the hardware was all being figured out, I was also researching and playing with different hypervisors – an operating system made for running multiple operating systems on the same hardware. The homelab community often refers to VMware ESXi, Proxmox VE, and even Unraid. I sampled out the first two, as Unraid didn’t have an ISO available to test with and wasn’t free.

Going through the pain of making a USB stick bootable for an afternoon, I eventually got ESXi installing on the system. Poking around, it was interesting to see that VM storage was handled by having a physical disk formatted to a VMware format specific to storing multiple VMs – vmfs. With the goal of having one of the VMs have full control over a drive formatted with the ZFS filesystem, ESXi provides a feature called hardware passthrough which bypasses virtualization of the physical hardware. One big blocker for myself was the restriction on the free version which limits VMs to a maximum of 8 vCPUs – a waste of resources when having 12 CPUs and not enough VMs to utilize them.

Next, I took a look at Proxmox by loading it up as a VM on ESXi. It was Debian based, which was a plus as I’m comfortable with systemd and Ubuntu systems already. The Proxmox UI appeared like it had quite a few useful features, but didn’t feel like what I needed. I was much more comfortable with the terminal, and these graphical interfaces to manage things felt more like a limitation than a benefit. I could always SSH into Proxmox and manage things there, but there’s always the aspect of learning the intricacies of how this turnkey system was setup. Who knows what was default Debian configured and what was modified by Proxmox. Not to mention, what if Docker or other software was out of date and couldn’t be upgraded? This would be an unnecessary limitation I could avoid if rolling my own.

Lastly, I went back to my roots – Ubuntu Server. I spun up a VM of it on ESXi. Since I’m quite used to the way Ubuntu works it was comfortable knowing what I could do. There were no 8 vCPU limitations with Ubuntu Server as the host OS – I can utilize all of the server’s resources. After some thinking I realized I didn’t have any need to run any VMs at the moment. In the past I’ve managed a number of VMs using QEMU using Ubuntu Server, therefore if the need arises again I can pull it off. The reason why I’m not using any VMs is because I’m using Docker for all of my application needs. I already have a few apps running in Docker containers on my laptop that I’ll eventually transfer over to the server. Next up, ZFS on Linux has been available for a while now in Ubuntu, giving me the confidence that the data drive will be formatted with ZFS without a problem.

The internals of the Dell R520 with the thermal cover removed. Note the row of six fans across the width of the case to keep things cool.

In the end I scrapped the idea of running a hypervisor such as EXSi and running multiple VMs on top of it because my workloads all live in Docker containers instead. Ubuntu Server is more suitable since I am able to configure everything from a SSH console. If I may conjecture why the r/homelab community loves their VMs, it may be because many of the hobbyists are used to using them for their day-jobs. There were a handful of folks who did run their own GUI-less, no-VM setups, but it was the minority.

In the end, Ubuntu Server 20.04 LTS was installed on a 1 TB SSD boot drive. A 10 TB HDD was formatted with ZFS in a single drive configuration. Docker daemon was installed from its official Apt repo, and a number of other non-root processes were installed from Nix and Nixpkgs.

Conclusion

There’s a few more things I want to discuss regarding the home server. Some of those include using Nix and Nixpkgs in a server environment and some of the difficulties, setting up a local DNS server to provide domain name resolution for devices on the network and in Docker containers, a reverse proxy for the webapps running in Docker containers using the Caddy webserver, and some DataDog monitoring.

In the future I have plans to expand the amount of storage while at the same time introducing some redundancy with ZFS RAIDz1, diving into being able to remotely access the local network via VPN or some other secure method, and better monitoring for uptime, ZFS notifications, OS notifications, and the like.