Completely Fake Blog

Welcome to my blog

Fedora on ZFS - WIP

Thanks to Pete Turner I got intrested in ZFS on Linux again. Last time I tried it, it was unstable, but now it's supposed to be pretty on par with FreeBSD and illumos implementations. While Pete mentioned configuring CentOS server to use ZFS for appliations data storage, I ventured further - Fedira 23 on ZFS /.

As Expected, I wasn't the first one with such idea. Some searching revealed Rudd's Guide to Fedora on ZFS. Very well written. I'd only be catious with following his advice to turn off atime. There are many applications that rely on that being present in the filesystem, with imap servers as one example. Pretty good explanation why you'd may consider NOT disabling it is in this mail thread. As a storage engineer, I'd also recommend using other ZFS features for speed up, like l2arc and additional ZIL devices.

Having ten years of ZFS experience under my belt and being absolutely certain of its abilities and stability, I decided to give it a try on my Linux notebook. There are two features I like very much: filesystem compression and boot environments. FS compression should be self explanatory, but for people outside illumos/FreeBSD boot environments may be bit more cryptic.

First thing to explain is that ZFS is a copy on write filesystem. For more techical explanation, please visit this Wikipedia page on Copy-on-write and ZFS. For a quick background, lets only mention, that as long as ZFS has free disk space, it's not gonna overwrite old blocks of data, but each time you write the data, it will write to new filesystem location. This allows for quick and cheap snapshots. The moment you create a snaphot, ZFS records location of your data blocks and once you start to update data (writing it's new versions to a vdev), it will keep adding old blocks to the snapshot list. This way, only blocks of data that were changed since the snapshot creation will be held on storage.

One characteristic of snapshots is, they are read only. So ZFS engineers went one step further and created snapshots that allow writing to them, calling them clones. You can create a filesystem snapshot, called a clone, and mount it and write to it.

So, imagine that you have a stable operating system, say Fedora 23, and you wish to update it to 24 beta, but wish to retain ability to boot back into known, working version. With ZFS on /, you create a clone of your root filesystem. At the moment of creation clone consumes virtually no disk space, since there are no data changed on disk. You can then update one of the /. Both, old and new are bootable and the clone takes only as much space, as data writes introduced changes to the original filesytem.

I came to like it very much and illumos distributions the like of OpenIndiana came to rely on it during each major system update, with FreeBSD having its own clone of beadm command. So, why not try it with Fedora, my main laptop OS?

Not so easy, Doc! While Rudd's guide is very detailed and easy to follow, I came to a halt during first reboot after updating grub to boot from ZFS. After reboot, a rather unfriendly message welcomed me:
Fedora can't boot
First thought, lets see what's mounted. zpool list and zfs list revealed a complete list of expected filesystems. also mount has shown a long listing of mounted filesystems.
Fedora filesystems
Next command to run is systemctl (yes, I am clearly ignoring the os suggestion to look into journalctl). Looking at things that did not run, I see a line:
sysroot.mount loaded failed failed /sysroot
What do you know? It's the temporary mount point for the pool during the system migration from default file system to zfs, from Rudd's Guide. So, currently, I am debugging where I may have gone wrong, or where zfs on linux might have. On first sight, it seems like some stale zfs mountpoint property.

First things first, I booted back to ext4 root to have a look at pool and configuration files. Looking at /run/initramfs/rdsosreport.txt sure contains lot of entries about trying to mount /sysroot and failing that. Since pool doesn't contain this option anymore, I guess it must be living within initramfs created by dracut. First thought of setting the mountpoint for pool/ROOT to /sysroot didn't help at all. I got landed in the same very place.

Some googling got me here. Reading the report and mailing list got me to understand, that dracut tool and systemd may be broken with zfs in this combination of version. So, stay tuned, for next step, once I install fixed packages.