Completely Fake Blog

Welcome to my blog

ZFS on root on LUKS - Ubuntu

I couldn't find a good, end to end, guide on getting Ubuntu installed on ZFS on LUKS encrypted device. What I present below is a step by step report of what I did to get it working in VirtualBox VM. Currently seperate /boot partition was tested only, old BIOS boot was tested only. Plans for UEFI and for /boot as part of the pool itself are underway.

Great kudos to Andrea Bernabei

for initial set of notes and steps that constitute majority of my guide. What I did actually was simply testing them out and straightening some wrinkles.

Kudos too to ZFS on Linux Ubuntu Howto document, from which I have taken most of non-LUKS related steps.

Thank also need to go to several people that found errors in this text:

Instead of running each command with sudo, you can choose to run

$ sudo -i
and work as root.

As a first step, download Ubuntu live CD. I preffered plain Ubuntu install CD. Boot to it and choose Try Ubuntu. After you get the desktop, open terminal and add universe repository:

$ sudo apt-add-repository universe
'universe' distribution component enabled for all sources.
Next step is to install several packages that will be needed. Be sure to only install essential pacakges. Remember, you are currently in a virtual filesystem from liveCD, it is not very big and you can easily run out of space. Before being able to use this repo, you need to run:
$ sudo apt update
gn:1 cdrom://Ubuntu 16.04.1 LTS _Xenial Xerus_ - Release amd64 (20160719) xenial InRelease
Hit:2 cdrom://Ubuntu 16.04.1 LTS _Xenial Xerus_ - Release amd64 (20160719) xenial Release
Hit:4 xenial InRelease                                                                                                                                              
Hit:5 xenial-updates InRelease                                                                                                                                      
Get:6 xenial/universe amd64 Packages [7532 kB]                                                                                                                      
Hit:7 xenial-security InRelease                                                                                                                                    
Get:8 xenial-security/universe amd64 Packages [73.8 kB]                                                                                                            
Get:9 xenial-security/universe Translation-en [41.7 kB]                                                                                                            
Get:10 xenial-security/universe amd64 DEP-11 Metadata [32.1 kB]                                                                                                    
Get:11 xenial-security/universe DEP-11 64x64 Icons [37.0 kB]                                                                                                       
Get:12 xenial/universe Translation-en [4354 kB]                                                                                                                     
Get:13 xenial/universe amd64 DEP-11 Metadata [3410 kB]                                                                                                              
Get:14 xenial/universe DEP-11 64x64 Icons [7448 kB]                                                                                                                 
Get:15 xenial-updates/universe amd64 Packages [396 kB]                                                                                                              
Get:16 xenial-updates/universe Translation-en [149 kB]                                                                                                              
Get:17 xenial-updates/universe amd64 DEP-11 Metadata [133 kB]                                                                                                       
Get:18 xenial-updates/universe DEP-11 64x64 Icons [162 kB]                                                                                                          
Fetched 23.8 MB in 8s (2961 kB/s)                                                                                                                                                                    
** (appstreamcli:7538): CRITICAL **: Error while moving old database out of the way.                                                                                                                 
AppStream cache update failed.                                                                                                                                                                       
Reading package lists... Done                                                                                                                                                                        
Building dependency tree                                                                                                                                                                             
Reading state information... Done
397 packages can be upgraded. Run 'apt list --upgradable' to see them.
Ignore the Critical error.
$ sudo apt install zfsutils-linux zfs-initramfs debootstrap
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libnvpair1linux libuutil1linux libzfs2linux libzpool2linux zfs-doc zfs-zed
Suggested packages:
  default-mta | mail-transport-agent nfs-kernel-server
The following NEW packages will be installed:
  debootstrap libnvpair1linux libuutil1linux libzfs2linux libzpool2linux zfs-doc zfs-initramfs zfs-zed zfsutils-linux
0 upgraded, 9 newly installed, 0 to remove and 397 not upgraded.
Need to get 938 kB of archives.
After this operation, 3185 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Choose "y" and watch packages being installed:
Get:1 xenial-updates/main amd64 debootstrap all 1.0.78+nmu1ubuntu1.2 [37.6 kB]
Get:2 xenial-updates/main amd64 zfs-doc all [49.9 kB]
Get:3 xenial-updates/main amd64 libuutil1linux amd64 [27.4 kB]
Get:4 xenial-updates/main amd64 libnvpair1linux amd64 [23.4 kB]
Get:5 xenial-updates/main amd64 libzpool2linux amd64 [384 kB]
Get:6 xenial-updates/main amd64 libzfs2linux amd64 [106 kB]
Get:7 xenial-updates/main amd64 zfsutils-linux amd64 [276 kB]
Get:8 xenial-updates/universe amd64 zfs-initramfs all [4130 B]
Get:9 xenial-updates/main amd64 zfs-zed amd64 [29.6 kB]
Fetched 938 kB in 0s (1310 kB/s)
Selecting previously unselected package debootstrap.
(Reading database ... 194689 files and directories currently installed.)
Preparing to unpack .../debootstrap_1.0.78+nmu1ubuntu1.2_all.deb ...
Unpacking debootstrap (1.0.78+nmu1ubuntu1.2) ...
Selecting previously unselected package zfs-doc.
Preparing to unpack .../zfs-doc_0.6.5.6-0ubuntu15_all.deb ...
Unpacking zfs-doc ( ...
Selecting previously unselected package libuutil1linux.
Preparing to unpack .../libuutil1linux_0.6.5.6-0ubuntu15_amd64.deb ...
Unpacking libuutil1linux ( ...
Selecting previously unselected package libnvpair1linux.
Preparing to unpack .../libnvpair1linux_0.6.5.6-0ubuntu15_amd64.deb ...
Unpacking libnvpair1linux ( ...
Selecting previously unselected package libzpool2linux.
Preparing to unpack .../libzpool2linux_0.6.5.6-0ubuntu15_amd64.deb ...
Unpacking libzpool2linux ( ...
Selecting previously unselected package libzfs2linux.
Preparing to unpack .../libzfs2linux_0.6.5.6-0ubuntu15_amd64.deb ...
Unpacking libzfs2linux ( ...
Selecting previously unselected package zfsutils-linux.
Preparing to unpack .../zfsutils-linux_0.6.5.6-0ubuntu15_amd64.deb ...
Unpacking zfsutils-linux ( ...
Selecting previously unselected package zfs-initramfs.
Preparing to unpack .../zfs-initramfs_0.6.5.6-0ubuntu15_all.deb ...
Unpacking zfs-initramfs ( ...
Selecting previously unselected package zfs-zed.
Preparing to unpack .../zfs-zed_0.6.5.6-0ubuntu15_amd64.deb ...
Unpacking zfs-zed ( ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Processing triggers for initramfs-tools (0.122ubuntu8.1) ...
update-initramfs is disabled since running on read-only media
Processing triggers for systemd (229-4ubuntu7) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up debootstrap (1.0.78+nmu1ubuntu1.2) ...
Setting up zfs-doc ( ...
Setting up libuutil1linux ( ...
Setting up libnvpair1linux ( ...
Setting up libzpool2linux ( ...
Setting up libzfs2linux ( ...
Setting up zfsutils-linux ( ...
zfs-import-cache.service is a disabled or a static unit, not starting it.
zfs-import-scan.service is a disabled or a static unit, not starting it.
zfs-mount.service is a disabled or a static unit, not starting it.
Processing triggers for initramfs-tools (0.122ubuntu8.1) ...
update-initramfs is disabled since running on read-only media
Setting up zfs-zed ( ...
zed.service is a disabled or a static unit, not starting it.
Setting up zfs-initramfs ( ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Processing triggers for systemd (229-4ubuntu7) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for initramfs-tools (0.122ubuntu8.1) ...
update-initramfs is disabled since running on read-only media

If you'd wish to access this computer from other terminal, also install openssh-server and set password for ubuntu user:
$ passwd ubuntu
Once you got packages installed, create partitions. Since I don't have lots of experience with UEFI and I don't want to mess around with /boot on ZFS, I opted for separate /boot partition. Thus, my partition table looks like this (cut down for brevity):
$ sudo fdisk -l /dev/sda

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1           2048  1001471   999424  488M 83 Linux
/dev/sda2        1001472 33277951 32276480 15.4G 83 Linux
/dev/sda3       33277952 33554431   276480  135M bf Solaris

It's a quick compilation of several other guides partitioning scheme. This Solaris partition seems to be important. I got kernel panic on boot if it's not present, even not formatted.

Once partitions are set up, create LUKS device and decrypt it:

$ $ sudo cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha512 /dev/sda2

This will overwrite data on /dev/sda2 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase: 
Verify passphrase: 
I thought it's a one time operation, but it seems you can change the passphrase later, so it may be wise to set it up as something simple for duration of works. You will be asked to provide the passphrase twice. Decrypt the device:
$ sudo cryptsetup luksOpen /dev/sda2 root_crypt
Enter passphrase for /dev/sda2:
Once decrypted, new device in the /dev/mapper/ directory will be created for you. You can use it to create the ZFS pool:
zpool create -o ashift=12 -O canmount=off -O compression=lz4  -O mountpoint=/ -R /mnt rpool /dev/mapper/root_crypt 
Confirm that the pool exists by running:
$ sudo zpool list
rpool  15.5G   308K  15.5G         -     0%     0%  1.00x  ONLINE  /mnt

Several things here need discussion.

  • Some guides over the net specify atime=off option when creating the pool. It may be and may be not the best choise. If you choose to use atime=off, you may see some preformance boost, as fileystem will not track file access times. On the other hand, some software depend on this to be correctly updated. You should research before using. I decided to not disable atime. This can be turned on/off later, during system operation.
  • ashift=12 - describes your hard drive physical block size. The better this parameter aligns with your drive's physical size, the better performance. If it's misaligned, performance penalty may be severe. New drives all have physical block size equal 4k bytes. If their firmware advertises this fact, you can leave off ashift parameter from pool creation line and ZFS will choose it for you properly. Some drives, however, report 512bytes block size, even though they realy have 4k blocks - for backwards compatibility. For those you have to specify manually ashift. Usually vendors are honest about this and smartmonctl will happily inform you about both phsycal and logical size. This is a per vdev (per disk) parameter and cannot be changed once set.
  • compression - in today computers, your slowest part is most usually your hard drive. It changes a little with SSD drives, but they still remain slowest part of your system. CPUs however, even in notebooks, become powerfull and threaded. It is in most case a safe bet, that you can enable filesystem compression and gain not only disk space, but also some speed, since less data will be written to and read from the hard drive. For data that is already compressed, like music or movies, you can reate a separate filesystem with compression turned off. This parameter can be changed during the system operation.

It's now time to create filesystems:

$ sudo zfs create -o canmount=off -o mountpoint=none rpool/ROOT
$ sudo zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/ubuntu
$ sudo zfs mount rpool/ROOT/ubuntu 
$ sudo zfs create -o setuid=off rpool/home 
$ sudo zfs create -o mountpoint=/root rpool/home/root 
$ sudo zfs create -o canmount=off -o setuid=off -o exec=off rpool/var 
$ sudo zfs create -o com.sun:auto-snapshot=false rpool/var/cache 
$ sudo zfs create rpool/var/log 
$ sudo zfs create rpool/var/spool 
$ sudo zfs create -o com.sun:auto-snapshot=false -o exec=on rpool/var/tmp 
$ sudo chmod 1777 /mnt/var/tmp
ZFS tools will not report anything if there's no error creating datasets. You can confirm they exist, by running:
$ sudo zfs list
rpool              1.30M  15.0G    96K  /mnt
rpool/ROOT          192K  15.0G    96K  none
rpool/ROOT/ubuntu    96K  15.0G    96K  /mnt
rpool/home          192K  15.0G    96K  /mnt/home
rpool/home/root      96K  15.0G    96K  /mnt/root
rpool/var           480K  15.0G    96K  /mnt/var
rpool/var/cache      96K  15.0G    96K  /mnt/var/cache
rpool/var/log        96K  15.0G    96K  /mnt/var/log
rpool/var/spool      96K  15.0G    96K  /mnt/var/spool
rpool/var/tmp        96K  15.0G    96K  /mnt/var/tmp
It's time to install the base minimum system to the newly created rpool:
$ sudo deboostrap xenial /mnt
The output is too long to be pasted here. You can watch packages being delivered to the /mnt directory, filling your new ZFS filesystems. You are waiting for this line:
I: Base system installed successfully.

There are several configuration points to be taken care of and you will then be chrooting into your new system
$ zfs set devices=off rpool
$ sudo vi /mnt/etc/hostname # in this point change hostname to whatever you wish
$ vi /mnt/etc/hosts # Change line to read your new honstname, ie:  kubuntu
Check what interface is your primary interface to the world. Remember, if you're working off WiFi and need wpa-supplicant, you will must install it during this phase.
$ sudo ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:94:88:ae brd ff:ff:ff:ff:ff:ff
    inet brd scope global dynamic enp0s3
       valid_lft 5727sec preferred_lft 5727sec
    inet6 fe80::5b00:5cb6:1bd3:f8be/64 scope link 
       valid_lft forever preferred_lft forever
I am using enp0s3 and this is going to be configured by hand:
$ sudo vi /mnt/etc/network/interfaces.d/enp0s3.cfg
  auto enp0s3
  iface enp0s3 inet dhcp

Mysetup expects /boot to be a separate partition, unencrypted and non-ZFS:

$ sudo mkfs.ext4 -m 0 -L /boot -j /dev/sda1
mke2fs 1.42.13 (17-May-2015)
/dev/sda1 contains a ext4 file system
        created on Mon Feb 13 19:35:45 2017
Proceed anyway? (y,n) y
Creating filesystem with 499712 1k blocks and 124928 inodes
Filesystem UUID: 2a0b225c-e1bb-43f2-961c-7925678fa42f
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done 
It's now time to enter chroot:
$ sudo mount --rbind /dev  /mnt/dev 
$ sudo mount --rbind /proc /mnt/proc 
$ sudo mount --rbind /sys  /mnt/sys 
$ sudo chroot /mnt /bin/bash --login

Once in chroot, set locale:

# locale-gen en_US.UTF-8 
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.

# echo 'LANG="en_US.UTF-8"' > /etc/default/locale 
# dpkg-reconfigure tzdata

Current default time zone: 'Europe/Warsaw'
Local time is now:      Mon Feb 13 20:38:34 CET 2017.
Universal Time is now:  Mon Feb 13 19:38:34 UTC 2017.

Before adding new sources list, mount boot partition and create symlink to the mapper device, so later on system will be properly booted, and symlink to mounted partitions:
# mount /dev/sda1 /boot
# ln -s /proc/self/mounts /etc/mtab 

# ln -s /dev/mapper/root_crypt /dev/root_crypt
Add also a udev rule to create this link each time you boot:
echo 'ENV{DM_NAME}=="root_crypt", SYMLINK+="root_crypt"' > /etc/udev/rules.d/99-local.rules
Edit apt sources list:
# vi /etc/apt/sources.list

deb xenial main universe
deb-src xenial main universe

deb xenial-security main universe
deb-src xenial-security main universe

deb xenial-updates main universe
deb-src xenial-updates main universe

Now it's time to update packages listings and install several of them:
# apt update
Get:1 xenial-security InRelease [102 kB]
Hit:2 xenial InRelease
Get:3 xenial-updates InRelease [102 kB]           
Get:4 xenial/main Sources [868 kB]                           
Get:5 xenial-security/main Sources [59.3 kB] 
Get:6 xenial-security/universe Sources [19.3 kB]                    
Get:7 xenial-security/main amd64 Packages [211 kB]                  
Get:8 xenial-security/main Translation-en [88.5 kB]
Get:9 xenial-security/universe amd64 Packages [73.8 kB]
Get:10 xenial/universe Sources [7728 kB]                
Get:11 xenial-security/universe Translation-en [41.7 kB]
Get:12 xenial/main Translation-en [568 kB]                      
Get:13 xenial/universe amd64 Packages [7532 kB]
Get:14 xenial/universe Translation-en [4354 kB]                                                                                                                    
Get:15 xenial-updates/main Sources [230 kB]                                                                                                                        
Get:16 xenial-updates/universe Sources [130 kB]                                                                                                                    
Get:17 xenial-updates/main amd64 Packages [474 kB]                                                                                                                 
Get:18 xenial-updates/main Translation-en [188 kB]                                                                                                                 
Get:19 xenial-updates/universe amd64 Packages [396 kB]                                                                                                             
Get:20 xenial-updates/universe Translation-en [149 kB]                                                                                                             
Fetched 23.3 MB in 9s (2450 kB/s)                                                                                                                                                                   
Reading package lists... Done
Building dependency tree... Done
72 packages can be upgraded. Run 'apt list --upgradable' to see them.

Install ubuntu minimal package, kernel image, zfs utils and grub. This assumes you use legacy booting:
# apt install --yes ubuntu-minimal ; apt install --yes --no-install-recommends linux-image-generic ; apt install --yes zfs-initramfs zfsutils-linux cryptsetup ; apt install --yes grub-pc

Reading package lists... Done
Building dependency tree... Done
ubuntu-minimal is already the newest version (1.361).
0 upgraded, 0 newly installed, 0 to remove and 72 not upgraded.
Reading package lists... Done
Building dependency tree... Done
The following additional packages will be installed:
  crda iw libnl-3-200 libnl-genl-3-200 linux-firmware linux-image-4.4.0-62-generic linux-image-extra-4.4.0-62-generic wireless-regdb
Suggested packages:
  fdutils linux-doc-4.4.0 | linux-source-4.4.0 linux-tools linux-headers-4.4.0-62-generic
Recommended packages:
  grub-pc | grub-efi-amd64 | grub-efi-ia32 | grub | lilo thermald
The following NEW packages will be installed:
  crda iw libnl-3-200 libnl-genl-3-200 linux-firmware linux-image-4.4.0-62-generic linux-image-extra-4.4.0-62-generic linux-image-generic wireless-regdb
0 upgraded, 9 newly installed, 0 to remove and 72 not upgraded.
Need to get 95.6 MB of archives.
After this operation, 375 MB of additional disk space will be used.
Get:1 xenial-security/main amd64 linux-firmware all 1.157.8 [37.7 MB]
Get:2 xenial/main amd64 libnl-3-200 amd64 3.2.27-1 [52.1 kB]
Get:3 xenial/main amd64 libnl-genl-3-200 amd64 3.2.27-1 [11.2 kB]
Get:4 xenial/main amd64 wireless-regdb all 2015.07.20-1ubuntu1 [9058 B]
Get:5 xenial/main amd64 iw amd64 3.17-1 [63.5 kB]
Get:6 xenial/main amd64 crda amd64 3.13-1 [60.5 kB]
Get:7 xenial-security/main amd64 linux-image-4.4.0-62-generic amd64 4.4.0-62.83 [21.3 MB]                                                                         
Get:8 xenial-security/main amd64 linux-image-extra-4.4.0-62-generic amd64 4.4.0-62.83 [36.3 MB]                                                                   
Get:9 xenial-security/main amd64 linux-image-generic amd64 [2290 B]                                                                                   
Fetched 95.6 MB in 44s (2168 kB/s)  
Output is cut out for brevity.

Set up root password:

# passwd
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
Run grub-probe. If you see output like below, everything is going okay, so far:
# grub-probe /
My favorite editor is vim, so I installed it. Feel free to use any other commandline editor, as graphical enironment is not available yet.

Set bootfs property on the rpool/ROOT/ubuntu filesystem:

# zpool set bootfs=rpool/ROOT rpool
Get UUID of boot partition and add it to /etc/fstab:
 blkid /dev/sda1
/dev/sda1: LABEL="/boot" UUID="2a0b225c-e1bb-43f2-961c-7925678fa42f" TYPE="ext4" PARTUUID="9ee977ba-01"
# cat /etc/fstab
/dev/mapper/root_crypt  /       zfs     defaults        0 0
UUID=2a0b225c-e1bb-43f2-961c-7925678fa42f /boot   auto    defaults        0 0
Get UUID of LUKS partition:
# blkid /dev/sda2
/dev/sda2: UUID="4c43a622-b7e2-4c2c-917b-b5e34a772e87" TYPE="crypto_LUKS" PARTUUID="9ee977ba-
Use this UUID to create /etc/crypttab file:
# cat /etc/crypttab
# blkid /dev/sda2
root_crypt UUID="4c43a622-b7e2-4c2c-917b-b5e34a772e87 none luks,discard
Edit /etc/default/grub and change line:
You can leave the quiet and splash paramerers, but if your new system won't boot, it may be more difficult to troubleshoot. Not much, just little.

You will now need to add proper LUKS device to initramfs. Get UUID of the encrypted partition:

# blkid /dev/sda2
/dev/sda2: UUID="4c43a622-b7e2-4c2c-917b-b5e34a772e87" TYPE="crypto_LUKS" PARTUUID="9ee977ba-02"
and add it to /etc/initramfs-tools/conf.d/cryptroot file:
# cat /etc/initramfs-tools/conf.d/cryptroot
Update the initrd file to include all the new goodness:
#  update-initramfs -c -k all
update-initramfs: Generating /boot/initrd.img-4.4.0-62-generic
Run update-grub:
# update-grub
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.4.0-62-generic
Found initrd image: /boot/initrd.img-4.4.0-62-generic
And finally install new grub:
#  grub-install /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.

This "No eror reported" line is critical. If you see any error message there, go back and troubleshoot.

It is now time to exit chroot, unmount all zfs filesystem and reboot to your new Ubuntu:

# exit
root@ubuntu:~# mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} && zpool export rpool

# reboot

After reboot you'll want to create swap zvol, add a regular user and install graphical environment.

# zfs snapshot rpool/ROOT/ubuntu@firstboot
# zfs create rpool/home/user
# useradd -d /home/user user
# chown -R user:user /home/user
# passwd user
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully

# apt install kubuntu-desktop