Installing ZFS on Root in a Running VPS

This guide explains how to install ZFS as the root filesystem of a VPS that contains an existing Linux installation without causing data loss. The VPS will have a small EXT4 boot partition for GRUB2 with the kernel and initrd, and a large ZFS pool mounted on root with compression enabled. Most commands will be issued from a live CD image and a workaround will be shown if the VPS does not support mounting ISO images.

Index:

1. Back up the data

2. Boot a live OS image

3. Install ZFS on the live OS environment

4. Partition the VPS disk

5. Install ZFS on the VPS disk

6. Restore the backup to the new ZFS root filesystem

7. Install GRUB2

8. Boot into the ZFS root pool

Step 1: Back up the data

If you have a second VPS or home computer with enough space to store the backup data, from within that computer run:

# rsync -aqrxz root@target:/ backup.d

where target is the hostname of the VPS being converted. This will copy the entirety of the root filesystem over SSH to the backup machine, and restoring this backup onto the target VPS later on will be straightforward. If you need to compress the backup for lack of space, you can alternatively create a compressed TGZ archive:

# ssh root@target "tar -cf - --acls --xattrs --one-file-system --absolute-names /" | gzip -1 >backup.tgz

However, when restoring a root filesystem from a TAR archive, it will be necessary to recreate all symlinks whose targets have absolute pathnames, because TAR converts their targets to relative pathnames, so using rsync is the simpler alternative. How to properly restore the TAR archive will be shown later.

Step 2: Boot a live OS image

The conversion to ZFS will be done from a live CD operating system. Depending on how capable your VPS control panel is, it may already have a live OS image ready to boot. Alternatively, you can download a Debian live ISO from the Debian mirror and upload it to your VPS control panel, then boot the ISO from there; this will work fine even if your VPS runs a different distribution. Be careful to use a live OS ISO image, not an installation ISO, because installation images lack apt and are unable to install the necessary software onto the live environment. If your VPS control panel supports booting from a user-supplied ISO image, skip to Step 3, otherwise you can still boot the downloaded ISO by converting a swap partition, if present, into EXT4 and storing the ISO in that reclaimed storage space, then GRUB2 may be configured to boot from that ISO. This is done as follows:

# fdisk --list /dev/vda
Disk /dev/vda: 35 GiB, 37580963840 bytes, 73400320 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x495b5ce4

Device Boot Start End Sectors Size Id Type
/dev/vda1 * 2048 69203583 69201536 33G 83 Linux
/dev/vda2 69203584 73397887 4194304 2G 82 Linux swap

This VPS has a 2GB swap partition in /dev/vda2 which is large enough to store the ISO. Create the EXT4 filesystem, mount it and download the ISO on it:

# swapoff /dev/vda2     # disable swap partition
# mkfs.ext4 /dev/vda2   # create EXT4 filesystem
# mount /dev/vda2 /mnt  # mount the reclaimed space
# mkdir /mnt/images/
# wget -O /mnt/images/image_file.iso <image_URL>

then instruct GRUB2 to boot off this ISO; edit /etc/grub.d/40_custom as follows:

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.

menuentry "Live ISO" {
  set ISOFile="/images/image_file.iso"
  loopback loop (hd0,2)$ISOFile
  linux (loop)/live/vmlinuz boot=live findiso=$ISOFile
  initrd (loop)/live/initrd.img
}

and apply the configuration with

update-grub2

When you reboot the VPS, the GRUB2 menu will offer the option to boot off the live image. Proceed from there using the VNC rescue console offered by the provider's control panel.

Step 3: Install ZFS on the live OS environment

Boot the live OS image and access its terminal. Then install the ZFS packages on the live OS environment:

# echo deb http://deb.debian.org/debian stretch contrib >> /etc/apt/sources.list
# apt update
# apt install zfs-dkms
# modprobe zfs

Now the live OS has the ZFS kernel module and the ZFS utilities required to format your VPS root filesystem.

Step 4: Partition the VPS disk

There are two alternative scenarios for partitioning the VPS disk:

1. If you stored the live ISO in /dev/vda2 you may not alter that partition now because the live system depends on it, and no partitioning whatsoever will be done to the VPS disk. ZFS will be formatted on the existing /dev/vda2 partition, and /dev/vda1 will remain as is and later be repurposed as your permanent boot partition for storing the kernel and initrd. Skip to Step 5.
2. If your live environment does not depend on an ISO file stored on the VPS disk, repartition the disk by creating one small boot partition for GRUB2, and one large partition for the ZFS root that extends over all remaining disk space. Execute fdisk to create the two following partitions:

   # fdisk /dev/vda
   
   Command (m for help): p
   
   Disk /dev/vda: 35 GiB, 37580963840 bytes, 73400320 sectors
   Units: sectors of 1 * 512 = 512 bytes
   Sector size (logical/physical): 512 bytes / 512 bytes
   I/O size (minimum/optimal): 512 bytes / 512 bytes
   Disklabel type: dos
   Disk identifier: 0x495b5ce4
   
   Device     Boot    Start      End  Sectors Size Id Type
   /dev/vda1           2048 69203583 69201536  33G bf Solaris
   /dev/vda2  *    69203584 73397887  4194304   2G 83 Linux
   

   Create a primary boot partition 100MB wide and mark it as bootable with an asterisk (*), with a partition ID = 83 (Linux). It will be formatted as EXT4 for compatibility with GRUB2, and it will contain the boot data. Then create another primary partition for ZFS, with ID = bf (Solaris). Write the changes to disk and close fdisk. Now that the disk is properly partitioned, you may format the boot partition:

   # mkfs.ext4 /dev/vda2
   

Step 5: Install ZFS on the VPS disk

Format /dev/vda1 as a ZFS volume:

# zpool create -o ashift=12 \
  -O acltype=posixacl -O canmount=on -O compression=zstd \
  -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
  -O mountpoint=/ -R /mnt \
  rpool /dev/vda1

This set of options is recommended by the ZFS documentation for the highest compatibility with the newest and future disks. The option compression=zstd enables the ZSTD compression algorithm which has a higher compression ratio than the default LZ4, although you may use compression=lz4 to reduce the CPU usage if your VPS has high disk I/O. The pool is named rpool which is the most commonly used name for root pools. When this command completes, rpool will be mounted on the /mnt directory of the live environment, as specified by the -R /mnt option above, but it will be automatically mounted on / at every subsequent boot. Mounting it on /mnt is necessary to restore your backup onto it from the live environment.

Step 6: Restore the backup to the new ZFS root filesystem

Install the OpenSSH server on the live environment to receive the backup data from the backup VPS:

# apt install openssh-server

For simplicity, enable PermitRootLogin yes in /etc/ssh/sshd_config of the live environment, then set a secure root password and restart OpenSSH:

# passwd
# service restart ssh

and initiate the restoration of the backup data from the backup VPS, which will populate the ZFS root pool mounted on /mnt:

## from the backup VPS
# rsync -arxz backup.d/* root@target:/mnt/

or if you used TAR to create a backup archive, restore it with:

## from the backup VPS
# cat backup.tgz | ssh root@target "tar -C /mnt -zaxf -"

however as I wrote in Step 1, upon extraction, TAR converts the target path of absolute symlinks into relative paths; these symlinks can all be properly restored as follows. Note that this restoration is not necessary if you didn't use TAR to back up the root file system. From the backup VPS, list all symlinks contained in the archive:

## from the backup VPS
# tar -tvf backup.tgz | egrep -- '->' >list_of_symlinks.txt

then copy the list to the target VPS into /mnt, which now contains the root filesystem repopulated from the backup:

## from the backup VPS
# scp list_of_symlinks.txt root@target:/mnt/root/

From the target VPS, chroot into /mnt and execute this Bash script to restore all absolute symlinks:

## from the chroot environment
while read -r row; do
  src=$(echo "$row" | grep -Po '(?<=[0-9]{2}:[0-9]{2} )[^ ]+')
  tgt=$(echo "$row" | grep -Po '(?<= -> ).+')

  echo "linking: $src -> $tgt"
  ln -fs "$tgt" "$src"
done </root/list_of_symlinks.txt

Your root filesystem is now fully restored from the backup and may be booted, but you need a functioning bootloader first.

Step 7: Install GRUB2

If you are inside the chroot, exit it and return to the live environment. Then bind the virtual filesystems to /mnt and mount the boot partition:

# mount --bind /dev  /mnt/dev
# mount --bind /proc /mnt/proc
# mount --bind /sys  /mnt/sys
# mount /dev/vda2 /mnt/boot

You may choose to copy any pre-existing content from /mnt/boot/ over to the new boot partition. Chroot into the root pool, and from within it, install the ZFS initramfs hook, the ZFS kernel module, and the ZFS userspace tools required to generate the initrd:

# chroot /mnt
# apt install pkg-dev linux-headers-amd64 linux-image-amd64
# apt install zfs-dkms zfs-initramfs

then install the GRUB2 package:

# apt install grub-pc
# update-initramfs -u -k all

edit /etc/default/grub to set the ZFS pool as the root device, by modifying the GRUB_CMDLINE_LINUX variable:

## this is /etc/default/grub
GRUB_CMDLINE_LINUX="root=ZFS=rpool"

then apply the configuration:

# update-grub

and finally install GRUB2 to the MBR:

# grub-install /dev/vda

Step 8: Boot into the ZFS root pool

The system is ready, fully restored from the backup, and may be booted. Exit the chroot and umount all virtual filesystems, then export the ZFS root pool:

# exit # exit the rpool chroot
# umount /mnt/dev
# umount /mnt/proc
# umount /mnt/sys
# umount /mnt/boot
# umount /mnt
# zpool export -a

You can now remove the ISO image from the VPS control panel, if you used this option, or delete the ISO from the hard drive, and reboot the system into the ZFS root pool:

# reboot

The conversion of the VPS root filesystem from EXT4 to ZFS is complete, and the VPS will boot into your old installed system as usual.