virtiofs in NixOS

virtiofs in NixOS

The current way I have landed on to run my NixOS environments is using VM’s on a Proxmox host. One thing that I have not been happy about in this setup is all data is on virtual disks that need to be partitioned, but I recently discovered the Proxmox now supports virtiofs!

Proxmox Virtual Environment

So now I can create a ZFS dataset zfs create tank/mydataset, add it as a Directory Mapping in the PVE Datacenter object pvesh create /cluster/mapping/dir --id mymapping --map node=node1,path=/tank/mydataset and add it as a virtiofs device in the hardware of the VM qm set 100 -virtiofs0 dirid=mymapping. That’s it from PVE’s side.

NixOS

Inside the NixOS config will need the standard filesystems mount that your hardware config should pick up (after mounting mount -t virtiofs mymapping /path/to/mount).

filesystems."/path/to/mount" = {
  device = "mymapping";
  fsType = "virtiofs";
};

This is good except NixOS will mount the following mount points in the initial ram disk /, /nix, /nix/store, /var, /var/log, /var/lib, /var/lib/nixos, /etc and /usr. If the mount is one of these locations or will use fileSystems.<name>.neededForBoot = true; then virtiofs needs to be usable in the initrd, thankfully this is just adding the kernel module in the initrd boot.initrd.kernelModules = [ "virtiofs" ];.

Overall I think this is a much better path forward as migrating data and restoring from backups is much easier when I am not worrying about virtual block devices and this should allow ZFS to do more of what it does best with the data in the dataset.

TODO

PVE has options for exposing ACL’s and extended attributes which I needs to look at a little bit further.