Single disk ZFS install options?

[otopetrik]

Hello,

Looking at the volume management epic #8367 I am still not certain, if or how it will be possible to use Talos on a single disk machine with ZFS.

The ideal goal would be to have something like zpool 'rpool' with following top-level datasets:

• /rpool/STATE
• /rpool/EPHEMERAL
• /rpool/USER (or some other reserved dataset name, where the subtree is never deleted or modified by Talos)

It looks like that there are multiple ways to use ZFS in Kubernetes:

openebs/zfs-localpv

Provides local-only zfs dataset as volume, with tunable zfs options.
Should be useful for things that handle replication themselves and can use tunable zfs options, e.g. PostgreSQL using cloudnative-pg.

piraeusdatastore/piraeus-operator

LINSTOR seems to have ZFS support, and provides replicated volume, but backed by formated ZVOL (documentation https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-kubernetes-file-system).
Should be useful for things that do not handle data replication themselves.

So there seem to be advantages in Talos having direct access to ZFS pool (instead of Talos VM on ZFS zvol in Proxmox).

Why would single disk ZFS mode be useful?

Many older, cheaply available, and still useful mini-PCs (Wyse 7020, Dell 3040M, ...) support only single disk and only 16 GB RAM.
(Things like RaspberryPI + NVMe hat also fall to similar category)

Single disk limitation makes running Talos on bare-metal risky to user data (though it looks like the defaults have changed).

I am running Talos in Proxmox VM on such machine, it works, but there are some limits:

• ZFS cache on Proxmox hosts reduces the memory available to Talos guest
• using ZFS inside Talos VM is not really possible (second virtual disk would be backed by ZFS ZVOL on the host. ZFS on ZFS would be slower, and it would further limit available memory due to another ZFS cache - this time inside Talos VM)
• no access to integrated gpu

Running Talos bare-metal on ZFS single disk machine would allow:

• access to full 16 GB (no need to reserve memory to ZFS cache on the Proxmox host)
• experiments with storage performance (zfs options vs postgresql options)
• reasonable virtualization speed in containers (when KVM is needed CI runner containers, or for KubeVirt)
• experiments with integrated gpu inside containers

Question:

It looks like the handling of volumes on system disk is tied to GPT partitions, i.e. volume cannot be a ZFS dataset nor LVM thinpool LV.
Therefore it will not be possible to have ZFS pool over the whole (except EFI/BIOS/BOOT) boot disk, have EPHEMERAL volume as ZFS dataset, and share free space between EPHEMERAL and user data.

However, it should be possible to limit the size of EPHEMERAL partition on install, and create ZFS pool on a new partition in the remaining space. Disk space will be permanetly split between EPHEMERAL and ZFS pool. The partition with ZFS pool will be never deleted by Talos.

Is that correct?

Thanks!

[smira]

At the moment, Talos has no native support for ZFS volumes. There's a community extension which might or might not work.

Talos no longer forces any wipe of data on upgrades, so it's safe to store data in EPHEMERAL partition.

What is available at the moment:

• you can shrink EPHEMERAL partition, or to be precise, limits its size (by default it spans the whole disk)
• coming in 1.10: you can create user volumes out of remaining disk space (or using other disks if you have more than one), and use them as xfs/ext4 partitions.

ZFS/LVM support is something we're looking into, but we don't have a commitment for it yet.