Disk out of space

[mpeterson]

I am trying to upgrade my nodes and I'm getting out of space on the disk.

Error: failed to install bootloader: error copying reader -> /boot/B/initramfs.xz: write /boot/B/initramfs.xz: no space left on device

A few days back I also had one of the nodes draining pods due to out of space.

The nodes have a minimum 32GB of disk, with one that I tested raising it to 42GB to see if it helps (it did not).

How can I investigate this?

Attaching support.zip if it helps
support.zip

[smira]

Talos boot partition should be 1 GB, I wonder do you use lots of system extensions?

You can try mounting it manually from the privileged pod and exploring it

[smira]

I just checked that the UKI size for this set of extensions is almost half a gigabyte, so it makes sense that Talos runs out of disk space.

I will look into increasing default boot partition size for new installations.

[mpeterson]

For my particular problem:

• anything I can do to fix this on a running node?
• should I wipe the node and provision it again? if yes, what changes you recommend?

For new installations, since with v1.10 you already know the extensions at installation time. Couldn't the boot partition size be decided dynamically?

[smira]

Dynamic boot size is probably possible, but it doesn't address the issue of upgrading to e.g. a bigger set of extensions.

We need to look into this from all angles, but for now we can try bumping boot size.

[smira]

Sorry, to answer your question - unfortunately I don't have a good workaround, except for upgrading to a version with the biggest extensions excluded (nvidia) and then upgrading once again to a version with extensions. As Talos keeps two versions of boot assets (previous and new), this should work. It will be always "small" and "big" in this sequence.

[mpeterson]

small, big sequence worked. Until now I didn't need this, I guess now the extensions became bigger? At least as a workaround, even though it's quite uncomfortable, it works.