Add support for bootc

[cgwalters]

So a while ago, we added the ostreecontainer kickstart verb. The advantage of this is that it works in exactly the same way/shape as the ostreesetup did.

However, the bigger picture direction I'd like to go is actually towards bootc install - where a container image can install itself.

The relevance to Anaconda here is that we'd support something like this:

# Basic partitioning
clearpart --all --initlabel --disklabel=gpt
reqpart
part / --grow # --fstype xfs (or could come from container)

bootc quay.io/fedora/fedora-bootc:40

Goals:

• Be able to define the filesystem type in the container with the bootc install config
• Potentially: Embed full or partial kickstart in container image (some users may want to even do machine-specific derived containers); in this path we could theoretically just have inst.bootc=quay.io/exampleos/blah
• More important: Things around tracker: logically bound app images bootc-dev/bootc#128 - also relates to anaconda flatpak bits

And here, this uses bootc install-to-filesystem; Anaconda takes care of initializing the filesystem tree, but otherwise just executes the container to do the install.

Crucially in the bootc install flow, it's actually the container that installs the bootloader to the target disk, not Anaconda.

Other higher level goals

Implementation goals

Implementing this is mostly:

• Add podman to the ISO
• Kickstart verb glue
• bootc expands to podman run --privileged --pid=host --net=none --security-opt label=type:unconfined_t <image> per the doc

Of course, one thing we could do is add generalized support for pulling and executing a container...basically do an install like this:

%post --no-chroot
podman run ...

But, that has a few downsides around ergonomics, fragility, etc.

[cgwalters]

xref https://gitlab.com/CentOS/cloud/sagano which would use this

[akdev1l]

Hi everyone,

I work on the installer ISOs for theUniversal Blue project and we’ve had issues on creating offline installer images for ostree container native systems due to limitations in the ostreecontainer verb. I would like to make sure we can get support for offline installation with the new bootc capabilities.

The current implementation of the ostreecontainer verb does not allow us to set the —target-imgref correctly so we’ve had to resort to hacks (rebasing on first boot) to work around this limitation (ref: ostreedev/ostree-rs-ext#512).

I’m not sure what this would mean for the bootc implementation. I imagine we would put the container archive in the ISO and pull from there similar to how the ostree repo is embedded in existing Silverblue images.

As a side note I was going to try to craft a patch to add this support to the ostreecontainer verb but it’s clear bootc switch the way of the future so not sure if such patch would be accepted - if so I will create a separate issue for that.

[cgwalters]

I'm sure we could get a patch merged that added better support for --target-imgref to the ostreecontainer verb!

But yes, would like to move in the bootc direction overall.

[jkonecny12]

I was already asked for support of --target-imgref, however, we are right now blocked by Anaconda doesn't support --imgref but only the split version (which I think is correct), however, that the split version doesn't exists for --target-imgref so without ostree-rs-ext support this doesn't fit well.
@achilleas-k

However, I see focus on the bootc would change this.

[cgwalters]

BTW an important detail in this is that bootc will own installing the bootloader (via bootupd by default) which also means that using bootc should automatically turn off anaconda's bootloader installation.

[fabiendupont]

Would there be other options than --to-filesystem or could it be an implicit destination?

[cgwalters]

Sorry yes this needs to be fleshed out more. Here's my thoughts: We want to support two modes:

• Anaconda defines the partitioning setup: If the kickstart has any partitioning at all this automatically switches the bootc install invocation to use install-to-filesystem
• The container defines the partitioning: This is IMO a cool part of this; say I want to install a RHEL system but I want to use fscrypt/fsverity which is only supported by ext4 today in RHEL9. In my container build, I just add a drop-in /usr/lib/bootc/install/50-mycustom.conf that sets root-fs-type = "ext4". A big point of this is that the "source of truth" becomes the container image, not the kickstart.

And what may be even cooler I think is a third model:

• Anaconda knows to look for /usr/lib/bootc/install-anaconda/inst.ks which contains arbitrary kickstart setup

[jkonecny12]

Hi @cgwalters thanks for all the information. I just want to clarify I understand it correctly.

So it means that we need to support 3 modes?

• with --to-filesystem where we do the partitioning and run bootc to install the bootloader stuff
• without --to-filesystem where we can't do partitioning and bootc call will handle everything
• where we would look for the kickstart inside the container which would be basically merging of the kickstart files

All these will skip the bootloader installation by Anaconda.

Am I correct @cgwalters ?

[cgwalters]

That's what I'm thinking, but of course this is a collaborative thing. I'd say probably the most important short term is supporting the first one because it will fix bootloader updates in general and help align things.

[jlebon]

Yes, one thing that worries me about the Anaconda flow in RHCOS/FCOS is around partition tables and filesystems. I'd like to see a mode where one can use kickstart partitioning to define additional partitions only (and maybe just extend a predefined rootfs partition) and not have to redefine everything to get that. I'm not familiar with existing kickstart merging semantics, and whether that'd be possible already just by how kickstarts work. This would make it work more closely to how Ignition does.

One seemingly minor thing related to this is that in CoreOS we've grown quite used to the current partition numbers used, and I think it'd cause confusion if in the Anaconda path, partitions can be numbered differently from our metal images.

Anaconda knows to look for /usr/lib/bootc/install-anaconda/inst.ks which contains arbitrary kickstart setup

Yeah, agreed. Then we'd be able to have a %post which does some validation before finishing up the install.

We also need to be able to easily tell systems which were installed by Anaconda from those that weren't. E.g. adding a stamp file at the root of the rootfs. So that any remediation script can filter that pool in/out using ConditionPathExists=.

That could also be done as a CoreOS-specific %post I think, though it might make sense to have that be generic over all bootc installs.

[cgwalters]

Yes, that also strongly relates to the concept of embedding desired partitioning directly inside the OS image, which goes into bootc-dev/bootc#448 (among others).

I need to do a deeper dive on repart but I think it probably makes sense to do a push on it.

I'm not familiar with existing kickstart merging semantics, and whether that'd be possible already just by how kickstarts work.

That said we should also support embedding kickstart defaults in the target container image too, and if we do that then any per-machine kickstarts would include that automatically.

Though for the "don't modify the image" case we'd need a commandline like inst.ks.nobase or something to skip including the default.

[cgwalters]

One thing I hope we can fix as we do this is e.g. ostreedev/ostree#3193 - the bootc install flow just uses kernel arguments for the rootfs and I think this is ultimately the right thing.

[cgwalters]

So recently we've done quite a lot in bootc (layered on top of ostree) and in the current situation none of those features will apply.

For example the kernel argument support.

But even more recently we're switching bootc to support logically bound containers

[cgwalters]

To xref I am working on bootc-dev/bootc#915 which will end up working around this.

[cgwalters]

Thinking about this more. First let's assume we have podman in the Anaconda ISO.

A core problem to solve is pulling the container image.

Let's assume that this phase happens after the target filesystem is created. One option I think would be viable is to always create a temporary swap file on that filesystem. Alternatively, pull that image to a temporary place on the target FS (e.g. podman run --root=/mnt/sysimage/var/tmp/anaconda-podman <image>).

The basic idea either way is to avoid the "running from RAM" penalty - and also in the bootc case, we could in theory optimize this to avoid writing the OS files twice by using reflinks (also xref containers/storage#1849 ).

This would greatly help in the implementation of a bootc install to-filesystem backend for Anaconda.

However one thing that would be significantly cleaned up here is if bootc itself offered a way to spawn code running against the target root after bootc install to-filesystem is complete. Right now Anaconda has a bunch of hand-crafted container stuff and ideally the %post verb when executing against a bootc target had a proper setup (w/composefs in particular).

[jkonecny12]

First drafts of the PRs:

• Add support for new bootc kickstart command pykickstart/pykickstart#536
• Add bootc kickstart command support #6298

As said first draft so a lot will change yet.

[champtar]

Today I modify an existing ISO using mkksiso to embeded:

• product.img (anaconda fix backport + .buildstamp + etc/anaconda/product.d/)
• a kickstart
• an ostree repo

This works for both offline installs (ostreesetup --url="file:///run/install/repo/ostree/repo") and automated installs (just extract the iso on a dumb http server and modify ostreesetup line) and doesn't require a registry.
(An important point is that we do a lot of automated deployments in infrastructure we don't own, so depending only on a dumb http is important.)

Now switching to ostreecontainer (and at some point bootc keyword), containers can only be pulled from proper registries, not dumb http servers, leaving us to choose from a list of workaround to be able to do both offline and http installs:

1. embed the container at the root of the ISO as ociarchive, for http installs use a %pre section for the download when doing http install:
   when doing an offline install with a slow virtual CD, as tar archive are not seek-able and in ociarchive the index.json is at the end after the blobs, I'm wondering how many times content is read from disk
2. embed the container as oci at the root of the ISO:
   this requires a smart script in %pre to pull the content for http install
3. embed the container as oci in the install.img (stage2):
   not sure if the squashfs is fully loaded in ram before install continue in the iso case (slow virtual CD use case), this might be the easiest
4. embed containers-storage in the install.img (stage2): allow to deduplicate data between the OS and Anaconda Add support for bootc #5197 (reply in thread)
5. oci inside update.img: cpio ustar format should not have limitations: Add support for bootc #5197 (reply in thread)

I tried to look at what is already done:

• Fedora Silverblue 42: ostree bare repo in stage2 (https://pagure.io/fedora-lorax-templates/blob/main/f/ostree-based-installer/lorax-configure-repo.tmpl)
• Universal Blue: container pre pulled by podman in the stage2 (https://github.com/ublue-os/titanoboa/blob/626369083b5afda7edceb9d5509b2a5b23b2f8af/Justfile#L160)
• bootc-image-builder: oci at the root of the ISO

I'm wondering what future bootc keyword users plan to do when building ISO for offline usage

[cgwalters]

My preference actually is that we use a containers-storage: instance, as it will aid in deduplication with the target content. However, doing this will currently introduce bootc-dev/bootc#1197

[jikortus]

@champtar Not sure if it could be helpful to you anyhow, but you might also consider experimenting with using updates image to supply the container image in the installer environment (to which you can point via ostreecontainer --url="file:///...") or mkksiso tool to bake the image into the boot ISO as a workaround. It's just an idea and I haven't tried that myself, so take this with a grain of salt, please :-).

[champtar]

@jikortus ostreecontainer --url="file:///..." is a bit misleading I think, it calls ostree container image deploy --image=URL --transport=TRANSPORT ... so you can access local files with --url=/mycontainerdir --transport=oci but you can't download ociarchive automatically

I've initially excluded updates.img because it's cpio+gzip and I remember stupid limitations with cpio, but maybe choosing the ustar format would work. updates.img is not lazy loaded so it would delay the start of the install, but i can't have it all

I'm already using mkksiso (see my first line :) ) and it has the advantage of being super fast

[jikortus]

Sorry, not sure how I could have missed that mention of mkksiso :-). You're right that the update.img is not limited only to cpio, ustar format works just fine as well.

[champtar]

Thinking about it more, to rebuilding the stage2 you either need root + SELinux support, or to launch a VM, which is not super CI friendly