Regression: crun-krun 1.23 fails to start when using rootless podman on AMD

[iczero]

On Fedora 42, after upgrade of crun-krun from 1.22-1.fc42 to 1.23.1-1.fc42, rootless podman can no longer start containers with --runtime krun. Still works if podman is running in rootful mode. Neither podman nor conmon were updated in this transaction.

Reproduction:

$ podman run --runtime krun --rm -it fedora:42 bash
[libcrun:krun]: `/dev/kvm` unavailable
(exit code 1)

I think it may be associated with this change: ac297b7. /dev/kvm does indeed exist and is accessible after running podman unshare.

crun version 1.23.1
commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL

podman version 5.5.2
conmon version 2.1.13

$ podman unshare
# ls -lah /dev/kvm
crw-rw-rw-. 1 nobody nobody 10, 232 Aug 23 21:44 /dev/kvm
# stat /dev/kvm
  File: /dev/kvm
  Size: 0               Blocks: 0          IO Block: 4096   character special file
Device: 0,6     Inode: 1863        Links: 1     Device type: 10,232
Access: (0666/crw-rw-rw-)  Uid: (65534/  nobody)   Gid: (65534/  nobody)
Context: system_u:object_r:kvm_device_t:s0
Access: 2025-08-23 21:44:38.542735460 -0700
Modify: 2025-08-23 21:44:38.542735460 -0700
Change: 2025-08-23 21:44:38.542735460 -0700
 Birth: 2025-07-12 23:10:20.260633340 -0700

[giuseppe]

@tylerfanelli PTAL

[tylerfanelli]

I've cloned the 1.23.1 branch and have tried the example, but I'm unable to reproduce the bug. @iczero Can you supply a bit more info on your environment so I can reproduce?

[iczero]

@tylerfanelli Looks like this issue is AMD specific somehow. I have reproduced it on two AMD systems (one with /dev/sev, one without SEV enabled). I have also tried on two Intel systems but neither had this issue. Both AMD systems were using EPYC Genoa CPUs (EPYC 9554 (+SEV) and EPYC 9354 (no SEV)). I do not believe SEV was in use by krun on the SEV-enabled system. Both AMD servers were bare metal (not nested virtualization), one Intel server was using nested virtualization and the other wasn't. Root filesystem is btrfs on all systems, but one of each AMD and Intel server were using the btrfs graph driver (the other default, overlay). Do you need any other system information?

[tylerfanelli]

So just to be clear, you are not trying to run with libkrun-sev? Coincidentally, I tested on a EPYC Milan machine myself, and can't reproduce. However, I did not try with libkrun-sev.

[iczero]

I can confirm that I am not trying to run with SEV. Additionally, the error on both AMD systems is the same, and only one of them has SEV enabled. Are you testing on Fedora 42?

I will try to further debug this on my system when I get a chance.

[tylerfanelli]

Fedora 42, but I cloned crun from upstream.

[iczero]

I will try that as well. I was using the crun-krun package from Fedora repositories.

[tylerfanelli]

Try with upstream crun:

$ git clone --branch 1.23.1 https://github.com/containers/crun.git
$ cd crun

# Make new version
$ ./autogen.sh && ./configure --with-libkrun
$ make
$ cp crun /bin/krun

You'll likely want to save the krun binary already present in /bin before doing this (and switch it back afterwards).

[iczero]

Built from source but got same error.

$ sudo cp -v crun /usr/bin/krun
'crun' -> '/usr/bin/krun'

$ podman run --runtime /usr/bin/krun --rm -it fedora:42 bash
[libcrun:krun]: `/dev/kvm` unavailable

$ ./crun --version
crun version 1.23.1
commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +LIBKRUN +YAJL

$ /usr/bin/crun --version
crun version 1.23.1
commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL

Manually running a container with krun works (using krun create then krun start). This works both inside and outside of podman unshare. The process name is [libcrun:krun] so it seems to be using KVM. Maybe this is a podman problem?

[iczero]

In libkrun_modify_oci_configuration when called from podman, def->linux->resources->devices is null, which prevents the rest of the function from running.

Aug 27 02:52:59 desktop loving_kirch[1962354]: debug(src/libcrun/handlers/krun.c:715 libkrun_modify_oci_configuration): def->linux = 0x37ca4420
Aug 27 02:52:59 desktop loving_kirch[1962354]: debug(src/libcrun/handlers/krun.c:717 libkrun_modify_oci_configuration): def->linux->resources = 0x37ca69b0
Aug 27 02:52:59 desktop loving_kirch[1962354]: debug(src/libcrun/handlers/krun.c:719 libkrun_modify_oci_configuration): def->linux->resources->devices = (nil)

.linux.resources.devices is not present in OCI config generated by podman when using the command above.

On working systems, we get

debug(src/libcrun/handlers/krun.c:371 libkrun_exec): kconf->has_kvm: 111

On non-working systems, we get kconf->has_kvm: 0 (above). I assume 111 is not what true is supposed to be.

This patch fixes the problem:

diff --git a/src/libcrun/handlers/krun.c b/src/libcrun/handlers/krun.c
index 5c6d2c82..65a1a58b 100644
--- a/src/libcrun/handlers/krun.c
+++ b/src/libcrun/handlers/krun.c
@@ -720,7 +721,7 @@ libkrun_modify_oci_configuration (void *cookie arg_unused, libcrun_context_t *co
 
   if (def->linux == NULL || def->linux->resources == NULL
       || def->linux->resources->devices == NULL)
-    return 0;
+    goto done;
 
   /* Always allow the /dev/kvm device.  */
 
@@ -773,6 +774,7 @@ libkrun_modify_oci_configuration (void *cookie arg_unused, libcrun_context_t *co
       def->linux->resources->devices_len++;
     }
 
+done:
   kconf->has_kvm = has_kvm;
   kconf->has_nitro = has_nitro;
 
   return 0;

but I assume that /dev/kvm should be checked even if .linux.resources.devices is missing, which this patch does not do. I suppose this would explain why it only fails on one specific platform.

[tylerfanelli]

I think that's a more of a workaround. The issue is that def->linux->resources->devices is NULL. @giuseppe Do you know why this could be? Strange that it's only on AMD systems, yet I'm not able to reproduce on my own EPYC system.

[iczero]

.linux.resources.devices is also missing on working systems. The has_kvm struct field remains uninitialized when .linux.resources.devices is missing. The reason why it works on most systems appears to be that has_kvm (which reads uninitialized memory) just happens to have a value that isn't 0.

[devwardiman]

Have the same issue on Fedora 42. Mount devpts error after upgrading to crun 1.23.1 — containers created with older crun fail to start

Summary

After upgrading Fedora and crun to version 1.23.1, containers that were previously created with crun 1.20 fail to start. The error message is:
unable to start container "b239ffb26f15cec6b8962ce4e87853355fc1b30d778def2042ae846dcabba4b7": crun: mount devpts to dev/pts: Invalid argument

Downgrading crun back to 1.20 resolves the issue, and containers start normally.

Steps to Reproduce

1. Create a container using crun 1.20 (Fedora 42)
2. Upgrade crun 1.23.1
3. Attempt to start the container
4. Observe the mount error

Environment

• OS: Fedora 42
• crun version: 1.23.1 (broken), 1.20 (working)
• Container engine: Podman
• Rootless: Yes
• Filesystem: Btrfs
• Kernel: 6.15.10-200.fc42.x86_64

Additional Notes

• The container was created and working fine before the upgrade
• dnf downgrade crun to version 1.20 works without issue

Please advise if this is a known regression or if additional logs are needed.

[iczero]

@devwardiman this is not the same issue. You are not using krun. This issue is specific to krun code paths. Check dmesg for mount errors related to devpts.