runc tests fail with criu-dev

[kolyshkin]

For the last few years, runc repo have a CI job to test runc against criu-dev branch, mostly to catch some regressions early.

We are seeing failures recently (this week), alas without too much diagnostics.

For example (from here, look for not ok 32):

2025-10-16T01:03:57.5662977Z not ok 32 checkpoint and restore
2025-10-16T01:03:57.5673614Z # (from function `simple_cr' in file tests/integration/checkpoint.bats, line 131,
2025-10-16T01:03:57.5675071Z #  in test file tests/integration/checkpoint.bats, line 192)
2025-10-16T01:03:57.5698743Z #   `simple_cr' failed
2025-10-16T01:03:57.5712389Z # runc spec (status=0)
2025-10-16T01:03:57.5713588Z #
2025-10-16T01:03:57.5714281Z # runc run -d --console-socket /tmp/bats-run-kiFEqC/runc.CtdpiC/tty/sock test_busybox (status=0)
2025-10-16T01:03:57.5715346Z #
2025-10-16T01:03:57.5715677Z # runc state test_busybox (status=0)
2025-10-16T01:03:57.5716304Z # {
2025-10-16T01:03:57.5716613Z #   "ociVersion": "1.2.1+dev",
2025-10-16T01:03:57.5716999Z #   "id": "test_busybox",
2025-10-16T01:03:57.5717360Z #   "pid": 13992,
2025-10-16T01:03:57.5717695Z #   "status": "running",
2025-10-16T01:03:57.5718167Z #   "bundle": "/tmp/bats-run-kiFEqC/runc.CtdpiC/bundle",
2025-10-16T01:03:57.5718804Z #   "rootfs": "/tmp/bats-run-kiFEqC/runc.CtdpiC/bundle/rootfs",
2025-10-16T01:03:57.5719398Z #   "created": "2025-10-16T01:03:57.172483254Z",
2025-10-16T01:03:57.5719845Z #   "owner": ""
2025-10-16T01:03:57.5720295Z # }
2025-10-16T01:03:57.5720817Z # runc checkpoint --work-path ./work-dir test_busybox (status=1)
2025-10-16T01:03:57.5721697Z # time="2025-10-16T01:03:57Z" level=warning msg="open work-dir/dump.log: no such file or directory"
2025-10-16T01:03:57.5722327Z # time="2025-10-16T01:03:57Z" level=error msg="read unixpacket @->@: EOF"
2025-10-16T01:03:57.5722676Z # --- teardown ---

It seems that criu swrk just dies under (without even leaving dump.log).

Tried to reproduce it locally with no luck so far.

[avagin]

it could be triggered by #2728

[rst0git]

It seems that criu swrk just dies under (without even leaving dump.log).

It usually happens when CRIU fails before logging has been initialized. For example, adding an invalid option to default.conf or runc.conf would cause similar error.

[rst0git]

@kolyshkin I am also not able to replicate this error locally. I've tried running the tests in my runc fork and they seem to pass.

I've also included the commits from opencontainers/runc#4928 and it still works:
https://github.com/rst0git/runc/actions/runs/18573059324/job/52951523117

[kolyshkin]

OK, I have removed caches used by GHA CI for this job, and it now succeeds. Must be some kind of a glitch. Let's close this for now.

[lifubang]

Must be some kind of a glitch.

Yes, @kolyshkin @rst0git could you please reopen this issue? It has reoccurred.

The problem appears to be isolated to the following environment: ubuntu-24.04 with golang 1.25.x and criu-dev.

Original log output:

# runc checkpoint --work-path ./work-dir test_busybox (status=1)
# time="2025-10-26T01:24:39Z" level=warning msg="open work-dir/dump.log: no such file or directory"
# time="2025-10-26T01:24:39Z" level=error msg="read unixpacket @->@: EOF"

[kolyshkin]

This is still happening and I'm not quite sure how to debug it.

[kolyshkin]

E.g. here https://github.com/opencontainers/runc/actions/runs/18824557338/job/53705159095?pr=4946

[kolyshkin]

Bisected in opencontainers/runc#4952 to commit 003fe32 (PR #2728)

[kir@kir-tp1 criu]$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# bad: [57251d8115ce58f14717d90b095901b6a1c50ea8] ci: add which dependency in dnf packages
git bisect bad 57251d8115ce58f14717d90b095901b6a1c50ea8
# status: waiting for good commit(s), bad commit known
# good: [59970a606db314fd6b24cc6e00de054f7aebfb92] images/Makefile: use msg-gen
git bisect good 59970a606db314fd6b24cc6e00de054f7aebfb92
# good: [31a58f6ababf6e987bdecd61020f7a57864a9876] zdtm: Remove junit_xml leftovers
git bisect good 31a58f6ababf6e987bdecd61020f7a57864a9876
# good: [08fa6c369a82b9945867c9f06fb1a44ff7a023c0] zdtm: add a test for --allow-uprobes option
git bisect good 08fa6c369a82b9945867c9f06fb1a44ff7a023c0
# bad: [68230aa58d9061f2435475f7e8f91f34724a24eb] x86/criu: shstk: restore SHSTK via premap loops
git bisect bad 68230aa58d9061f2435475f7e8f91f34724a24eb
# bad: [800872cb834e4423d45edc4f5cf12946c4fecde2] restorer: shstk: add restorer shadow stack stubs
git bisect bad 800872cb834e4423d45edc4f5cf12946c4fecde2
# bad: [3dde8338e9292d832f1d26e1718c609e45998df0] test/others/rpc: parse action-script via config
git bisect bad 3dde8338e9292d832f1d26e1718c609e45998df0
# bad: [003fe3223a67df4bb9a1c785d4f4a02a6075b30f] cr-service: refactor rpc config parsing
git bisect bad 003fe3223a67df4bb9a1c785d4f4a02a6075b30f
# first bad commit: [003fe3223a67df4bb9a1c785d4f4a02a6075b30f] cr-service: refactor rpc config parsing

@rst0git PTAL 🙏🏻

[kolyshkin]

it could be triggered by #2728

Guess what, @avagin, you were right.

[kolyshkin]

The only time we write something to /etc/criu is the log-file test case, initially added here: opencontainers/runc@6f3e13c

It leaves the following after being run:

$ cat /etc/criu/default.conf 
log-file=runc-criu-log-U0HAda.log

but it doesn't seem invalid to me, and I just checked the failure happens even if it's absent.

Please suggest how do I debug it further. IOW how to run criu swrk so we can see what's wrong.

[kolyshkin]

Please suggest how do I debug it further. IOW how to run criu swrk so we can see what's wrong.

@rst0git any suggestions?