criu 4.1.1 times out in runc c/r tests on GHA arm (ubuntu-24.04-arm)

[kolyshkin]

Here's the situation with ubuntu-24.04-arm and criu packages:

• when criu-dev HEAD is used, it works fine;
• when 4.1-1 from https://download.opensuse.org/repositories/devel:/tools:/criu/xUbuntu is used, it ends up being stuck on restore, and the test is aborted via a timeout.

Here you can find all the c/r logs for a test which timed out: test (ubuntu-24.04-arm, 1.24.x), raw logs

Here is a job with criu-dev which doesn't fail: https://github.com/opencontainers/runc/actions/runs/16841823814/job/47714378171?pr=4844

It's the same with criu-4.1.1 from opensuse build farm, see e.g. https://github.com/opencontainers/runc/actions/runs/16920767253/job/47946071066

I'll be happy to help with reproducing this, testing any fixes etc.

[avagin]

Does it fail if criu is compiled from the master branch?

[avagin]

   999.572 ( 0.004 ms): criu/22829 rt_sigprocmask(how: SETMASK, nset: 0xffffc78e4988, sigsetsize: 8)     = 0
   999.597 (         ): criu/22829 signal:signal_deliver(sig: 17, code: 1, sa_handler: 204828200487664, sa_flags: 268435460)
   999.619 ( 0.004 ms): criu/22829 wait4(upid: -1, stat_addr: 0xffffc78e35a4, options: NOHANG)           =
   999.626 (         ): criu/22829 rt_sigreturn()                                                     ...
   999.634 ( 0.006 ms): criu/22829 close(fd: 14)                                                         = 0
   999.644 ( 0.006 ms): criu/22829 setns(fd: 17, flags: 1073741824)                                      = 0
   999.653 ( 0.004 ms): criu/22829 close(fd: 17)                                                         = 0
   999.661 ( 0.003 ms): criu/22829 rt_sigaction(sig: CHLD, act: 0xffffc78e4a88, sigsetsize: 8)           = 0
   999.669 ( 0.006 ms): criu/22829 ptrace(request: 16903, pid: 22832)                                    = 0
   970.227 (29.501 ms): sh/22832  ... [continued]: futex())                                            = -1 (unknown) (INTERNAL ERROR: strerror_r(512, [buf], 128)=22)
   999.679 (         ): criu/22829 wait4(upid: 22832 (sh), stat_addr: 0xffffc78e4bec, options: 0x40000000) ...
   999.743 (         ): sh/22832 signal:signal_generate(sig: 17, code: 5, comm: "criu", pid: 22829 (criu), group: 1, result: 1)
   999.679 ( 0.082 ms): criu/22829  ... [continued]: wait4())                                            = 22832 (sh)
   999.767 ( 0.003 ms): criu/22829 ptrace(request: 16896, pid: 22832, data: 1)                           = 0
   999.779 ( 0.010 ms): criu/22829 write(fd: 6</tmp/bats-run-Gxsd45/runc.E4GHSW/bundle/work-dir/restore.log>, buf: 0xba4a4835fe10, count: 38) = 38
   999.793 ( 0.004 ms): criu/22829 ptrace(request: 16901, pid: 22832, addr: 0x407, data: 281474029734704) = 0
   999.799 ( 0.003 ms): criu/22829 ptrace(request: 16901, pid: 22832, addr: 0x40a, data: 281474029734704) = 0
   999.803 ( 0.002 ms): criu/22829 ptrace(request: 16901, pid: 22832, addr: 0x408, data: 281474029734704) = 0
   999.807 ( 0.003 ms): criu/22829 ptrace(request: 7, pid: 22832)                                        = 0
   999.812 ( 0.004 ms): criu/22829 futex(uaddr: 0xff398a87b010, op: WAKE, val: 2147483647)               = 0
   999.817 (         ): criu/22829 futex(uaddr: 0xff398a87b00c, val: 1, utime: 0xffffc78e4b78)        ...
   999.845 ( 0.005 ms): sh/22832 futex(uaddr: 0x27010, val: 4)                                         = -1 EAGAIN (Resource temporarily unavailable)
   999.866 (         ): sh/22832 signal:signal_generate(sig: 4, code: 2, comm: "sh", pid: 22832 (sh))
   999.880 (         ): sh/22832 signal:signal_generate(sig: 17, code: 4, comm: "criu", pid: 22829 (criu), group: 1, result: 1)
   999.817 (14337.191 ms): criu/22829  ... [continued]: futex()) 

sh/22832 is a restored process and it gets SIGILL

(gdb) disassemble 0x0000000000010a70,0x0000000000010a94
Dump of assembler code from 0x10a70 to 0x10a94:
   0x0000000000010a70:	ldr	w0, [x20]
   0x0000000000010a74:	cmp	w19, w0
   0x0000000000010a78:	b.eq	0x10a48  // b.none
   0x0000000000010a7c:	ldr	x20, [sp, #16]
   0x0000000000010a80:	ldp	x30, x19, [sp], #32
=> 0x0000000000010a84:	autiasp
   0x0000000000010a88:	ret
   0x0000000000010a8c:	mov	x0, #0x0                   	// #0
   0x0000000000010a90:	str	xzr, [x0]
End of assembler dump.

[21291.286876] sh: sh: potentially unexpected fatal signal 4.
[21291.286884] CPU: 0 UID: 0 PID: 26283 Comm: sh Not tainted 6.14.0-1011-gcp #11~24.04.1-Ubuntu
[21291.286887] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/30/2025
[21291.286889] pstate: 80001000 (Nzcv daif -PAN -UAO -TCO -DIT +SSBS BTYPE=--)
[21291.286891] pc : 0000000000010a84
[21291.286892] lr : 0037000000014048
[21291.286892] sp : 0000000000021eb0
[21291.286893] x29: 0000ffffe951e9e0 x28: 0000000000000000 x27: 0000000000000000
[21291.286896] x26: 0000000000021fb0 x25: 00000000000198c0 x24: 0000000000019150
[21291.286898] x23: 000000000001a000 x22: 0000000000019180 x21: 00000000000198c0
[21291.286900] x20: 0000000000024000 x19: 0000000000019000 x18: 0000000000000030
[21291.286902] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000065
[21291.286905] x14: 0000000000000035 x13: 0000000000000066 x12: 0000000000000000
[21291.286907] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[21291.286909] x8 : 0000000000000062 x7 : 00000000ffffffd0 x6 : 0000000000000000
[21291.286911] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[21291.286913] x2 : 0000000000000004 x1 : 0000000000000000 x0 : 0000000000000005

[rst0git]

@avagin Thanks! This is very helpful. It seems to be the problem fixed in #2689.

[avagin]

@avagin Thanks! This is very helpful. It seems to be the problem fixed in #2689.

I don't think that it is about #2689. When I compiled criu from the master branch, I can't reproduce the issue.

[avagin]

I think the criu has been compiled with PAC (-mbranch-protection=standard):

$ objdump -dl /usr/sbin/criu | grep autiasp | wc -l
2144

We probably need something like:

diff --git a/Makefile b/Makefile
index 5d8e89ac1..3bc42594d 100644
--- a/Makefile
+++ b/Makefile
@@ -64,6 +64,7 @@ endif
 
 ifeq ($(ARCH),aarch64)
         DEFINES                := -DCONFIG_AARCH64
+        CFLAGS_PIE     := -mbranch-protection=none
 endif
 
 ifeq ($(ARCH),ppc64)

Cc: @0x7f454c46

[avagin]

@rst0git please rebuild the package with these two addition patches: 6f0e4e8 and #2712.

[rst0git]

@avagin Thanks again for your help! I was able to confirm that the runc tests are passing with this patch.

[rst0git]

@kolyshkin The runc tests should work with the latest CRIU package in OBS (version 4.1.1-3):
https://github.com/rst0git/runc/actions/runs/16985852000/job/48154500314