Problems SMP an new SoC(ARM64: Cortex-A53) based board.

[dylan-hong-nc]

Environment

• HOST : ubnutu-20.04(x86_64)
• Zephyr:
  • sdk-version: v0.17.0
  • version: v4.1.0
  • note: zephyr rtos operates in non-secure area and activates secondary core with psci via TF-A (BL31).
    For reference, the same TF-A is shared and used by Linux and commercial rtos.
• SoC:
  • vendor: Nextchip
  • model: nvs2900(apache5)
    • core: cortex-a53

Describe Problem

Hello, this is my first time contacting you.
We are porting zephyr v4.1.0 to our SoC and are having trouble with SMP bring-up.

For testing, I built and ran the test code in the following path:
tests/kernel/smp

However, the test stops after the log below.
The location where the test stops is different each time you test.

NOTICE:  DRAM_FREQ: 1866 [MHz]
NOTICE:  EXT OSC     : 24 MHz
NOTICE:  DDR DENSITY : 2048 MB
NOTICE:  BL31: v2.10.4(release):v2.7.0-3025-g018cbe6ae-dirty
NOTICE:  BL31: Built : 15:18:34, Jun 20 2025
NOTICE:  APACHE5 SIP SVC Version 0.8
NOTICE:  verify header done - rsd: 2, sclk: 67000000, pol(H)-pha(H), quad
NOTICE:  leave nc_loader_read_and_verify_header...


U-Boot 2025.04-00011-g9289c08cd026 (Jun 24 2025 - 14:41:08 +0900)

DRAM:  2 GiB
Core:  39 devices, 16 uclasses, devicetree: fit
MMC:   mmc@34000000: 0, mmc@35000000: 1
Loading Environment from SPIFlash... default rx sample delay: 2
SF: Detected mt25ql512a with page size 256 Bytes, erase size 4 KiB, total 64 MiB
OK
In:    serial
Out:   serial
Err:   serial
Net:   eth0: gmac0@43020000
Hit any key to stop autoboot:  0
PHY Reset
MEGACHIPS MAAE1003S ID OK : 0x1a d414
Using gmac0@43020000 device
TFTP from server 192.168.13.31; our IP address is 192.168.13.35
Filename 'dylan7h/zephyr/zephyr.bin'.
Load address: 0x80000000
Loading: #####################
         4.5 MiB/s
done
Bytes transferred = 103016 (19268 hex)
## Starting application at 0x80000000 ...
*** Booting Zephyr OS build v4.1.0-27-g334c975fffdd ***
Secondary CPU core 1 (MPID:0x1) is up
Secondary CPU core 2 (MPID:0x2) is up
Secondary CPU core 3 (MPID:0x3) is up
Running TESTSUITE smp
===================================================================
START - test_coop_resched_threads
 PASS - test_coop_resched_threads in 0.051 seconds
===================================================================
START - test_coop_switch_in_abort
 PASS - test_coop_switch_in_abort in 0.201 seconds
===================================================================
START - test_cpu_id_threads
 PASS - test_cpu_id_threads in 1.001 seconds
===================================================================
START - test_fatal_on_smp
E: ELR_ELn: 0x00000000800014e0
E: ESR_ELn: 0x0000000056000002
E:   EC:  0x15 (Unknown)
E:   IL:  0x1
E:   ISS: 0x2
E: TPIDRRO: 0x010000008002a0d0
E: x0:  0x0000000000000000  x1:  0x0000000000000000
E: x2:  0x0000000000000000  x3:  0x0000000000000040
E: x4:  0x0000000000000001  x5:  0x0000000000000000
E: x6:  0x0000000000000000  x7:  0x0000000000000000
E: x8:  0x0000000000000003  x9:  0x0000000000000000
E: x10: 0x000000008012dbe0  x11: 0x0000000080002f98
E: x12: 0x000000008012dc38  x13: 0x0000000000000000
E: x14: 0x0000000000000000  x15: 0xfffffff7ffdfffff
E: x16: 0x0000000000000000  x17: 0x0000000000000000
E: x18: 0x0000000000000000  lr:  0x0000000080002f90
E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
E: Current thread: 0x80027e00 (unknown)
E: ELR_ELn: 0x00000000800014e0
E: ESR_ELn: 0x0000000056000002
E:   EC:  0x15 (Unknown)
E:   IL:  0x1
E:   ISS: 0x2
E: TPIDRRO: 0x010000008002a110
E: x0:  0x0000000000000000  x1:  0x0000000000000000
E: x2:  0x0000000000000000  x3:  0x000000000000032b
E: x4:  0x0000000080015838  x5:  0x0000000080014912
E: x6:  0x0000000000000001  x7:  0x0000000000000002
E: x8:  0x0000000000000003  x9:  0x0000000000000000
E: x10: 0x000000008012e7e0  x11: 0x0000000080002f98
E: x12: 0x000000008012e838  x13: 0x000000008002a000
E: x14: 0x000000008012e730  x15: 0x0000000000000000
E: x16: 0x0000000000000001  x17: 0x00000000000000b1
E: x18: 0x0000000080028180  lr:  0x0000000080001d64
E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 1
E: Current thread: 0x80028180 (test_fatal_on_smp)
 PASS - test_fatal_on_smp in 0.321 seconds
===================================================================
START - test_get_cpu
 PASS - test_get_cpu in 0.051 seconds
===================================================================
START - test_inc_concurrency
type 0: cnt 60000, spend 15 ms
type 1: cnt 60000, spend 570 ms
type 2: cnt 60000, spend 565 ms

After quite some time of debugging, I found out via jtag debugger that the flush_owned_fpu function in arch/arm64/core/fpu.c was stuck in an infinite loop due to while (atomic_ptr_get(&_kernel.cpus[i].arch.fpu_owner) == thread).
I inserted the following memory dump code to analyze the cause.

I checked the buffer printed to the memory and confirmed that it was recorded as follows.
(I set it to record all cores from 0 to 3, but the cores actually recorded were 0 and 1.)

[CPU-0]

[CPU-1]

I ran the test with FPU_SHARING disabled with CONFIG_FPU_SHARING=n because fpu_ipi didn't seem to work properly. The test went through to the end as follows:
However, the test_smp_ipi test fails.
(If I repeat the test, it often succeeds.)

NOTICE:  DRAM_FREQ: 1866 [MHz]
NOTICE:  EXT OSC     : 24 MHz
NOTICE:  DDR DENSITY : 2048 MB
NOTICE:  BL31: v2.10.4(release):v2.7.0-3025-g018cbe6ae-dirty
NOTICE:  BL31: Built : 15:18:34, Jun 20 2025
NOTICE:  APACHE5 SIP SVC Version 0.8
NOTICE:  verify header done - rsd: 2, sclk: 67000000, pol(H)-pha(H), quad
NOTICE:  leave nc_loader_read_and_verify_header...


U-Boot 2025.04-00011-g9289c08cd026 (Jun 24 2025 - 14:41:08 +0900)

DRAM:  2 GiB
Core:  39 devices, 16 uclasses, devicetree: fit
MMC:   mmc@34000000: 0, mmc@35000000: 1
Loading Environment from SPIFlash... default rx sample delay: 2
SF: Detected mt25ql512a with page size 256 Bytes, erase size 4 KiB, total 64 MiB
OK
In:    serial
Out:   serial
Err:   serial
Net:   eth0: gmac0@43020000
Hit any key to stop autoboot:  0
PHY Reset
MEGACHIPS MAAE1003S ID OK : 0x1a d414
Using gmac0@43020000 device
TFTP from server 192.168.13.31; our IP address is 192.168.13.35
Filename 'dylan7h/zephyr/zephyr.bin'.
Load address: 0x80000000
Loading: ####################
         4.3 MiB/s
done
Bytes transferred = 98888 (18248 hex)
## Starting application at 0x80000000 ...
*** Booting Zephyr OS build v4.1.0-27-g334c975fffdd ***
Secondary CPU core 1 (MPID:0x1) is up
Secondary CPU core 2 (MPID:0x2) is up
Secondary CPU core 3 (MPID:0x3) is up
Running TESTSUITE smp
===================================================================
START - test_coop_resched_threads
 PASS - test_coop_resched_threads in 0.051 seconds
===================================================================
START - test_coop_switch_in_abort
 PASS - test_coop_switch_in_abort in 0.201 seconds
===================================================================
START - test_cpu_id_threads
 PASS - test_cpu_id_threads in 1.001 seconds
===================================================================
START - test_fatal_on_smp
E: ELR_ELn: 0x00000000800014e0
E: ESR_ELn: 0x0000000056000002
E:   EC:  0x15 (Unknown)
E:   IL:  0x1
E:   ISS: 0x2
E: TPIDRRO: 0x0100000080027230
E: x0:  0x0000000000000000  x1:  0x0000000000000000
E: x2:  0x0000000000000000  x3:  0x0000000000000040
E: x4:  0x0000000000000001  x5:  0x0000000000000000
E: x6:  0x0000000000000000  x7:  0x0000000000000000
E: x8:  0x0000000000000003  x9:  0x0000000000000000
E: x10: 0x000000008012afd0  x11: 0x0000000080002f98
E: x12: 0x000000008012b028  x13: 0x0000000000000000
E: x14: 0x0000000000000000  x15: 0xffffffffffffffff
E: x16: 0x0000000000000000  x17: 0x0000000000000000
E: x18: 0x0000000000000000  lr:  0x0000000080002f90
E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
E: Current thread: 0x80026778 (unknown)
E: ELR_ELn: 0x00000000800014e0
E: ESR_ELn: 0x0000000056000002
E:   EC:  0x15 (Unknown)
E:   IL:  0x1
E:   ISS: 0x2
E: TPIDRRO: 0x0100000080027258
E: x0:  0x0000000000000000  x1:  0x0000000000000000
E: x2:  0x0000000000000000  x3:  0x000000000000032b
E: x4:  0x0000000080014838  x5:  0x0000000080013912
E: x6:  0x0000000000000001  x7:  0x0000000000000002
E: x8:  0x0000000000000003  x9:  0x0000000000000000
E: x10: 0x000000008012bbd0  x11: 0x0000000080002f98
E: x12: 0x000000008012bc28  x13: 0x0000000080027000
E: x14: 0x000000008012bb20  x15: 0x0000000000000000
E: x16: 0x0000000000000001  x17: 0x00000000000000b1
E: x18: 0x0000000080026948  lr:  0x0000000080001d64
E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 1
E: Current thread: 0x80026948 (test_fatal_on_smp)
 PASS - test_fatal_on_smp in 0.321 seconds
===================================================================
START - test_get_cpu
 PASS - test_get_cpu in 0.051 seconds
===================================================================
START - test_inc_concurrency
type 0: cnt 60000, spend 15 ms
type 1: cnt 60000, spend 547 ms
type 2: cnt 60000, spend 547 ms
 PASS - test_inc_concurrency in 1.116 seconds
===================================================================
START - test_preempt_resched_threads
 PASS - test_preempt_resched_threads in 0.512 seconds
===================================================================
START - test_sleep_threads
 PASS - test_sleep_threads in 1.001 seconds
===================================================================
START - test_smp_affinity
 PASS - test_smp_affinity in 0.001 seconds
===================================================================
START - test_smp_coop_threads
 PASS - test_smp_coop_threads in 0.613 seconds
===================================================================
START - test_smp_ipi
cpu num=4
    Assertion failed at WEST_TOPDIR/zephyr/tests/kernel/smp/src/main.c:753: smp_test_smp_ipi: (sched_ipi_has_called != 0 is false)
did not receive IPI.(0)
 FAIL - test_smp_ipi in 0.115 seconds
===================================================================
START - test_smp_release_global_lock
 PASS - test_smp_release_global_lock in 0.021 seconds
===================================================================
START - test_smp_switch_torture
 PASS - test_smp_switch_torture in 15.001 seconds
===================================================================
START - test_wakeup_threads
 PASS - test_wakeup_threads in 0.201 seconds
===================================================================
START - test_workq_on_smp
 PASS - test_workq_on_smp in 0.051 seconds
===================================================================
START - test_yield_threads
 PASS - test_yield_threads in 0.301 seconds
===================================================================
TESTSUITE smp failed.

------ TESTSUITE SUMMARY START ------

SUITE FAIL -  93.75% [smp]: pass = 15, fail = 1, skip = 0, total = 16 duration = 20.558 seconds
 - PASS - [smp.test_coop_resched_threads] duration = 0.051 seconds
 - PASS - [smp.test_coop_switch_in_abort] duration = 0.201 seconds
 - PASS - [smp.test_cpu_id_threads] duration = 1.001 seconds
 - PASS - [smp.test_fatal_on_smp] duration = 0.321 seconds
 - PASS - [smp.test_get_cpu] duration = 0.051 seconds
 - PASS - [smp.test_inc_concurrency] duration = 1.116 seconds
 - PASS - [smp.test_preempt_resched_threads] duration = 0.512 seconds
 - PASS - [smp.test_sleep_threads] duration = 1.001 seconds
 - PASS - [smp.test_smp_affinity] duration = 0.001 seconds
 - PASS - [smp.test_smp_coop_threads] duration = 0.613 seconds
 - FAIL - [smp.test_smp_ipi] duration = 0.115 seconds
 - PASS - [smp.test_smp_release_global_lock] duration = 0.021 seconds
 - PASS - [smp.test_smp_switch_torture] duration = 15.001 seconds
 - PASS - [smp.test_wakeup_threads] duration = 0.201 seconds
 - PASS - [smp.test_workq_on_smp] duration = 0.051 seconds
 - PASS - [smp.test_yield_threads] duration = 0.301 seconds

------ TESTSUITE SUMMARY END ------

===================================================================
PROJECT EXECUTION FAILED

I tried running the shell (not smp test) with FPU_SHARING disabled and the options below.

CONFIG_SHELL=y
CONFIG_SHELL_BACKEND_SERIAL=y

As an experiment, I tried entering kernel sleep 100 as shell input, but it did not wake up from sleep.

However, I confirmed that if I apply the scheduling options below, it wakes up normally regardless of the sleep value.

CONFIG_TIMESLICE_PER_THREAD=y
CONFIG_TIMESLICE_SIZE=1
CONFIG_TICKET_SPINLOCKS=y

Unfortunately, I tried re-enabling FPU_SHARING while keeping this option enabled, but the result was the same.

What else can I test?
Does SMP work properly on other ARM64 based boards?
I've attached some of my config files.

Please help,
Thank you.

zephyr_smp_debug.zip

++++++++++++++++++++++++++
I tried to reflect the changes ARM64_DCACHE_ALL_OPS, ARM64_BOOT_DISABLE_DCACHE and arm/arm64/core/switch.S from the main branch to my workspace, but it didn't give me any meaningful results.

[arm/arm64/core/switch.S]
#ifdef CONFIG_SMP
	/* Write barrier: ensure all preceding writes are executed
	* before writing the switch handle
	*/
	dmb sy
#endif

When jumping from u-boot to zephyr I used the following commands:

dcache flush; icache flush; dcache off; icache off; go ${image_addr_r}

[JarmouniA]

Cc @wearyzen

[wearyzen]

Hi @dylan-hong-nc there is a qemu board https://github.com/zephyrproject-rtos/zephyr/blob/main/boards/qemu/cortex_a53/qemu_cortex_a53_qemu_cortex_a53_smp.yaml which is tested in ci and doesn't show any failure. I tried west twister -p qemu_cortex_a53/qemu_cortex_a53/smp -T tests/kernel/smp locally and it didn't have the issue you reported.

[dylan-hong-nc]

Hi @wearyzen.

Thank you for your cooperation.

As you answered, it seems that this issue does not occur in the emulator environment.
I think it is caused by differences in operation between the HW and the emulator.
(mmu, cache, errata, etc...)

I will get another reference board (Raspberry Pi5) and test it again.
I will share the test results again when I get them.

Thank you.

[dylan-hong-nc]

Hi @wearyzen.

I got a raspberry pi5 and tried running 'tests/kernel/smp' with SMP related options added.

The symptoms are different from the SoC under test, but it seems that the threads do not work properly in the SMP environment.

• An assert is raised in 'test_coop_resched_threads' as shown in the log.
• 'test_cpu_id_threads' starts but never finishes.

Are there any tuning points that can be applied?

I am sharing the changes applied to apply SMP to the Raspberry Pi 5 as a patch file.
0001-enable-smp-for-rpi5.patch

[LOG Summary]

[Full Log]

  0.43 RPi: BOOTSYS release VERSION:26826259 DATE: 2024/09/23 TIME: 14:02:56
  0.48 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1727096576 serial 12a9a82d boardrev d04170 stc 848052
  0.58 AON_RESET: 00000003 PM_RSTS 00001000
  0.65 RP1_BOOT chip ID: 0x20001927
  0.68 PM_RSTS: 0x00001000
  0.68 part 00000000 reset_info 00000000
  0.72 PMIC reset-event 00000000 rtc 00000000 alarm 00000000 enabled 0
  0.78 uSD voltage 3.3V
  0.97 Initialising SDRAM rank 2 total-size: 64 Gbit 4267 (0x14 0x00)
  1.00 DDR 4267 1 0 64 152
  2.22 OTP boardrev d04170 bootrom a a
  2.23 Customer key hash 0000000000000000000000000000000000000000000000000000000000000000
  2.30 VC-JTAG unlocked
  2.53 RP1_BOOT chip ID: 0x20001927

  3.55 RP1_BOOT chip ID: 0x20001927
  3.55 RP1_BOOT: fw size 25992
  3.10 PCI2 init
  3.10 PCI2 reset
  3.55 PCIe scan 00001de4:00000001
  3.56 RP1_CHIP_INFO 20001927

  3.58 RPi: BOOTLOADER release VERSION:26826259 DATE: 2024/09/23 TIME: 14:02:56
  3.65 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1727096576 serial 12a9a82d boardrev d04170 stc 3865914
  3.75 AON_RESET: 00000003 PM_RSTS 00001000
  3.79 PCIEx1: PWR 0 DET_WAKE 0
  3.82 M.2 PCIe HAT not detected.
  3.06 usb_pd_init status 3
  3.08 USB_PD CONFIG 0 41
  3.14 XHCI-STOP
  3.14 xHC0 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
  3.19 USBSTS 1
  3.21 xHC0 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
  3.26 xHC0 ports 3 slots 64 intrs 4
  3.38 XHCI-STOP
  3.38 xHC1 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
  3.43 USBSTS 1
  3.45 xHC1 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
  3.50 xHC1 ports 3 slots 64 intrs 4
  3.58 Boot mode: SD (01) order f4
  4.59 USB-PD: src-cap PDO object1 0x0801912c
  4.60 Current 3000 mA
  4.62 Voltage 5000 mV
  4.64 USB-PD: src-cap PDO object2 0x0002d116
  4.68 Current 2780 mA
  4.70 Voltage 9000 mV
  4.72 USB-PD: src-cap PDO object3 0x0003c0d0
  4.76 Current 2080 mA
  4.79 Voltage 12000 mV
  4.81 USB-PD: src-cap PDO object4 0x0004b0a7
  4.85 Current 1670 mA
  4.87 Voltage 15000 mV
  4.89 USB-PD: src-cap PDO object5 0x0006407d
  4.93 Current 1250 mA
  4.95 Voltage 20000 mV
  4.97 USB-PD: src-cap PDO object6 0xc0dc2137
  4.16 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
  4.27 SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
  4.01 OCR c0ff8000 [104]
CID: 00035344534436344785fb314ef2a17a
CSD: 400e00325b590001db937f800a400000
  4.09 SD: bus-width: 4 spec: 2 SCR: 0x02458083 0x00000000
  4.16 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2
  4.26 MBR: 0x00004000, 1048576 type: 0x0c
  4.28 MBR: 0x00104000,123604992 type: 0x83
  4.32 MBR: 0x00000000,       0 type: 0x00
  4.36 MBR: 0x00000000,       0 type: 0x00
  4.40 Trying partition: 0
  4.42 type: 32 lba: 16384 'mkfs.fat' ' bootfs     ' clusters 261116 (4)
  4.48 rsc 32 fat-sectors 2040 root dir cluster 2 sectors 0 entries 0
  4.55 FAT32 clusters 261116
  4.62 [sdcard] autoboot.txt not found
  4.63 Select partition rsts 0 C(boot_partition) 0 EEPROM config 0 result 0
  4.69 Trying partition: 0
  4.72 type: 32 lba: 16384 'mkfs.fat' ' bootfs     ' clusters 261115 (4)
  4.78 rsc 32 fat-sectors 2040 root dir cluster 2 sectors 0 entries 0
  4.85 FAT32 clusters 261115
  4.90 Read config.txt bytes     1361 hnd 0x27e
  4.93 [sdcard] pieeprom.upd not found
  4.95 usb_max_current_enable default 0 max-current 3000
  4.06 Read bcm2712-rpi-5-b.dtb bytes    78231 hnd 0x125
  4.09 dt-match: compatible: raspberrypi,5-model-b match: brcm,bcm2712
  4.15 dt-match: compatible: brcm,bcm2712 match: brcm,bcm2712
  4.21 Selecting USB low current limit
  4.28 Read /config.txt bytes     1361 hnd 0x27e
  4.32 Read /config.txt bytes     1361 hnd 0x27e
  4.37 MESS:00:00:04.837268:0: Initial voltage 800000 temp 32334
  5.37 MESS:00:00:05.037568:0: avs_2712: AVS pred 8466 846600 temp 33983
  5.41 MESS:00:00:05.041173:0: vpred 846 mV +0
  5.69 MESS:00:00:05.669271:0: FB framebuffer_swap 1
  5.88 MESS:00:00:05.688636:0: Select resolution HDMI0/2 hotplug 0 max_mode 2
  5.92 MESS:00:00:05.692685:0: Select resolution HDMI1/2 hotplug 0 max_mode 2
  5.01 Loading 'initramfs_2712' to 0x00000000 offset 0x0
  6.75 Read initramfs_2712 bytes 18075599 hnd 0x7c0c
  6.63 MESS:00:00:06.563244:0: dtb_file 'bcm2712-rpi-5-b.dtb'
  6.65 Loading 'bcm2712-rpi-5-b.dtb' to 0x00000000 offset 0x100
  6.78 Read bcm2712-rpi-5-b.dtb bytes    78231 hnd 0x125
  6.16 Read /overlays/overlay_map.dtb bytes     5627 hnd 0x7b13
  6.46 PCIEx1: PWR 0 DET_WAKE 0
  6.59 Read /config.txt bytes     1361 hnd 0x27e
  6.60 MESS:00:00:06.660673:0: dtparam: audio=on
  6.69 MESS:00:00:06.669183:0: Unknown dtparam 'audio' - ignored
  6.88 Read /overlays/vc4-kms-v3d-pi5.dtbo bytes     3350 hnd 0x7bf3
  6.35 MESS:00:00:06.735573:0: Loaded overlay 'vc4-kms-v3d-pi5'
  6.06 Read /cmdline.txt bytes      177 hnd 0x27d
  7.46 BMD "armstub8-2712.bin" not found
  7.47 fs_open: 'armstub8-2712.bin'
  7.50 Loading 'zephyr.bin' to 0x00000000 offset 0x200000
  7.62 Read zephyr.bin bytes    90392 hnd 0x30005
  7.63 PCI1 reset
  7.75 PCI2 reset
  7.85 USB-OTG disconnect
  8.25 MESS:00:00:08.225338:0: Starting OS 8225 ms
  8.29 MESS:00:00:08.229651:0: 00000040: -> 00000480
  8.31 MESS:00:00:08.231743:0: 00000030: -> 00100080
  8.36 MESS:00:00:08.236456:0: 00000034: -> 00100080
  8.41 MESS:00:00:08.241168:0: 00000038: -> 00100080
  8.45 MESS:00:00:08.245881:0: 0000003c: -> 00100080

NOTICE:  BL31: v2.6(release):v2.6-239-g2a9ede0bd
NOTICE:  BL31: Built : 14:26:57, Jun 22 2023
*** Booting Zephyr OS build v4.1.0 ***
Secondary CPU core 1 (MPID:0x100) is up
Secondary CPU core 2 (MPID:0x200) is up
Secondary CPU core 3 (MPID:0x300) is up
Running TESTSUITE smp
===================================================================
START - test_coop_resched_threads

    Assertion failed at WEST_TOPDIR/zephyr/tests/kernel/smp/src/main.c:388: smp_test_coop_resched_threads: (tinfo[i].executed == 1 is false)
cooperative thread 0 didn't run
 FAIL - test_coop_resched_threads in 1.063 seconds
===================================================================
START - test_coop_switch_in_abort
 PASS - test_coop_switch_in_abort in 0.201 seconds
===================================================================
START - test_cpu_id_threads

[wearyzen]

Hi @dylan-hong-nc rpi doesn't seem to have smp support yet and I am not an expert on aarch64 so, I am not sure if your changes would be enough.
If you would like someone to take a look debug further on your rpi changes I would suggest checking out Zephyr's raspberry discord channel: https://discord.com/channels/720317445772017664/938474761405726800
For your original issue I would suggest comparing your dts with https://github.com/zephyrproject-rtos/zephyr/blob/main/dts/arm64/qemu/qemu-virt-a53.dtsi and

zephyr/boards/qemu/cortex_a53/qemu_cortex_a53.dts

Line 9 in fdd02c5

#include <arm64/qemu/qemu-virt-a53.dtsi>

and defconfig with https://github.com/zephyrproject-rtos/zephyr/blob/fdd02c5105d2a40cf6410cf89ccd22c43a48399e/boards/qemu/cortex_a53/qemu_cortex_a53_qemu_cortex_a53_smp_defconfig

[dylan-hong-nc]

Hi @wearyzen.

I have written about the causes and solutions to this phenomenon on the following pages.
This definitely looks like a bug.
#93545

[wearyzen]

Hi @dylan-hong-nc, good to know that you were able to get the tests working. Could you kindly confirm if the change you mentioned below in the #93545 the only change needed or is it different from the issue being discussed here?

I modified it to backup x0 as follows and compare only InterruptID with x1.

[dylan-hong-nc]

Hi @wearyzen.
I have enabled the 'TICKET_SPINLOCKS' option in addition to the changes suggested in #93545.
If you do not turn on the 'TICKET_SPINLOCKS' option, SMP TESTSUITE will fail.

I think that the fact that TESTSUITE will run properly only when 'TICKET_SPINLOCKS' is turned on is separate from this issue.

The error message that is printed when it fails is as follows:

===================================================================
START - test_smp_switch_torture
ASSERTION FAIL [!arch_is_in_isr()] @ WEST_TOPDIR/zephyr/kernel/poll.c:297

[00:00:04.275,000] <err> os: ELR_ELn: 0x000000008000373c
[00:00:04.277,000] <err> os: ESR_ELn: 0x0000000056000002
[00:00:04.278,000] <err> os:   EC:  0x15 (Unknown)
[00:00:04.279,000] <err> os:   IL:  0x1
[00:00:04.280,000] <err> os:   ISS: 0x2
[00:00:04.281,000] <err> os: TPIDRRO: 0x010000008002cf80
[00:00:04.282,000] <err> os: x0:  0x000000008001b579  x1:  0x0000000000000129
[00:00:04.283,000] <err> os: x2:  0x0000000000000000  x3:  0x0000000080018388
[00:00:04.285,000] <err> os: x4:  0x000000008043ed60  x5:  0x0000000000000000
[00:00:04.287,000] <err> os: x6:  0x000000008043ed50  x7:  0x000000008043ed60
[00:00:04.288,000] <err> os: x8:  0x0000000000000004  x9:  0x00000000ffffffc8
[00:00:04.290,000] <err> os: x10: 0x000000008043ef60  x11: 0x000000008043ef20
[00:00:04.291,000] <err> os: x12: 0x0000000000000001  x13: 0x0000000200007fed
[00:00:04.293,000] <err> os: x14: 0x1c4000001c400000  x15: 0x0300200000000000
[00:00:04.294,000] <err> os: x16: 0x0000000000000000  x17: 0x0000000000000000
[00:00:04.296,000] <err> os: x18: 0x000000008043ef60  lr:  0x00000000800131a4
[00:00:04.297,000] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
[00:00:04.299,000] <err> os: Current thread: 0x8002a6c0 (unknown)
wrong error reason
===================================================================
PROJECT EXECUTION FAILED
ASSERTION FAIL [!arch_is_in_isr()] @ WEST_TOPDIR/zephyr/kernel/poll.c:297

[00:00:04.305,000] <err> os: ELR_ELn: 0x000000008000373c
[00:00:04.306,000] <err> os: ESR_ELn: 0x0000000056000002
[00:00:04.307,000] <err> os:   EC:  0x15 (Unknown)
[00:00:04.308,000] <err> os:   IL:  0x1
[00:00:04.309,000] <err> os:   ISS: 0x2
[00:00:04.310,000] <err> os: TPIDRRO: 0x010000008002cfc0
[00:00:04.311,000] <err> os: x0:  0x000000008001b579  x1:  0x0000000000000129
[00:00:04.313,000] <err> os: x2:  0x0000000000000000  x3:  0x0000000080018388
[00:00:04.314,000] <err> os: x4:  0x000000008043e3e0  x5:  0x0000000000000000
[00:00:04.316,000] <err> os: x6:  0x000000008043e3d0  x7:  0x000000008043e3e0
[00:00:04.317,000] <err> os: x8:  0x0000000000000004  x9:  0x00000000ffffffc8
[00:00:04.319,000] <err> os: x10: 0x000000008043e5e0  x11: 0x000000008043e5a0
[00:00:04.320,000] <err> os: x12: 0x0000000000000001  x13: 0x0000000200007fed
[00:00:04.322,000] <err> os: x14: 0x1c4000001c400000  x15: 0x0300200000000000
[00:00:04.323,000] <err> os: x16: 0x0000000000000000  x17: 0x0000000000000000
[00:00:04.325,000] <err> os: x18: 0x000000008043e5e0  lr:  0x00000000800131a4
[00:00:04.327,000] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 1
[00:00:04.328,000] <err> os: Current thread: 0x8002a360 (unknown)
wrong error reason
===================================================================
PROJECT EXECUTION FAILED