AArch64: cxl-region-sysfs.sh: line 49: echo: write error: Numerical result out of range

[ikitayama]

$ meson test cxl-region-sysfs.sh fails with the messages below from testlog.txt file:

[...]
+ uuidgen
+ nr_targets=8
+ echo 8
++ cat /sys/bus/cxl/devices/decoder2.3/interleave_granularity
+ r_ig=4096
+ echo 4096
+ echo 2147483648
/home/realm/projects/ndctl/test/cxl-region-sysfs.sh: line 49: echo: write error: Numerical result out of range
++ err 49
+++ basename /home/realm/projects/ndctl/test/cxl-region-sysfs.sh
++ echo test/cxl-region-sysfs.sh: failed at line 49
++ '[' -n '' ']'
++ exit 1
[...]

The kernel and cxl_test module is built from cxl-next:
https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/

ndctl version is HEAD.

[marc-hb]

I think I know why, stay tuned. Will share more in the next few days.

[ikitayama]

@marc-hb thanks! cxl-create-region.sh , cxl-xor-region.sh , cxl-qos-class.sh , and cxl-poison.sh too are affected?

[ikitayama]

@marc-hb I couldn't understand well why echo'ing 2GB to the sysfs size entry ends up with an out of range, but smaller values fails with invalid argument. Writing 0 seems to work but no meaning at all. The string to integer conversion shouldn't be an issue should it?

[marc-hb]

Sorry I got distracted. I just replied on the list:
https://lore.kernel.org/linux-cxl/43568B03-6832-4EB1-BF46-EF0F176509E2@linux.dev/T/#u

[marc-hb]

https://lore.kernel.org/linux-cxl/43568B03-6832-4EB1-BF46-EF0F176509E2@linux.dev/T/#u

I spent a bit more time on this and realized that kaslr seems 100% x86-specific. So we're probably experiencing different issues, sorry for the noise. That error message seems quite generic actually. You need to collect more logs and information.

[ikitayama]

@marc-hb thanks for letting me know! I'm still looking at the host physical address allocation code, that ends up searching for free area, provided we only need 8 x 256MB region size, something kernel or QEMU is having issues or both?

[ikitayama]

This is a follow-up on this Issue, again tested on a CXL capable emulated system laucned from run_qemu.sh.

testlog.txt

[ikitayama]

This is a follow-up on this Issue, again tested on a CXL capable emulated system laucned from run_qemu.sh.

testlog.txt

setup_cxl() bash function is intact while I had to make some editing to the other areas of the run_qemu.sh script, so I will try to look at the kernel which @jic23 maintains.

[marc-hb]

Jonathan replied in pmem/run_qemu#150, you can find his account name there.

A trick to find anyone's account is to look at any commit they authored in any git repo hosted or mirrored on GitHub.

[marc-hb]

testlog.txt

The kernel logs are very likely more useful.

[ikitayama]

Is this better?

root@localhost:~/ndctl/build# meson test cxl-region-sysfs.sh
ninja: Entering directory `/root/ndctl/build'
[1/55] Generating version.h with a custom command
[ 1168.911645] cxl_region region2: Bypassing cpu_cache_invalidate_memregion() for testing!
[ 1170.025972] calling  cxl_port_init+0x0/0x1000 [cxl_port] @ 1019
[ 1170.026559] initcall cxl_port_init+0x0/0x1000 [cxl_port] returned 0 after 279 usecs
[ 1170.033707] calling  cxl_acpi_init+0x0/0x1000 [cxl_acpi] @ 1019
[ 1170.066885] probe of port1 returned 0 after 23023 usecs
[ 1170.067394]  pci0000:bf: host supports CXL
[ 1170.098168] probe of port2 returned 0 after 29658 usecs  2/600s
[ 1170.098754]  pci0000:35: host supports CXL
[ 1170.100979] probe of ACPI0017:00 returned 0 after 66209 usecs
[ 1170.101537] initcall cxl_acpi_init+0x0/0x1000 [cxl_acpi] returned 0 after 67515 usecs
[ 1170.111145] calling  cxl_pmem_init+0x0/0x1000 [cxl_pmem] @ 1019
[ 1170.132105] probe of ndbus0 returned 0 after 1526 usecs
[ 1170.133293] probe of nvdimm-bridge0 returned 0 after 21130 usecs
[ 1170.134197] initcall cxl_pmem_init+0x0/0x1000 [cxl_pmem] returned 0 after 22250 usecs
[ 1170.137744] calling  cxl_mem_driver_init+0x0/0x1000 [cxl_mem] @ 1019
[ 1170.167006] probe of port3 returned 0 after 27343 usecs
[ 1170.174010] cxl_nvdimm pmem11: GPF: could not set dirty shutdown state
[ 1170.176910] probe of nmem0 returned 0 after 1145 usecs
[ 1170.177519] probe of pmem11 returned 0 after 3926 usecs
[ 1170.206636] probe of endpoint4 returned 0 after 28398 usecs
[ 1170.208604] probe of mem11 returned 0 after 70057 usecs
[ 1170.209787] cxl_nvdimm pmem12: GPF: could not set dirty shutdown state
[ 1170.225910] probe of nmem1 returned 0 after 2603 usecs
[ 1170.226936] probe of pmem12 returned 0 after 17298 usecs
[ 1170.271764] probe of endpoint5 returned 0 after 29547 usecs
[ 1170.272374] probe of mem12 returned 0 after 63332 usecs
[ 1170.277418] initcall cxl_mem_driver_init+0x0/0x1000 [cxl_mem] returned 0 after 138998 usecs
[ 1170.284687] calling  cxl_test_init+0x0/0x1000 [cxl_test] @ 1019
[ 1170.711571] platform cxl_host_bridge.0: Unsupported platform config, mixed Virtual Host and Restricted CXL Host hierarchy.
[ 1170.714631] platform cxl_host_bridge.1: Unsupported platform config, mixed Virtual Host and Restricted CXL Host hierarchy.
[ 1170.715683] platform cxl_host_bridge.2: Unsupported platform config, mixed Virtual Host and Restricted CXL Host hierarchy.
[ 1170.716569] platform cxl_host_bridge.3: Unsupported platform config, mixed Virtual Host and Restricted CXL Host hierarchy.
[ 1170.727275] platform cxl_host_bridge.0: Unsupported platform config, mixed Virtual Host and Restricted CXL Host hierarchy.
[ 1170.757243] probe of port7 returned 0 after 28879 usecs
[ 1170.757940] platform cxl_host_bridge.0: host supports CXL
[ 1170.758165] platform cxl_host_bridge.1: Unsupported platform config, mixed Virtual Host and Restricted CXL Host hierarchy.
[ 1170.760660] probe of port8 returned 0 after 1897 usecs
[ 1170.760917] platform cxl_host_bridge.1: host supports CXL
[ 1170.764436] platform cxl_host_bridge.2: Unsupported platform config, mixed Virtual Host and Restricted CXL Host hierarchy.
[ 1170.783793] probe of port9 returned 0 after 18356 usecs
[ 1170.784508] platform cxl_host_bridge.2: host supports CXL
[ 1170.788123] platform cxl_host_bridge.3: host supports CXL (restricted)
[ 1170.795005] probe of ndbus1 returned 0 after 717 usecs
[ 1170.795767] probe of nvdimm-bridge1 returned 0 after 1758 usecs
[ 1170.798731] probe of cxl_acpi.0 returned 0 after 87790 usecs
[ 1170.824632] initcall cxl_test_init+0x0/0x1000 [cxl_test] returned 0 after 538728 usecs
[ 1170.832593] cxl_mock_mem cxl_mem.0: CXL MCE unsupported
[ 1170.842623] cxl_mock_mem cxl_mem.4: CXL MCE unsupported
[ 1170.851235] probe of port10 returned 0 after 2038 usecs
[ 1170.852016] cxl_mock_mem cxl_mem.2: CXL MCE unsupported
[ 1170.857719] cxl_mock_mem cxl_mem.5: CXL MCE unsupported
[ 1170.864625] cxl_mock_mem cxl_mem.1: CXL MCE unsupported
[ 1170.869519] probe of nmem2 returned 0 after 307 usecs
[ 1170.869550] probe of nmem3 returned 0 after 479 usecs
[ 1170.869676] probe of pmem0 returned 0 after 17586 usecs
[ 1170.870168] probe of pmem2 returned 0 after 13243 usecs
[ 1170.872743] probe of region2 returned 6 after 56 usecs
[ 1170.873131] cxl_mock_mem cxl_mem.4: Extended linear cache calculation failed rc:-2
[ 1170.875226] cxl_mock_mem cxl_mem.3: CXL MCE unsupported
[ 1170.878772] cxl_mock_mem cxl_mem.6: CXL MCE unsupported
[ 1170.893275] cxl_mock_mem cxl_mem.7: CXL MCE unsupported
[ 1170.899496] cxl_mock_mem cxl_mem.8: CXL MCE unsupported
[ 1170.903380] cxl_mock_mem cxl_mem.9: CXL MCE unsupported
[ 1170.907442] cxl_mock_mem cxl_rcd.10: CXL MCE unsupported
[ 1170.919345] probe of endpoint11 returned 0 after 48758 usecs
[ 1170.930629] probe of mem0 returned 0 after 81921 usecs
[ 1170.931396] probe of cxl_mem.4 returned 0 after 112290 usecs
[ 1170.932469] probe of port12 returned 0 after 57455 usecs
[ 1170.937940] Hotplug memory [0xfffffff010000000-0xfffffff030000000] exceeds maximum addressable range [0x0-0x487e67fffffff]
[ 1170.938621] kmem dax2.0: mapping0: 0xfffffff010000000-0xfffffff02fffffff memory add failed
[ 1170.938908] kmem dax2.0: probe with driver kmem failed with error -7
[ 1170.939144] probe of dax2.0 returned 7 after 1368 usecs
[ 1170.948633] probe of endpoint16 returned 0 after 2475 usecs
[ 1170.949625] probe of nmem4 returned 0 after 719 usecs
[ 1170.949745] probe of mem10 returned 0 after 31080 usecs
[ 1170.950285] probe of pmem4 returned 0 after 4223 usecs
[ 1170.950888] probe of cxl_rcd.10 returned 0 after 72239 usecs
[ 1170.952332] probe of dax_region2 returned 0 after 29055 usecs
[ 1170.952724] probe of region2 returned 0 after 29714 usecs
[ 1170.953760] probe of endpoint15 returned 0 after 33541 usecs
[ 1170.954299] probe of mem2 returned 0 after 97812 usecs
[ 1170.955180] probe of cxl_mem.0 returned 0 after 151506 usecs
[ 1170.956418] probe of port14 returned 0 after 18871 usecs
[ 1170.957493] probe of port13 returned 0 after 81125 usecs
[ 1170.959361] probe of nmem5 returned 0 after 170 usecs
[ 1170.959712] probe of pmem6 returned 0 after 1169 usecs
[ 1170.960560] probe of nmem7 returned 0 after 294 usecs
[ 1170.960900] probe of pmem8 returned 0 after 1494 usecs
[ 1170.967410] probe of endpoint18 returned 0 after 11859 usecs
[ 1170.971395] probe of nmem6 returned 0 after 8898 usecs
[ 1170.971724] probe of pmem1 returned 0 after 13147 usecs
[ 1170.973271] probe of mem4 returned 0 after 98654 usecs
[ 1170.973766] probe of endpoint19 returned 0 after 13492 usecs
[ 1170.974100] probe of cxl_mem.1 returned 0 after 165966 usecs
[ 1170.975162] probe of port17 returned 0 after 28467 usecs
[ 1170.990984] probe of mem6 returned 0 after 105441 usecs
[ 1170.991973] probe of cxl_mem.6 returned 0 after 153482 usecs
[ 1170.992518] probe of nmem8 returned 0 after 262 usecs
[ 1170.992912] probe of pmem5 returned 0 after 1528 usecs
[ 1171.008019] probe of endpoint22 returned 0 after 2509 usecs
[ 1171.010738] probe of nmem9 returned 0 after 15358 usecs
[ 1171.017272] probe of pmem7 returned 0 after 24832 usecs
[ 1171.017568] probe of mem5 returned 0 after 141004 usecs
[ 1171.018595] probe of cxl_mem.3 returned 0 after 200129 usecs
[ 1171.019844] probe of nmem10 returned 0 after 214 usecs
[ 1171.020729] probe of pmem3 returned 0 after 23600 usecs
[ 1171.021539] probe of endpoint20 returned 0 after 46805 usecs
[ 1171.029229] probe of mem8 returned 0 after 124606 usecs
[ 1171.030245] probe of cxl_mem.8 returned 0 after 170447 usecs
[ 1171.030867] probe of endpoint23 returned 0 after 12984 usecs
[ 1171.031725] probe of mem7 returned 0 after 127008 usecs
[ 1171.032658] probe of cxl_mem.7 returned 0 after 185239 usecs
[ 1171.036191] probe of endpoint24 returned 0 after 14865 usecs
[ 1171.039651] probe of nmem11 returned 0 after 275 usecs
[ 1171.040281] probe of pmem9 returned 0 after 1676 usecs
[ 1171.053750] probe of endpoint25 returned 0 after 12892 usecs
[ 1171.068773] probe of mem3 returned 0 after 191752 usecs
[ 1171.073524] probe of mem9 returned 0 after 165319 usecs
[ 1171.087152] probe of cxl_mem.5 returned 0 after 254973 usecs00s
[ 1171.112416] probe of cxl_mem.9 returned 0 after 247756 usecs
[ 1171.268521] probe of endpoint21 returned 0 after 266319 usecs
[ 1171.281923] probe of mem1 returned 0 after 405914 usecs
[ 1171.348228] probe of cxl_mem.2 returned 0 after 533905 usecs
[ 1174.275907] probe of region5 returned 6 after 94 usecs   6/600s
1/1 ndctl:cxl / cxl-region-sysfs.sh        FAIL             6.28s   exit status 1
>>> MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MESON_TEST_ITERATION=1 MALLOC_PERTURB_=116 NDCTL=/root/ndctl/build/ndctl/ndctl LD_LIBRARY_PATH=/root/ndctl/build/cxl/lib:/root/ndctl/build/ndctl/lib:/root/ndctl/build/daxctl/lib DATA_PATH=/root/ndctl/test ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 TEST_PATH=/root/ndctl/build/test DAXCTL=/root/ndctl/build/daxctl/daxctl /bin/bash /root/ndctl/test/cxl-region-sysfs.sh


Ok:                 0
Expected Fail:      0
Fail:               1
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /root/ndctl/build/meson-logs/testlog.txt

Could @jic23 comment on this too?

[marc-hb]

[ 1170.937940] Hotplug memory [0xfffffff010000000-0xfffffff030000000] exceeds maximum addressable range [0x0-0x487e67fffffff]

Interesting, I get a similar error when x86/KASLR is enabled by default. That's because KASLR reduces the maximum range available[*]. You seem to land in a similar place in a different and unrelated way?

[*] EDIT: the relevant kernel commits are torvalds/linux@ea72ce5da22806 and torvalds/linux@afe789b7367a

[ikitayama]

[...]
[ 1170.758165] platform cxl_host_bridge.1: Unsupported platform config, mixed Virtual Host and Restricted CXL Host hierarchy.
[ 1170.760660] probe of port8 returned 0 after 1897 usecs

@marc-hb @AlisonSchofield do you happen to know how the CXL setup on the q35 machine avoids this situation? The VM launched from the run_qemu.sh script has CEDT, but not sure if it is a valid ACPI table, nor if the kernel (driver) parses it correctly. I never edited the options set in setup_cxl(). QEMU I'm using is the latest development of @jic23. The kernel is the cxl-next the Intel CXL folks maintains.

[ikitayama]

Dump of the ACPI CEDT table I am using:

/*
 * Intel ACPI Component Architecture
 * AML/ASL+ Disassembler version 20230628 (64-bit version)
 * Copyright (c) 2000 - 2023 Intel Corporation
 * 
 * Disassembly of cedt.dat
 *
 * ACPI Data Table [CEDT]
 *
 * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue (in hex)
 */

[000h 0000 004h]                   Signature : "CEDT"    [CXL Early Discovery Table]
[004h 0004 004h]                Table Length : 000000B8
[008h 0008 001h]                    Revision : 01
[009h 0009 001h]                    Checksum : 4E
[00Ah 0010 006h]                      Oem ID : "BOCHS "
[010h 0016 008h]                Oem Table ID : "BXPC    "
[018h 0024 004h]                Oem Revision : 00000001
[01Ch 0028 004h]             Asl Compiler ID : "BXPC"
[020h 0032 004h]       Asl Compiler Revision : 00000001


[024h 0036 001h]               Subtable Type : 00 [CXL Host Bridge Structure]
[025h 0037 001h]                    Reserved : 00
[026h 0038 002h]                      Length : 0020
[028h 0040 004h]      Associated host bridge : 00000035
[02Ch 0044 004h]       Specification version : 00000001
[030h 0048 004h]                    Reserved : 00000000
[034h 0052 008h]               Register base : 0000004004010000
[03Ch 0060 008h]             Register length : 0000000000010000

[044h 0068 001h]               Subtable Type : 00 [CXL Host Bridge Structure]
[045h 0069 001h]                    Reserved : 00
[046h 0070 002h]                      Length : 0020
[048h 0072 004h]      Associated host bridge : 000000BF
[04Ch 0076 004h]       Specification version : 00000001
[050h 0080 004h]                    Reserved : 00000000
[054h 0084 008h]               Register base : 0000004004000000
[05Ch 0092 008h]             Register length : 0000000000010000

[064h 0100 001h]               Subtable Type : 01 [CXL Fixed Memory Window Structure]
[065h 0101 001h]                    Reserved : 00
[066h 0102 002h]                      Length : 0028
[068h 0104 004h]                    Reserved : 00000000
[06Ch 0108 008h]         Window base address : 0000010000000000
[074h 0116 008h]                 Window size : 0000000100000000
[07Ch 0124 001h]    Interleave Members (2^n) : 00
[07Dh 0125 001h]       Interleave Arithmetic : 00
[07Eh 0126 002h]                    Reserved : 0000
[080h 0128 004h]                 Granularity : 00000005
[084h 0132 002h]                Restrictions : 000F
[086h 0134 002h]                       QtgId : 0000
[088h 0136 004h]                First Target : 00000035

[08Ch 0140 001h]               Subtable Type : 01 [CXL Fixed Memory Window Structure]
[08Dh 0141 001h]                    Reserved : 00
[08Eh 0142 002h]                      Length : 002C
[090h 0144 004h]                    Reserved : 00000000
[094h 0148 008h]         Window base address : 0000010100000000
[09Ch 0156 008h]                 Window size : 0000000100000000
[0A4h 0164 001h]    Interleave Members (2^n) : 01
[0A5h 0165 001h]       Interleave Arithmetic : 00
[0A6h 0166 002h]                    Reserved : 0000
[0A8h 0168 004h]                 Granularity : 00000005
[0ACh 0172 002h]                Restrictions : 000F
[0AEh 0174 002h]                       QtgId : 0000
[0B0h 0176 004h]                First Target : 00000035
[0B4h 0180 004h]                 Next Target : 000000BF

Raw Table Data: Length 184 (0xB8)

    0000: 43 45 44 54 B8 00 00 00 01 4E 42 4F 43 48 53 20  // CEDT.....NBOCHS 
    0010: 42 58 50 43 20 20 20 20 01 00 00 00 42 58 50 43  // BXPC    ....BXPC
    0020: 01 00 00 00 00 00 20 00 35 00 00 00 01 00 00 00  // ...... .5.......
    0030: 00 00 00 00 00 00 01 04 40 00 00 00 00 00 01 00  // ........@.......
    0040: 00 00 00 00 00 00 20 00 BF 00 00 00 01 00 00 00  // ...... .........
    0050: 00 00 00 00 00 00 00 04 40 00 00 00 00 00 01 00  // ........@.......
    0060: 00 00 00 00 01 00 28 00 00 00 00 00 00 00 00 00  // ......(.........
    0070: 00 01 00 00 00 00 00 00 01 00 00 00 00 00 00 00  // ................
    0080: 05 00 00 00 0F 00 00 00 35 00 00 00 01 00 2C 00  // ........5.....,.
    0090: 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00 00  // ................
    00A0: 01 00 00 00 01 00 00 00 05 00 00 00 0F 00 00 00  // ................
    00B0: 35 00 00 00 BF 00 00 00                          // 5.......

[ikitayama]

[...]
[  353.612795][  T808] cxl_core:alloc_hpa:651: cxl region5: HPA allocation error (-34) for size:0x0000000080000000 in CXL Window 3 [mem 0xfffffff110000000-0xfffffff18fffffff flags 0x200]
1/1 ndctl:cxl / cxl-region-sysfs.sh        FAIL             8.14s   exit status 1

@djbw on my QEMU emulation (aarch64-softmmu), the CEDT table only have two CFMWSes defined. Is this not the case for q35 emulation?