pmem-ns test: frequent "exit status 237 or signal 109 SIGinvalid"

[marc-hb]

This happens frequently on a fresh QEMU boot and full test suite run. Then it cannot be reproduced. Trying to extract log differences with a successful run:

=================================== 34/49 ====================================
test:         ndctl:ndctl / pmem-ns
start time:   23:45:24
duration:     0.45s
result:       (exit status 237 or signal 109 SIGinvalid)


libndctl: __sysfs_read_attr: failed to open /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus1/region3/persistence_domain: No such file or d>
libndctl: __sysfs_device_parse: region3: processed
ACPI.NFIT unavailable falling back to nfit_test           <-----------
libndctl: __sysfs_device_parse: base: '/sys/class/nd' dev: 'ndctl'
libndctl: __sysfs_read_attr: failed to open /sys/class/nd/ndctl2/device/of_node/compatible: No such file or directory


libndctl: __sysfs_read_attr: failed to open /sys/class/nd/ndctl1/device/firmware/capability: No such file or directory
libndctl: __sysfs_device_parse: ndctl1: processed
/root/ndctl/build/test/pmem-ns: found provider: nfit_test.0    <-----
libndctl: ndctl_bus_wait_probe: waited 11 milliseconds for bus2...
libndctl: __sysfs_device_parse: base: '/sys/devices/platform/nfit_test.0/ndbus2' dev: 'region'



libndctl: __sysfs_device_parse: base: '/sys/devices/platform/nfit_test.0/ndbus2/region7' dev: 'namespace7.'
libndctl: __util_modalias_to_module: alias: nd:t4 module: nd_pmem
libndctl: __sysfs_device_parse: namespace7.0: processed
libndctl: write_attr: failed to open /sys/devices/platform/nfit_test.0/ndbus2/region7/namespace7.0/uuid: No such file or directory
/root/ndctl/build/test/pmem-ns: failed to create PMEM namespace   <------------
libndctl: ndctl_region_disable: region7: disabled
libndctl: ndctl_bus_wait_probe: waited 2 milliseconds for bus2...
libndctl: __sysfs_device_parse: base: '/sys/devices/platform/nfit_test.1/ndbus3' dev: 'region'

Reproduced with ndctl v80, kernel v6.14 + many other combinations. Same test failure observed by other people on other systems.

[marc-hb]

With this test hack the same failure reproduces every time:

--- a/test/pmem_namespaces.c
+++ b/test/pmem_namespaces.c
@@ -189,7 +189,7 @@ int test_pmem_namespaces(int log_level, struct ndctl_test *test,
      bus = NULL;
  }

-  if (!bus) {
+  if (!bus || true) {
    fprintf(stderr, "ACPI.NFIT unavailable falling back to nfit_test\n");
    rc = ndctl_test_init(&kmod_ctx, &mod, NULL, log_level, test);
    ndctl_invalidate(ctx);

[marc-hb]

This is supposed to help:

https://lore.kernel.org/nvdimm/20240709044400.679650-1-alison.schofield@intel.com/
ndctl,cxl/test: Add a common unit test for creating pmem namespaces

I don't yet understand how because I miss the connection with pmem_namespaces.c but I didn't yet try hard to understand.

For sure the test still fails when combining with the test hack above.

[marc-hb]

I don't yet understand how because I miss the connection with pmem_namespaces.c but I didn't yet try hard to understand.

I tried a bit harder and i still don't see the connection. With combined the test hack below, both tests added by this patch fail, yet pmem-ns still passes (at least most of the time).

--- a/test/namespace.sh
+++ b/test/namespace.sh
@@ -27,6 +27,8 @@ do_set_up()
 
 test_namespace_create()
 {
+	return 1
+
 	local mode_list=("raw" "fsdax" "devdax" "sector")
 
 	for ns_mode in "${mode_list[@]}"; do

EDIT: is this all means to just replace pmem-ns and get rid of it?

[AlisonSchofield]

Your right @marc-hb I don't think that patch will help todays pmem-ns intermittent failure. When it fails for me it's on the uuid namespace sysfs attribute. It fails all on it's lonesome too, ie. just run it as the first test and it repeatedly fails. It's not self-sufficient as it should be.

[marc-hb]

When it fails for me it's on the uuid namespace sysfs attribute.

I don't know if that's my problem too.

ie. just run it as the first test and it repeatedly fails

That's definitely not my experience. In my system it fails rarely and never again.

I added some logging and by chance I just captured a failure. Here's the ndctl list ... difference, just before the test starts. From passing to failing.

This was with the most recent ndctl v80 and the gnr-6.6-v37.1 kernel here:
https://github.com/intel-innersource/os.linux.kernel.bkc/commit/19d9bf264a995ecf2b34541dbde2f6e42fe0d9da

Looks like some previous test passes but "damages" region3 somehow.

@@ -1,51 +1,47 @@
 [
   {
     "provider":"ACPI.NFIT",
     "dev":"ndbus1",
     "regions":[
       {
         "dev":"region3",
         "size":"15.00 GiB (16.11 GB)",
         "align":"16.00 MiB (16.78 MB)",
-        "available_size":"15.00 GiB (16.11 GB)",
-        "max_available_extent":"15.00 GiB (16.11 GB)",
+        "available_size":0,
+        "max_available_extent":0,
         "type":"pmem",
-        "iset_id":"0x2468b000123458",
         "persistence_domain":"unknown"
       },
       {
         "dev":"region5",
         "size":"15.00 GiB (16.11 GB)",
         "align":"16.00 MiB (16.78 MB)",
-        "available_size":"15.00 GiB (16.11 GB)",
-        "max_available_extent":"15.00 GiB (16.11 GB)",
+        "available_size":0,
+        "max_available_extent":0,
         "type":"pmem",
-        "iset_id":"0x2468b200123459",
         "persistence_domain":"unknown"
       },
       {
         "dev":"region2",
         "size":"15.00 GiB (16.11 GB)",
         "align":"16.00 MiB (16.78 MB)",
-        "available_size":"15.00 GiB (16.11 GB)",
-        "max_available_extent":"15.00 GiB (16.11 GB)",
+        "available_size":0,
+        "max_available_extent":0,
         "type":"pmem",
-        "iset_id":"0x2468ac00123456",
         "persistence_domain":"unknown"
       },
       {
         "dev":"region4",
         "size":"15.00 GiB (16.11 GB)",
         "align":"16.00 MiB (16.78 MB)",
-        "available_size":"15.00 GiB (16.11 GB)",
-        "max_available_extent":"15.00 GiB (16.11 GB)",
+        "available_size":0,
+        "max_available_extent":0,
         "type":"pmem",
-        "iset_id":"0x2468ae00123457",
         "persistence_domain":"unknown"
       }
     ]
   },
   {
     "provider":"e820",
     "dev":"ndbus0",
     "regions":[

[marc-hb]

I added some logging to test_pmem_namespaces() (it has practically none) and I captured the region states just more failures. The code checked all 4 regions and found none usable. The available_size is again 0 for each and the corresponding namespaces were all disabled. Does that make the ndctl_region_get_nstype(region) == ND_DEVICE_NAMESPACE_PMEM) condition wrong? It looks like it does.

    "regions":[
      {
        "dev":"region3",
        "size":"15.00 GiB (16.11 GB)",
        "align":"16.00 MiB (16.78 MB)",
        "available_size":0,          <---------
        "max_available_extent":0,
        "type":"pmem",
        "mappings":[
          {
            "dimm":"nmem2",
            "offset":"0",
            "length":"0x3c0000000",
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace3.0",
            "mode":"raw",
            "size":"15.00 GiB (16.11 GB)",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      },
      {
        "dev":"region5",
        "size":"15.00 GiB (16.11 GB)",
        "align":"16.00 MiB (16.78 MB)",
        "available_size":0,
        "max_available_extent":0,
        "type":"pmem",
        "mappings":[
          {
            "dimm":"nmem3",
            "offset":"0",
            "length":"0x3c0000000",
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace5.0",
            "mode":"raw",
            "size":"15.00 GiB (16.11 GB)",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      },
      {
        "dev":"region2",
        "size":"15.00 GiB (16.11 GB)",
        "align":"16.00 MiB (16.78 MB)",
        "available_size":0,
        "max_available_extent":0,
        "type":"pmem",
        "mappings":[
          {
            "dimm":"nmem0",
            "offset":"0",
            "length":"0x3c0000000",
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace2.0",
            "mode":"raw",
            "size":"15.00 GiB (16.11 GB)",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      },
      {
        "dev":"region4",
        "size":"15.00 GiB (16.11 GB)",
        "align":"16.00 MiB (16.78 MB)",
        "available_size":0,
        "max_available_extent":0,
        "type":"pmem",
        "mappings":[
          {
            "dimm":"nmem1",
            "offset":"0",
            "length":"0x3c0000000",
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace4.0",
            "mode":"raw",
            "size":"15.00 GiB (16.11 GB)",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      }

[marc-hb]

I added some logging to test_pmem_namespaces()

--- a/test/pmem_namespaces.c
+++ b/test/pmem_namespaces.c
@@ -178,19 +178,37 @@ int test_pmem_namespaces(int log_level, struct ndctl_test *test,
 
        ndctl_set_log_priority(ctx, log_level);
 
+       system("tree /sys/class/nd/");
+/*
+       /sys/class/nd/
+                |-- ndctl0 -> ../../devices/platform/e820_pmem/ndbus0/nd/ndctl0
+                |-- ndctl1 -> ../../devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus1/nd/ndctl1
+*/
+       system("ls -l /sys/bus/nd/drivers/nd_region/");
+       system("grep ^ /sys/class/nd/ndctl*/device/provider");
+       system("ndctl list -BRNDciu");
+
        bus = ndctl_bus_get_by_provider(ctx, "ACPI.NFIT");
        if (bus) {
+               int c = 0;
                /* skip this bus if no label-enabled PMEM regions */
-               ndctl_region_foreach(bus, region)
+               ndctl_region_foreach(bus, region) {
+                       printf("Checking %dth region: %s\n", c, ndctl_region_get_devname(region));
+                       c++;
                        if (ndctl_region_get_nstype(region)
                                        == ND_DEVICE_NAMESPACE_PMEM)
                                break;
-               if (!region)
+               }
+               if (!region) {
+                       fprintf(stderr, "%d ACPI.NFIT region(s) found; no suitable PMEM region, \n", c);
                        bus = NULL;
-       }
+               }
+       } else
+               fprintf(stderr, "ACPI.NFIT unavailable, ");
+
 
-       if (!bus) {
-               fprintf(stderr, "ACPI.NFIT unavailable falling back to nfit_test\n");
+       if (!bus || false) {
+               fprintf(stderr, "falling back to nfit_test\n");
                rc = ndctl_test_init(&kmod_ctx, &mod, NULL, log_level, test);
                ndctl_invalidate(ctx);
                bus = ndctl_bus_get_by_provider(ctx, "nfit_test.0");