ndctl test suite dm.sh failed at cmd `ndctl create-namespace -b ACPI.NFIT -s 1073741824 -t pmem -m fsdax -n test_pmem`

[yizhanglinux]

Hi
I found this issue with 6.17.0-rc4+ and the latest ndctl, here is the log [1]:
[1]

================================== 40/44 ====================================
test:         ndctl:dax / dm.sh
start time:   02:52:53
duration:     19.06s
result:       exit status 1
command:      CXL=/root/ndctl/build/cxl/cxl UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 NDCTL=/root/ndctl/build/ndctl/ndctl DATA_PATH=/root/ndctl/test TEST_PATH=/root/ndctl/build/test MALLOC_PERTURB_=17 LD_LIBRARY_PATH=/root/ndctl/build/cxl/lib:/root/ndctl/build/daxctl/lib:/root/ndctl/build/ndctl/lib MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 DAXCTL=/root/ndctl/build/daxctl/daxctl ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 LC_ALL=C /root/ndctl/test/dm.sh
----------------------------------- stdout -----------------------------------
test/dm.sh: failed at line 48
test/dm.sh: failed at line
----------------------------------- stderr -----------------------------------
+ set -e
+ SKIP=77
+ FAIL=1
+ SUCCESS=0
++ dirname /root/ndctl/test/dm.sh
+ . /root/ndctl/test/common
+++ basename /root/ndctl/test/dm.sh
++ test_basename=dm.sh
++ '[' -z /root/ndctl/build/ndctl/ndctl ']'
++ '[' -z /root/ndctl/build/daxctl/daxctl ']'
++ '[' -z /root/ndctl/build/cxl/cxl ']'
++ '[' -z /root/ndctl/build/test ']'
++ NFIT_TEST_BUS0=nfit_test.0
++ NFIT_TEST_BUS1=nfit_test.1
++ CXL_TEST_BUS=cxl_test
++ ACPI_BUS=ACPI.NFIT
++ E820_BUS=e820
++ CXL_TEST_QOS_CLASS=42
+ MNT=test_dax_mnt
+ TEST_DM_PMEM=/dev/mapper/test_pmem
++ basename /dev/mapper/test_pmem
+ NAME=test_pmem
+ mkdir -p test_dax_mnt
+ TEST_SIZE=1073741824
+ rc=1
+ trap 'err $LINENO cleanup' ERR
+ ndctl list -RBNiD
+ dev=x
++ /root/ndctl/build/ndctl/ndctl create-namespace -b ACPI.NFIT -s 1073741824 -t pmem -m fsdax -n test_pmem
libndctl: ndctl_pfn_enable: pfn1.2: failed to enable
  Error: namespace1.0: failed to enable

failed to create namespace: No such device or address
+ json=
++ err 48 cleanup
++ echo test/dm.sh: failed at line 48
++ '[' -n cleanup ']'
++ cleanup
++ '[' 1 -ne 0 ']'
++ echo 'test/dm.sh: failed at line '
++ mountpoint -q test_dax_mnt
++ '[' -L /dev/mapper/test_pmem ']'
++ rm -rf test_dax_mnt
+++ /root/ndctl/build/ndctl/ndctl list -N
+++ jq -r '.[] | select(.name=="test_pmem") | .dev'
++ namespaces=
++ exit 1
==============================================================================

[2]

# ndctl create-namespace -b ACPI.NFIT -s 1073741824 -t pmem -m fsdax -n test_pmem
libndctl: ndctl_pfn_enable: pfn1.2: failed to enable
  Error: namespace1.0: failed to enable

failed to create namespace: No such device or address
# dmesg
[ 6209.084121] nd_pmem pfn1.2: could not reserve region [??? 0x2070000000-0x20afffffff flags 0x0]
[ 6209.092739] nd_pmem pfn1.2: probe with driver nd_pmem failed with error -16

[yizhanglinux]

Here is the ndctl list output before the test:

[root@storageqe-31 ndctl]# cat ~/dm.dbg
[
  {
    "provider":"ACPI.NFIT",
    "dev":"ndbus1",
    "scrub_state":"idle",
    "dimms":[
      {
        "dev":"nmem1",
        "id":"802c-0f-1730-18311fe3",
        "handle":17,
        "phys_id":4359
      },
      {
        "dev":"nmem3",
        "id":"802c-0f-1730-181d9a41",
        "handle":4113,
        "phys_id":4371
      },
      {
        "dev":"nmem0",
        "id":"802c-0f-1930-22fa8618",
        "handle":1,
        "phys_id":4358
      },
      {
        "dev":"nmem2",
        "id":"802c-0f-1730-181999b3",
        "handle":4097,
        "phys_id":4370
      }
    ],
    "regions":[
      {
        "dev":"region1",
        "size":17179869184,
        "align":16777216,
        "available_size":17163091968,
        "max_available_extent":17112760320,
        "type":"pmem",
        "iset_id":3534749855862159906,
        "mappings":[
          {
            "dimm":"nmem0",
            "offset":0,
            "length":17179869184,
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace1.0",
            "mode":"raw",
            "size":0,
            "uuid":"00000000-0000-0000-0000-000000000000",
            "sector_size":512,
            "state":"disabled",
            "name":"test_pmem"
          },
          {
            "dev":"namespace1.1",
            "mode":"raw",
            "size":0,
            "uuid":"00000000-0000-0000-0000-000000000000",
            "sector_size":512,
            "state":"disabled"
          },
          {
            "dev":"namespace1.3",
            "mode":"raw",
            "size":16777216,
            "uuid":"dbda4866-d50f-465c-8996-1e0cfb7c8af7",
            "sector_size":512,
            "blockdev":"pmem1.3",
            "name":"subsection-test"
          }
        ]
      },
      {
        "dev":"region3",
        "size":17179869184,
        "align":16777216,
        "available_size":17179869184,
        "max_available_extent":17179869184,
        "type":"pmem",
        "iset_id":7436061119495084312,
        "mappings":[
          {
            "dimm":"nmem2",
            "offset":0,
            "length":17179869184,
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace3.0",
            "mode":"raw",
            "size":0,
            "uuid":"00000000-0000-0000-0000-000000000000",
            "sector_size":512,
            "state":"disabled",
            "name":"test_pmem"
          },
          {
            "dev":"namespace3.2",
            "mode":"raw",
            "size":0,
            "uuid":"00000000-0000-0000-0000-000000000000",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      },
      {
        "dev":"region2",
        "size":17179869184,
        "align":16777216,
        "available_size":17179869184,
        "max_available_extent":17179869184,
        "type":"pmem",
        "iset_id":14284963030174216472,
        "mappings":[
          {
            "dimm":"nmem1",
            "offset":0,
            "length":17179869184,
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace2.0",
            "mode":"raw",
            "size":0,
            "uuid":"00000000-0000-0000-0000-000000000000",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      },
      {
        "dev":"region4",
        "size":17179869184,
        "align":16777216,
        "available_size":17179869184,
        "max_available_extent":17179869184,
        "type":"pmem",
        "iset_id":9454245496690973976,
        "mappings":[
          {
            "dimm":"nmem3",
            "offset":0,
            "length":17179869184,
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace4.0",
            "mode":"raw",
            "size":0,
            "uuid":"00000000-0000-0000-0000-000000000000",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      }
    ]
  },
  {
    "provider":"e820",
    "dev":"ndbus0",
    "regions":[
      {
        "dev":"region0",
        "size":4294967296,
        "align":16777216,
        "available_size":0,
        "max_available_extent":0,
        "type":"pmem",
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace0.0",
            "mode":"fsdax",
            "map":"mem",
            "size":4294967296,
            "sector_size":512,
            "blockdev":"pmem0"
          }
        ]
      }
    ]
  }
]

[yizhanglinux]

If I run the test suite for the first time after the system boots up, the test case dm.sh can pass, but it fails at the second time.

[1/70] Generating version.h with a custom command
 1/12 ndctl:ndctl / firmware-update.sh           OK              11.10s
 2/12 ndctl:cxl / cxl-update-firmware.sh         OK              25.83s
 3/12 ndctl:dax / sub-section.sh                 FAIL             0.11s   exit status 1
>>> TEST_PATH=/root/ndctl/build/test MALLOC_PERTURB_=227 DATA_PATH=/root/ndctl/test UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 LD_LIBRARY_PATH=/root/ndctl/build/ndctl/lib:/root/ndctl/build/daxctl/lib:/root/ndctl/build/cxl/lib DAXCTL=/root/ndctl/build/daxctl/daxctl MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 NDCTL=/root/ndctl/build/ndctl/ndctl ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 LC_ALL=C CXL=/root/ndctl/build/cxl/cxl /root/ndctl/test/sub-section.sh

 4/12 ndctl:dax / dax-dev                        OK               0.10s
 5/12 ndctl:dax / dax-ext4.sh                    OK               2.71s
 6/12 ndctl:dax / dax-xfs.sh                     OK               2.87s
 7/12 ndctl:ndctl / align.sh                     SKIP             0.06s   exit status 77
 8/12 ndctl:dax / device-dax                     OK               7.50s
 9/12 ndctl:dax / revoke-devmem                  OK               0.15s
10/12 ndctl:dax / daxctl-devices.sh              FAIL            20.10s   exit status 1
>>> MALLOC_PERTURB_=220 TEST_PATH=/root/ndctl/build/test DATA_PATH=/root/ndctl/test UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 LD_LIBRARY_PATH=/root/ndctl/build/ndctl/lib:/root/ndctl/build/daxctl/lib:/root/ndctl/build/cxl/lib DAXCTL=/root/ndctl/build/daxctl/daxctl MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 NDCTL=/root/ndctl/build/ndctl/ndctl ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 LC_ALL=C CXL=/root/ndctl/build/cxl/cxl /root/ndctl/test/daxctl-devices.sh

11/12 ndctl:dax / dm.sh                          FAIL            19.34s   exit status 1
>>> TEST_PATH=/root/ndctl/build/test DATA_PATH=/root/ndctl/test UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 LD_LIBRARY_PATH=/root/ndctl/build/ndctl/lib:/root/ndctl/build/daxctl/lib:/root/ndctl/build/cxl/lib DAXCTL=/root/ndctl/build/daxctl/daxctl MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MALLOC_PERTURB_=28 NDCTL=/root/ndctl/build/ndctl/ndctl ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 LC_ALL=C CXL=/root/ndctl/build/cxl/cxl /root/ndctl/test/dm.sh

12/12 ndctl:cxl / cxl-features.sh                SKIP             0.16s   exit status 77

Ok:                 7
Expected Fail:      0
Fail:               3
Unexpected Pass:    0
Skipped:            2
Timeout:            0

[AlisonSchofield]

Hi Yi Zhang,

Thanks for reporting this issue and the others while running the unit tests. I see 4 tests failing:
dm.sh, align.sh, sub-section.sh, and daxctl-devices.sh

I'm not able to reproduce any of these using 6.17-rc4 or -rc6 using ndctl from the pending branch.
If you are not using pending, there is one change, to use the built ndctl, that could affect sub-section.sh.

Please let me know what version of ndctl you are using.

I'll look into each of them separately but it is weird that what was once working is now failing for 4 tests.

--Alison

[yizhanglinux]

Hi Alison
I already use the ndctl pending branch for the test, btw, the nvdimm I used is nvdimm-n on one DELL server, here is the ndctl output[1].
I tried the Intel NVDIMM on one DELL server today, the issue and the other issues I reported recently (align.sh, sub-section.sh, and daxctl-devices.sh) cannot be reproduced.
[1]

[root@storageqe-31 ~]# ndctl list -RBNiD
[
  {
    "provider":"ACPI.NFIT",
    "dev":"ndbus1",
    "scrub_state":"active",
    "dimms":[
      {
        "dev":"nmem1",
        "id":"802c-0f-1730-18311fe3",
        "handle":17,
        "phys_id":4359
      },
      {
        "dev":"nmem3",
        "id":"802c-0f-1730-181d9a41",
        "handle":4113,
        "phys_id":4371
      },
      {
        "dev":"nmem0",
        "id":"802c-0f-1930-22fa8618",
        "handle":1,
        "phys_id":4358
      },
      {
        "dev":"nmem2",
        "id":"802c-0f-1730-181999b3",
        "handle":4097,
        "phys_id":4370
      }
    ],
    "regions":[
      {
        "dev":"region1",
        "size":17179869184,
        "align":16777216,
        "available_size":17179869184,
        "max_available_extent":17179869184,
        "type":"pmem",
        "iset_id":3534749855862159906,
        "mappings":[
          {
            "dimm":"nmem0",
            "offset":0,
            "length":17179869184,
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace1.0",
            "mode":"raw",
            "size":0,
            "uuid":"00000000-0000-0000-0000-000000000000",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      },
      {
        "dev":"region3",
        "size":17179869184,
        "align":16777216,
        "available_size":17179869184,
        "max_available_extent":17179869184,
        "type":"pmem",
        "iset_id":7436061119495084312,
        "mappings":[
          {
            "dimm":"nmem2",
            "offset":0,
            "length":17179869184,
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace3.0",
            "mode":"raw",
            "size":0,
            "uuid":"00000000-0000-0000-0000-000000000000",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      },
      {
        "dev":"region2",
        "size":17179869184,
        "align":16777216,
        "available_size":17179869184,
        "max_available_extent":17179869184,
        "type":"pmem",
        "iset_id":14284963030174216472,
        "mappings":[
          {
            "dimm":"nmem1",
            "offset":0,
            "length":17179869184,
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace2.0",
            "mode":"raw",
            "size":0,
            "uuid":"00000000-0000-0000-0000-000000000000",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      },
      {
        "dev":"region4",
        "size":17179869184,
        "align":16777216,
        "available_size":17179869184,
        "max_available_extent":17179869184,
        "type":"pmem",
        "iset_id":9454245496690973976,
        "mappings":[
          {
            "dimm":"nmem3",
            "offset":0,
            "length":17179869184,
            "position":0
          }
        ],
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace4.0",
            "mode":"raw",
            "size":0,
            "uuid":"00000000-0000-0000-0000-000000000000",
            "sector_size":512,
            "state":"disabled"
          }
        ]
      }
    ]
  },
  {
    "provider":"e820",
    "dev":"ndbus0",
    "regions":[
      {
        "dev":"region0",
        "size":4294967296,
        "align":16777216,
        "available_size":0,
        "max_available_extent":0,
        "type":"pmem",
        "persistence_domain":"unknown",
        "namespaces":[
          {
            "dev":"namespace0.0",
            "mode":"fsdax",
            "map":"mem",
            "size":4294967296,
            "sector_size":512,
            "blockdev":"pmem0"
          }
        ]
      }
    ]
  }
]

[djbw]

@yizhanglinux I believe this test assumes that the NVDIMM under test supports namespace labels. To be clear this test suite is meant to be run against canned configurations from QEMU, it is not meant to be run against hardware. The only hardware it might work against is Intel NVDIMMs, but other than that I think the fix here is to make the test SKIP instead of FAIL when run against unexpected hardware configurations.

[yizhanglinux]

That looks reasonable, both the nvdimm-n and intel nvdimm have the "provider":"ACPI.NFIT", is there a way to distinguish them? If so, we can update the tests to skip on nvdimm-n HW.

[djbw]

That looks reasonable, both the nvdimm-n and intel nvdimm have the "provider":"ACPI.NFIT", is there a way to distinguish them? If so, we can update the tests to skip on nvdimm-n HW.

Perhaps something like this attached (UNTESTED!)

check-label.patch

[yizhanglinux]

I tried it on the nvdimm-n[1] and intel nvdimm server[2], found both of them support labels.

[1] nvdimm-n

[root@storageqe-31 ndctl]# tail /sys/bus/nd/devices/nmem*/available_slots
==> /sys/bus/nd/devices/nmem0/available_slots <==
3

==> /sys/bus/nd/devices/nmem1/available_slots <==
3

==> /sys/bus/nd/devices/nmem2/available_slots <==
3

==> /sys/bus/nd/devices/nmem3/available_slots <==
3
[root@storageqe-31 ndctl]# ndctl check-labels nmem0
successfully verified 1 nmem label
[root@storageqe-31 ndctl]# ndctl check-labels nmem1
successfully verified 1 nmem label
[root@storageqe-31 ndctl]# ndctl check-labels nmem2
successfully verified 1 nmem label
[root@storageqe-31 ndctl]# ndctl check-labels nmem3
successfully verified 1 nmem label

[2] Intel nvdimm

[root@storageqe-36 ~]# tail /sys/bus/nd/devices/nmem*/available_slots
==> /sys/bus/nd/devices/nmem0/available_slots <==
509

==> /sys/bus/nd/devices/nmem1/available_slots <==
509

[djbw]

Oh interesting... nvdimm-n do have a few label slots, and I missed that you said dm.sh passes sometimes. So that has me looking at expectations that get violated in other tests. I notice sub-section.sh fails and that test tries to create 10 namespaces. So that one should have a patch like this (UNTESTED):

patch1.patch

...and I would be interested to know if skipping sub-section.sh makes the dm.sh more reliable.

[yizhanglinux]

Oh interesting... nvdimm-n do have a few label slots, and I missed that you said dm.sh passes sometimes. So that has me looking at expectations that get violated in other tests. I notice sub-section.sh fails and that test tries to create 10 namespaces. So that one should have a patch like this (UNTESTED):

patch1.patch

...and I would be interested to know if skipping sub-section.sh makes the dm.sh more reliable.

The patch can make sub-section.sh pass now, but the dm.sh still failed.

[yizhanglinux]

I have more finding today:
The fist time I execute[1], the two cases(firmware-update.sh and dm.sh) passed.
The second time executing three cases: daxctl-devices.sh failed
The third time executing the same three cases: both daxctl-devices.sh and dm.sh failed.

Seems it was due to the failed daxctl-devices.sh on [2] and firmware-update.sh on [3] that lead dm.sh

[1]

1/2 ndctl:ndctl / firmware-update.sh        OK              11.16s
2/2 ndctl:dax / dm.sh                       OK              38.36s
dmesg:
[  311.937852] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0x0000000100000000-0x0000000101ffffff]
[  311.937862] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0x0000000100000000-0x0000000101ffffff]
[  311.937880] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0x0000000108000000-0x000000010bffffff]
[  311.937882] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0x0000000108000000-0x000000010bffffff]
[  311.937892] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd0f4a3905000-0xffffd0f4a390500b]
[  311.937894] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd0f4a3905000-0xffffd0f4a390500b]
[  311.937897] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd0f4a39c1000-0xffffd0f4a39c100b]
[  311.937899] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd0f4a39c1000-0xffffd0f4a39c100b]
[  311.937902] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd0f4a3a0d000-0xffffd0f4a3a0d00b]
[  311.937903] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd0f4a3a0d000-0xffffd0f4a3a0d00b]
[  311.937906] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd0f4a3acd000-0xffffd0f4a3acd00b]
[  311.937908] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd0f4a3acd000-0xffffd0f4a3acd00b]
[  311.937978] nfit_test nfit_test.0: failed to evaluate _FIT
[  311.948878] nfit_test nfit_test.1: Error found in NVDIMM nmem8 flags: save_fail restore_fail flush_fail not_armed
[  311.959140] nfit_test nfit_test.1: Error found in NVDIMM nmem9 flags: map_fail
[  311.966369] nfit_test nfit_test.1: changing numa node from -1 to 0 for nfit region [0x0000000140000000-0x0000000141ffffff]
[  311.966372] nfit_test nfit_test.1: changing target node from -1 to 0 for nfit region [0x0000000140000000-0x0000000141ffffff]
[  311.966387] nfit_test nfit_test.1: changing numa node from -1 to 0 for nfit region [0xffffd0f4a55fc000-0xffffd0f4a59fbfff]
[  311.966390] nfit_test nfit_test.1: changing target node from -1 to 0 for nfit region [0xffffd0f4a55fc000-0xffffd0f4a59fbfff]
[  311.966781] nd_pmem namespace8.0: unable to guarantee persistence of writes
[  311.967069] nd_pmem namespace7.0: region7 read-only, marking pmem7 read-only
[  321.444525] Freezing user space processes
[  321.445921] Freezing user space processes completed (elapsed 0.001 seconds)
[  321.445929] OOM killer disabled.
[  321.445931] Freezing remaining freezable tasks
[  321.447479] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[  321.448602] printk: Suspending console(s) (use no_console_suspend to debug)
[  321.451510] serial 00:03: disabled
[  321.451559] serial 00:02: disabled
[  321.536422] megaraid_sas 0000:18:00.0: megasas_suspend is called
[  321.817559] megaraid_sas 0000:18:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  321.841954] megaraid_sas 0000:18:00.0: megasas_resume is called
[  321.841960] megaraid_sas 0000:18:00.0: Waiting for FW to come to ready state
[  321.841964] megaraid_sas 0000:18:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  321.844627] tg3 0000:01:00.0 eno3: Link is down
[  321.851744] power_meter ACPI000D:00: Found ACPI power meter.
[  321.852661] serial 00:02: activated
[  321.853029] serial 00:03: activated
[  321.905645] megaraid_sas 0000:18:00.0: FW now in Ready state
[  321.905648] megaraid_sas 0000:18:00.0: 63 bit DMA mask and 32 bit consistent mask
[  321.906450] megaraid_sas 0000:18:00.0: requested/available msix 49/49 poll_queue 0
[  321.906541] megaraid_sas 0000:18:00.0: Performance mode :Latency (latency index = 1)
[  321.906544] megaraid_sas 0000:18:00.0: FW supports sync cache	: No
[  321.906548] megaraid_sas 0000:18:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  321.980910] hub 1-14.4:1.0: hub_ext_port_status failed (err = -71)
[  321.980911] hub 1-14.1:1.0: hub_ext_port_status failed (err = -71)
[  322.139681] megaraid_sas 0000:18:00.0: FW provided supportMaxExtLDs: 0	max_lds: 32
[  322.139690] megaraid_sas 0000:18:00.0: controller type	: iMR(0MB)
[  322.139695] megaraid_sas 0000:18:00.0: Online Controller Reset(OCR)	: Enabled
[  322.139700] megaraid_sas 0000:18:00.0: Secure JBOD support	: No
[  322.139703] megaraid_sas 0000:18:00.0: NVMe passthru support	: No
[  322.139707] megaraid_sas 0000:18:00.0: FW provided TM TaskAbort/Reset timeout	: 0 secs/0 secs
[  322.139712] megaraid_sas 0000:18:00.0: JBOD sequence map support	: No
[  322.139715] megaraid_sas 0000:18:00.0: PCI Lane Margining support	: No
[  322.140404] megaraid_sas 0000:18:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[  322.140410] megaraid_sas 0000:18:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5803
[  322.163164] ata3: SATA link down (SStatus 0 SControl 300)
[  322.163686] ata13: SATA link down (SStatus 0 SControl 300)
[  322.164023] ata6: SATA link down (SStatus 0 SControl 300)
[  322.164065] ata7: SATA link down (SStatus 0 SControl 300)
[  322.164121] ata14: SATA link down (SStatus 0 SControl 300)
[  322.164158] ata11: SATA link down (SStatus 0 SControl 300)
[  322.164158] ata1: SATA link down (SStatus 0 SControl 300)
[  322.164186] ata10: SATA link down (SStatus 0 SControl 300)
[  322.164607] ata4: SATA link down (SStatus 0 SControl 300)
[  322.164650] ata12: SATA link down (SStatus 0 SControl 300)
[  322.166050] ata2: SATA link down (SStatus 0 SControl 300)
[  322.166094] ata5: SATA link down (SStatus 0 SControl 300)
[  322.172542] ata8: SATA link down (SStatus 0 SControl 300)
[  322.196810] ata9: SATA link down (SStatus 0 SControl 300)
[  322.639470] bnxt_en 0000:19:00.1 eno2np1: NIC Link is Down
[  322.641664] bnxt_en 0000:19:00.0 eno1np0: NIC Link is Down
[  322.667664] Restarting kernel threads ...
[  322.668252] Done restarting kernel threads.
[  322.668823] OOM killer enabled.
[  322.668824] Restarting tasks: Starting
[  322.668969] Restarting tasks: Done
[  326.790298] tg3 0000:01:00.0 eno3: Link is up at 1000 Mbps, full duplex
[  326.790302] tg3 0000:01:00.0 eno3: Flow control is on for TX and on for RX
[  326.790304] tg3 0000:01:00.0 eno3: EEE is disabled
[  336.952532] nd_pmem namespace0.0: unable to guarantee persistence of writes
[  351.283795] EXT4-fs (dm-3): mounted filesystem d8bea094-fcc2-4b77-a71b-2f3f7a212fd0 r/w with ordered data mode. Quota mode: none.
[  351.288603] EXT4-fs (dm-3): unmounting filesystem d8bea094-fcc2-4b77-a71b-2f3f7a212fd0.

[2]

1/3 ndctl:ndctl / firmware-update.sh        OK              11.09s
2/3 ndctl:dax / daxctl-devices.sh           FAIL            14.32s   exit status 1
>>> NDCTL=/root/ndctl/build/ndctl/ndctl DATA_PATH=/root/ndctl/test MALLOC_PERTURB_=50 LD_LIBRARY_PATH=/root/ndctl/build/ndctl/lib:/root/ndctl/build/daxctl/lib:/root/ndctl/build/cxl/lib LC_ALL=C DAXCTL=/root/ndctl/build/daxctl/daxctl UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 TEST_PATH=/root/ndctl/build/test CXL=/root/ndctl/build/cxl/cxl /root/ndctl/test/daxctl-devices.sh

3/3 ndctl:dax / dm.sh                       OK              38.38s

daxctl-devcies.sh failed as 
+ [[ system-ram == \s\y\s\t\e\m\-\r\a\m ]]
+ /root/ndctl/build/daxctl/daxctl online-memory dax1.0
dax1.0:
  WARNING: detected a race while onlining memory
  Some memory may not be in the expected zone. It is
  recommended to disable any other onlining mechanisms,
  and retry. If onlining is to be left to other agents,
  use the --no-online option to suppress this warning
dax1.0: all memory sections (1) already online
onlined memory for 0 devices

dmesg:
[  603.344725] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0x0000000100000000-0x0000000101ffffff]
[  603.344734] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0x0000000100000000-0x0000000101ffffff]
[  603.344751] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0x0000000108000000-0x000000010bffffff]
[  603.344753] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0x0000000108000000-0x000000010bffffff]
[  603.344763] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd0f4a39c1000-0xffffd0f4a39c100b]
[  603.344766] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd0f4a39c1000-0xffffd0f4a39c100b]
[  603.344769] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd0f4a3a75000-0xffffd0f4a3a7500b]
[  603.344771] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd0f4a3a75000-0xffffd0f4a3a7500b]
[  603.344774] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd0f4a3acd000-0xffffd0f4a3acd00b]
[  603.344776] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd0f4a3acd000-0xffffd0f4a3acd00b]
[  603.344779] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd0f4a3b05000-0xffffd0f4a3b0500b]
[  603.344780] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd0f4a3b05000-0xffffd0f4a3b0500b]
[  603.344853] nfit_test nfit_test.0: failed to evaluate _FIT
[  603.354730] nfit_test nfit_test.1: Error found in NVDIMM nmem8 flags: save_fail restore_fail flush_fail not_armed
[  603.364989] nfit_test nfit_test.1: Error found in NVDIMM nmem9 flags: map_fail
[  603.372217] nfit_test nfit_test.1: changing numa node from -1 to 0 for nfit region [0x0000000140000000-0x0000000141ffffff]
[  603.372220] nfit_test nfit_test.1: changing target node from -1 to 0 for nfit region [0x0000000140000000-0x0000000141ffffff]
[  603.372233] nfit_test nfit_test.1: changing numa node from -1 to 0 for nfit region [0xffffd0f4a576e000-0xffffd0f4a5b6dfff]
[  603.372235] nfit_test nfit_test.1: changing target node from -1 to 0 for nfit region [0xffffd0f4a576e000-0xffffd0f4a5b6dfff]
[  603.372676] nd_pmem namespace8.0: unable to guarantee persistence of writes
[  603.373164] nd_pmem namespace7.0: region7 read-only, marking pmem7 read-only
[  612.807960] Freezing user space processes
[  612.809482] Freezing user space processes completed (elapsed 0.001 seconds)
[  612.809489] OOM killer disabled.
[  612.809491] Freezing remaining freezable tasks
[  612.811013] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[  612.812055] printk: Suspending console(s) (use no_console_suspend to debug)
[  612.815383] serial 00:03: disabled
[  612.815467] serial 00:02: disabled
[  612.902144] megaraid_sas 0000:18:00.0: megasas_suspend is called
[  613.181938] megaraid_sas 0000:18:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  613.208002] megaraid_sas 0000:18:00.0: megasas_resume is called
[  613.208007] megaraid_sas 0000:18:00.0: Waiting for FW to come to ready state
[  613.208010] megaraid_sas 0000:18:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  613.210531] tg3 0000:01:00.0 eno3: Link is down
[  613.217424] power_meter ACPI000D:00: Found ACPI power meter.
[  613.218972] serial 00:02: activated
[  613.219561] serial 00:03: activated
[  613.273263] megaraid_sas 0000:18:00.0: FW now in Ready state
[  613.273269] megaraid_sas 0000:18:00.0: 63 bit DMA mask and 32 bit consistent mask
[  613.274359] megaraid_sas 0000:18:00.0: requested/available msix 49/49 poll_queue 0
[  613.274470] megaraid_sas 0000:18:00.0: Performance mode :Latency (latency index = 1)
[  613.274474] megaraid_sas 0000:18:00.0: FW supports sync cache	: No
[  613.274478] megaraid_sas 0000:18:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  613.305751] hub 1-14.1:1.0: hub_ext_port_status failed (err = -71)
[  613.305826] hub 1-14.4:1.0: hub_ext_port_status failed (err = -71)
[  613.505213] megaraid_sas 0000:18:00.0: FW provided supportMaxExtLDs: 0	max_lds: 32
[  613.505222] megaraid_sas 0000:18:00.0: controller type	: iMR(0MB)
[  613.505227] megaraid_sas 0000:18:00.0: Online Controller Reset(OCR)	: Enabled
[  613.505231] megaraid_sas 0000:18:00.0: Secure JBOD support	: No
[  613.505235] megaraid_sas 0000:18:00.0: NVMe passthru support	: No
[  613.505238] megaraid_sas 0000:18:00.0: FW provided TM TaskAbort/Reset timeout	: 0 secs/0 secs
[  613.505243] megaraid_sas 0000:18:00.0: JBOD sequence map support	: No
[  613.505246] megaraid_sas 0000:18:00.0: PCI Lane Margining support	: No
[  613.505930] megaraid_sas 0000:18:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[  613.505937] megaraid_sas 0000:18:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5803
[  613.530081] ata7: SATA link down (SStatus 0 SControl 300)
[  613.531585] ata4: SATA link down (SStatus 0 SControl 300)
[  613.562211] ata6: SATA link down (SStatus 0 SControl 300)
[  613.562718] ata12: SATA link down (SStatus 0 SControl 300)
[  613.562744] ata8: SATA link down (SStatus 0 SControl 300)
[  613.562883] ata5: SATA link down (SStatus 0 SControl 300)
[  613.562891] ata10: SATA link down (SStatus 0 SControl 300)
[  613.562913] ata1: SATA link down (SStatus 0 SControl 300)
[  613.562920] ata14: SATA link down (SStatus 0 SControl 300)
[  613.562941] ata3: SATA link down (SStatus 0 SControl 300)
[  613.562995] ata13: SATA link down (SStatus 0 SControl 300)
[  613.562995] ata2: SATA link down (SStatus 0 SControl 300)
[  613.563026] ata9: SATA link down (SStatus 0 SControl 300)
[  613.563053] ata11: SATA link down (SStatus 0 SControl 300)
[  613.979989] bnxt_en 0000:19:00.0 eno1np0: NIC Link is Down
[  613.982587] bnxt_en 0000:19:00.1 eno2np1: NIC Link is Down
[  614.008636] Restarting kernel threads ...
[  614.009578] Done restarting kernel threads.
[  614.009990] OOM killer enabled.
[  614.009993] Restarting tasks: Starting
[  614.010250] Restarting tasks: Done
[  614.402624] nd_pmem namespace0.0: unable to guarantee persistence of writes
[  618.255667] tg3 0000:01:00.0 eno3: Link is up at 1000 Mbps, full duplex
[  618.255676] tg3 0000:01:00.0 eno3: Flow control is on for TX and on for RX
[  618.255679] tg3 0000:01:00.0 eno3: EEE is disabled
[  628.593640] kmem dax1.0: DAX region truncated by 1.94 GiB due to alignment
[  628.634905] Fallback order for Node 0: 0 1
[  628.634911] Fallback order for Node 1: 1 0
[  628.634925] Built 2 zonelists, mobility grouping on.  Total pages: 65044931
[  628.634928] Policy zone: Normal
[  656.979162] EXT4-fs (dm-3): mounted filesystem 5ce8eb99-00f8-47c1-a6e4-cd22b59434f7 r/w with ordered data mode. Quota mode: none.
[  656.984671] EXT4-fs (dm-3): unmounting filesystem 5ce8eb99-00f8-47c1-a6e4-cd22b59434f7.

[3]

1/3 ndctl:ndctl / firmware-update.sh        OK              11.11s
2/3 ndctl:dax / daxctl-devices.sh           FAIL            24.16s   exit status 1
>>> TEST_PATH=/root/ndctl/build/test LC_ALL=C ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 NDCTL=/root/ndctl/build/ndctl/ndctl DAXCTL=/root/ndctl/build/daxctl/daxctl DATA_PATH=/root/ndctl/test MALLOC_PERTURB_=106 LD_LIBRARY_PATH=/root/ndctl/build/daxctl/lib:/root/ndctl/build/ndctl/lib:/root/ndctl/build/cxl/lib CXL=/root/ndctl/build/cxl/cxl MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 /root/ndctl/test/daxctl-devices.sh

3/3 ndctl:dax / dm.sh                       FAIL            19.33s   exit status 1
>>> TEST_PATH=/root/ndctl/build/test LC_ALL=C ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 NDCTL=/root/ndctl/build/ndctl/ndctl DAXCTL=/root/ndctl/build/daxctl/daxctl MALLOC_PERTURB_=40 DATA_PATH=/root/ndctl/test LD_LIBRARY_PATH=/root/ndctl/build/daxctl/lib:/root/ndctl/build/ndctl/lib:/root/ndctl/build/cxl/lib CXL=/root/ndctl/build/cxl/cxl MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 /root/ndctl/test/dm.sh

dmesg:
[  278.594172] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0x0000000100000000-0x0000000101ffffff]
[  278.594187] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0x0000000100000000-0x0000000101ffffff]
[  278.594229] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0x0000000108000000-0x000000010bffffff]
[  278.594236] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0x0000000108000000-0x000000010bffffff]
[  278.594257] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd2ea63981000-0xffffd2ea6398100b]
[  278.594264] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd2ea63981000-0xffffd2ea6398100b]
[  278.594272] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd2ea639b9000-0xffffd2ea639b900b]
[  278.594278] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd2ea639b9000-0xffffd2ea639b900b]
[  278.594285] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd2ea639f9000-0xffffd2ea639f900b]
[  278.594291] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd2ea639f9000-0xffffd2ea639f900b]
[  278.594298] nfit_test nfit_test.0: changing numa node from -1 to 0 for nfit region [0xffffd2ea63a1a000-0xffffd2ea63a1a00b]
[  278.594303] nfit_test nfit_test.0: changing target node from -1 to 0 for nfit region [0xffffd2ea63a1a000-0xffffd2ea63a1a00b]
[  278.594392] nfit_test nfit_test.0: failed to evaluate _FIT
[  278.608474] nfit_test nfit_test.1: Error found in NVDIMM nmem8 flags: save_fail restore_fail flush_fail not_armed
[  278.618736] nfit_test nfit_test.1: Error found in NVDIMM nmem9 flags: map_fail
[  278.625962] nfit_test nfit_test.1: changing numa node from -1 to 0 for nfit region [0x0000000140000000-0x0000000141ffffff]
[  278.625965] nfit_test nfit_test.1: changing target node from -1 to 0 for nfit region [0x0000000140000000-0x0000000141ffffff]
[  278.625982] nfit_test nfit_test.1: changing numa node from -1 to 0 for nfit region [0xffffd2ea64a10000-0xffffd2ea64e0ffff]
[  278.625984] nfit_test nfit_test.1: changing target node from -1 to 0 for nfit region [0xffffd2ea64a10000-0xffffd2ea64e0ffff]
[  278.626206] nd_pmem namespace8.0: unable to guarantee persistence of writes
[  278.626964] nd_pmem namespace7.0: region7 read-only, marking pmem7 read-only
[  288.075860] Freezing user space processes
[  288.077425] Freezing user space processes completed (elapsed 0.001 seconds)
[  288.077429] OOM killer disabled.
[  288.077432] Freezing remaining freezable tasks
[  288.078859] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[  288.079873] printk: Suspending console(s) (use no_console_suspend to debug)
[  288.082571] serial 00:03: disabled
[  288.082627] serial 00:02: disabled
[  288.175479] megaraid_sas 0000:18:00.0: megasas_suspend is called
[  288.456633] megaraid_sas 0000:18:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  288.481407] megaraid_sas 0000:18:00.0: megasas_resume is called
[  288.481417] megaraid_sas 0000:18:00.0: Waiting for FW to come to ready state
[  288.481422] megaraid_sas 0000:18:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  288.484174] tg3 0000:01:00.0 eno3: Link is down
[  288.489592] power_meter ACPI000D:00: Found ACPI power meter.
[  288.490573] serial 00:02: activated
[  288.490839] serial 00:03: activated
[  288.608828] megaraid_sas 0000:18:00.0: FW now in Ready state
[  288.608835] megaraid_sas 0000:18:00.0: 63 bit DMA mask and 32 bit consistent mask
[  288.610182] megaraid_sas 0000:18:00.0: requested/available msix 49/49 poll_queue 0
[  288.610349] megaraid_sas 0000:18:00.0: Performance mode :Latency (latency index = 1)
[  288.610355] megaraid_sas 0000:18:00.0: FW supports sync cache	: No
[  288.610362] megaraid_sas 0000:18:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  288.620376] hub 1-14.4:1.0: hub_ext_port_status failed (err = -71)
[  288.620376] hub 1-14.1:1.0: hub_ext_port_status failed (err = -71)
[  288.800021] ata6: SATA link down (SStatus 0 SControl 300)
[  288.800546] ata4: SATA link down (SStatus 0 SControl 300)
[  288.800578] ata1: SATA link down (SStatus 0 SControl 300)
[  288.800697] ata3: SATA link down (SStatus 0 SControl 300)
[  288.800739] ata5: SATA link down (SStatus 0 SControl 300)
[  288.800765] ata2: SATA link down (SStatus 0 SControl 300)
[  288.808040] ata8: SATA link down (SStatus 0 SControl 300)
[  288.808730] ata11: SATA link down (SStatus 0 SControl 300)
[  288.808770] ata7: SATA link down (SStatus 0 SControl 300)
[  288.808796] ata13: SATA link down (SStatus 0 SControl 300)
[  288.808823] ata10: SATA link down (SStatus 0 SControl 300)
[  288.808850] ata14: SATA link down (SStatus 0 SControl 300)
[  288.808877] ata12: SATA link down (SStatus 0 SControl 300)
[  288.836310] ata9: SATA link down (SStatus 0 SControl 300)
[  288.843265] megaraid_sas 0000:18:00.0: FW provided supportMaxExtLDs: 0	max_lds: 32
[  288.843274] megaraid_sas 0000:18:00.0: controller type	: iMR(0MB)
[  288.843279] megaraid_sas 0000:18:00.0: Online Controller Reset(OCR)	: Enabled
[  288.843283] megaraid_sas 0000:18:00.0: Secure JBOD support	: No
[  288.843287] megaraid_sas 0000:18:00.0: NVMe passthru support	: No
[  288.843290] megaraid_sas 0000:18:00.0: FW provided TM TaskAbort/Reset timeout	: 0 secs/0 secs
[  288.843294] megaraid_sas 0000:18:00.0: JBOD sequence map support	: No
[  288.843297] megaraid_sas 0000:18:00.0: PCI Lane Margining support	: No
[  288.844010] megaraid_sas 0000:18:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[  288.844016] megaraid_sas 0000:18:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5803
[  289.249614] bnxt_en 0000:19:00.1 eno2np1: NIC Link is Down
[  289.251783] bnxt_en 0000:19:00.0 eno1np0: NIC Link is Down
[  289.277891] Restarting kernel threads ...
[  289.278521] Done restarting kernel threads.
[  289.278882] OOM killer enabled.
[  289.278883] Restarting tasks: Starting
[  289.279019] Restarting tasks: Done

[ 289.561524] removing memory fails, because memory [0x0000002080000000-0x00000020ffffffff] is onlined

[  289.570693] kmem dax1.0: mapping0: 0x2080000000-0x20ffffffff cannot be hotremoved until the next reboot
[  293.443750] tg3 0000:01:00.0 eno3: Link is up at 1000 Mbps, full duplex
[  293.443758] tg3 0000:01:00.0 eno3: Flow control is on for TX and on for RX
[  293.443760] tg3 0000:01:00.0 eno3: EEE is disabled
[  293.450722] nd_pmem namespace0.0: unable to guarantee persistence of writes
[  293.483779] nd_pmem namespace1.0: could not reserve region [??? 0x2070000000-0x216fffffff flags 0x0]
[  293.492920] nd_pmem namespace1.0: probe with driver nd_pmem failed with error -16
[  293.500872] nd_pmem namespace1.0: could not reserve region [??? 0x2070000000-0x216fffffff flags 0x0]
[  293.510003] nd_pmem namespace1.0: probe with driver nd_pmem failed with error -16
[  308.703080] kmem dax1.0: DAX region truncated by 1.94 GiB due to alignment
[  308.709978] kmem dax1.0: mapping0: 0x2080000000-0x20ffffffff could not reserve region
[  308.717810] kmem dax1.0: probe with driver kmem failed with error -16
[  308.729743] device_dax dax1.0: mapping0: 0x2074200000-0x216fffffff could not reserve range
[  308.738009] device_dax dax1.0: probe with driver device_dax failed with error -16
[  308.796906] kmem dax1.0: DAX region truncated by 1.94 GiB due to alignment
[  308.803810] kmem dax1.0: mapping0: 0x2080000000-0x20ffffffff could not reserve region
[  308.811647] kmem dax1.0: probe with driver kmem failed with error -16
[  308.818104] kmem dax1.0: DAX region truncated by 1.94 GiB due to alignment
[  308.824977] kmem dax1.0: mapping0: 0x2080000000-0x20ffffffff could not reserve region
[  308.832811] kmem dax1.0: probe with driver kmem failed with error -16
[  312.752909] nd_pmem namespace1.0: could not reserve region [??? 0x2070000000-0x216fffffff flags 0x0]
[  312.762042] nd_pmem namespace1.0: probe with driver nd_pmem failed with error -16
[  315.809601] sched: DL replenish lagged too much
[  328.002872] nd_pmem pfn1.1: could not reserve region [??? 0x2070000000-0x20afffffff flags 0x0]
[  328.011488] nd_pmem pfn1.1: probe with driver nd_pmem failed with error -16