Add memspace "highest bandwidth"

[kswiecicki]

Description

This memspace contains memory targets ordered by the highest bandwidth value (targets higher in the ordering are prioritised).
There is also an alternative way to assign nodes to this memspace by setting UMF_MEMSPACE_HIGHEST_BANDWIDTH environment variable.

TODO

• Add tests for "highest bandwidth" memspace from UMF_MEMSPACE_HIGHEST_BANDWIDTH
• Verify functionality on QEMU runner

Checklist

• Code compiles without errors locally
• All tests pass locally
• CI workflows execute properly

• CI workflows, not executed per PR (e.g. Nightly), execute properly
• New tests added, especially if they will fail without my changes
• Extended the README/documentation
• All newly added source files have a license
• All newly added source files are referenced in CMake files
• Logger (with debug/info/... messages) is used

[kswiecicki]

To use HWLOC property query API, we need the the lib version >= 2.3.0. Unfortunately on the Ubuntu 20.04 runners the latest available package is version 2.1.0 (https://packages.ubuntu.com/search?suite=all&searchon=names&keywords=hwloc). @lukaszstolarczuk do you think we could use a script to build and install HWLOC from source instead of relying on the package manager? Or alternatively install the package from newer repo eg. jammy (22.04LTS)?

[lukaszstolarczuk]

To use HWLOC property query API, we need the the lib version >= 2.3.0. Unfortunately on the Ubuntu 20.04 runners the latest available package is version 2.1.0 (https://packages.ubuntu.com/search?suite=all&searchon=names&keywords=hwloc). @lukaszstolarczuk do you think we could use a script to build and install HWLOC from source instead of relying on the package manager? Or alternatively install the package from newer repo eg. jammy (22.04LTS)?

heh, we are using Ubuntu 22.04. I think the error you have is caused because of typo - you specified to look for >=2.30.0 😉
We have 2.7.0 installed: https://github.com/oneapi-src/unified-memory-framework/pull/408/checks#step:6:189

[kswiecicki]

To use HWLOC property query API, we need the the lib version >= 2.3.0. Unfortunately on the Ubuntu 20.04 runners the latest available package is version 2.1.0 (https://packages.ubuntu.com/search?suite=all&searchon=names&keywords=hwloc). @lukaszstolarczuk do you think we could use a script to build and install HWLOC from source instead of relying on the package manager? Or alternatively install the package from newer repo eg. jammy (22.04LTS)?

heh, we are using Ubuntu 22.04. I think the error you have is caused because of typo - you specified to look for >=2.30.0 😉 We have 2.7.0 installed: https://github.com/oneapi-src/unified-memory-framework/pull/408/checks#step:6:189

My bad, I've left the version 2.30.0 as I was testing the FindLIBHWLOC.cmake module.
We are also using Ubuntu 20.04 runners that use the HWLOC version 2.1.0 installed via apt package manager.

[lukaszstolarczuk]

To use HWLOC property query API, we need the the lib version >= 2.3.0. Unfortunately on the Ubuntu 20.04 runners the latest available package is version 2.1.0 (https://packages.ubuntu.com/search?suite=all&searchon=names&keywords=hwloc). @lukaszstolarczuk do you think we could use a script to build and install HWLOC from source instead of relying on the package manager? Or alternatively install the package from newer repo eg. jammy (22.04LTS)?

heh, we are using Ubuntu 22.04. I think the error you have is caused because of typo - you specified to look for >=2.30.0 😉 We have 2.7.0 installed: https://github.com/oneapi-src/unified-memory-framework/pull/408/checks#step:6:189

My bad, I've left the version 2.30.0 as I was testing the FindLIBHWLOC.cmake module. We are also using Ubuntu 20.04 runners that use the HWLOC version 2.1.0 installed via apt package manager.

Ok, right. For the few jobs using Ubuntu 20.04 we can do what you proposed - install from source. If this will end up being more than a few lines of code, please make this a python script, e.g. install-hwloc.py

[kswiecicki]

The original implementation of the highest bandwidth memspace was supposed to contain NUMA nodes (represented as memory targets in UMF) sorted by either the average or maximum (it was not yet decided) bandwidth value. This value was obtained by considering each NUMA node as the initiator for the bandwidth query and selecting all NUMA nodes as targets consecutively.
Since control over allocation placement is restricted to the nodemask provided to the membind function in the OS provider, we were unable to prioritize NUMA nodes with better bandwidth values. In that case, the closest nodes contained in the nodemask would be prioritized. For now, we've settled on the highest bandwidth memspace implementation that contains an aggregated list of NUMA nodes identified as the best targets after selecting each NUMA node as the initiator.

This comment serves as documentation for the development of the memspace API.
@igchor would you like to add something?

[kswiecicki]

@kswiecicki could you add a test on qemu that uses the highest bandwidth memspace and verifies that in case of equal bandwidths between nodes, local node is preferred?

The test should spawn multiple threads and pin every thread to a different NUMA node and the allocate memory from each thread. The allocated memory should come from the local node.

Done. I've added the test case, but there's no QEMU configuration that satisfies the equal bandwidth requirement. I think @KFilipek is working on adding new topologies for testing.

[lukaszstolarczuk]

👍

[lplewa]

Do those functions works on windows - you did not updated .def file.

[kswiecicki]

Do those functions works on windows - you did not updated .def file.

Memspace functionality is currently built only on Linux platforms.