Move conformance tests over to lit (#17998)

Conformance tests are now ran by LLVM's lit framework rather than ctest.
A new `check-unified-runtime-conformance` target is available, which
builds and runs all conformance tests. The `test` target will still run
them.

Some additional housekeeping was done:
* `add_conformance_test_with_x_environment` was renamed to
  `add_conformance_x_test`.
* Conformance tests now explicitly respect `ONEAPI_DEVICE_SELECTOR`.
* If unspecified, `ONEAPI_DEVICE_SELECTOR` will now be populated with
  the list of built targets. Notably, this will exclude adapters which
  exist in the build directory but were not specified using a
  `BUILD_ADAPTER` cmake option (e.g, from a previous build before a
  reconfigure).
* References to `UR_CTS_ADAPTER_PLATFORM` and `UR_CTS_BACKEND` are
  removed (they never did anything anyway).
* Github build jobs have been updated.
* The generated gtest binaries have been moved from `build/bin` to
  `build/test/conformance`
* `requirements_testing.txt` now imports psutil for using timeouts.
* Output format has been changed to show a progress bar and print
  the test output in real time.
* Testing now respects the "device" filters from
  `ONEAPI_DEVICE_SELECTOR` (previously, the filter was only used to
  select the platform. `opencl:cpu` now works as expected.
* Some CUDA tests are marked as failing since they are failing
  after this change.

As a resut of these changes, a build with both the v1 and v2
versions of the level zero adapter will only run the v2 adapter.
Building with only one version of the level zero adapter (as the
Github CI currently does) works as expected and only runs that
version.

Author: RossBrunton

CMakeLists.txt

@@ -65,8 +65,7 @@ set(UR_SYCL_LIBRARY_DIR "" CACHE PATH
 set(UR_CONFORMANCE_TARGET_TRIPLES "" CACHE STRING
     "List of sycl targets to build CTS device binaries for")
 set(UR_CONFORMANCE_AMD_ARCH "" CACHE STRING "AMD device target ID to build CTS binaries for")
-option(UR_CONFORMANCE_ENABLE_MATCH_FILES "Enable CTS match files" ON)
-option(UR_CONFORMANCE_TEST_LOADER "Also test the loader in the conformance tests" OFF)
+set(UR_CONFORMANCE_SELECTOR "" CACHE STRING "If nonempty, the device selector for conformance tests")
 option(UR_USE_DEBUG_POSTFIX "Enable debug postfix 'd' for libraries" OFF)
 set(UR_ADAPTER_LEVEL_ZERO_SOURCE_DIR "" CACHE PATH
     "Path to external 'level_zero' adapter source dir")

README.md

@@ -127,8 +127,7 @@ List of options provided by CMake:
 | UR_ENABLE_SANITIZER | Enable device sanitizer layer | ON/OFF | ON |
 | UR_CONFORMANCE_TARGET_TRIPLES | SYCL triples to build CTS device binaries for | Comma-separated list | spir64 |
 | UR_CONFORMANCE_AMD_ARCH | AMD device target ID to build CTS binaries for | string | `""` |
-| UR_CONFORMANCE_ENABLE_MATCH_FILES | Enable CTS match files | ON/OFF | ON |
-| UR_CONFORMANCE_TEST_LOADER | Additionally build and run "loader" tests for the CTS | ON/OFF | OFF |
+| UR_CONFORMANCE_SELECTOR | `ONEAPI_DEVICE_SELECTOR` for conformance testing | string | All enabled adapters |
 | UR_BUILD_ADAPTER_L0     | Build the Level-Zero adapter            | ON/OFF     | OFF     |
 | UR_BUILD_ADAPTER_OPENCL | Build the OpenCL adapter                | ON/OFF     | OFF     |
 | UR_BUILD_ADAPTER_CUDA   | Build the CUDA adapter                  | ON/OFF     | OFF     |

scripts/core/CONTRIB.rst

@@ -272,14 +272,21 @@ Conformance tests *must* not make assumptions about the adapter under test.
 Tests fixtures or cases *must* query for support of optional features and skip
 testing if unsupported by the adapter.
 
-All tests in the Unified Runtime project are configured to use CTest to run. 
-All conformance tests have the ``conformance`` label attached to them which 
-allows them to be run independently. To run all the conformance tests, execute 
-the following command from the build directory.
+Conformance tests are ran as part of the main ``test`` target, but can also be
+executed using the `check-unified-runtime-conformance` target as follows:
 
 .. code-block:: console
      
-    ctest -L "conformance"
+    cmake --build build --target check-unified-runtime-conformance
+
+A specific device/adapter can be specified by using ``ONEAPI_DEVICE_SELECTOR``
+in the same way as in the `DPC++ compiler
+<https://github.com/intel/llvm/blob/sycl/sycl/doc/EnvironmentVariables.md#oneapi_device_selector>`_.
+
+A number of tests are skipped on certain adapters due to being known failures.
+To force all tests to run, including known failures,
+``UR_CTS_ALSO_RUN_KNOWN_FAILURES`` may be set to ``1`` as an environment
+variable.
 
 Experimental Features
 =====================

scripts/core/INTRO.rst

@@ -417,26 +417,12 @@ CTS Environment Variables
 
 The following environment variables are used by the CTS runner and can be used to specify the platform that the test
 framework should run on. This can be used during development and testing to run CTS tests in case multiple platforms
-are available. If both filters are specified, then they both must match a platform for it to be selected. If there are
-no valid platforms, then the tests will fail. Command line arguments take priority over these variables.
+are available. Conformance tests will be skipped if there are no valid platforms.
 
-.. envvar:: UR_CTS_ADAPTER_PLATFORM
+.. envvar:: ONEAPI_DEVICE_SELECTOR
 
-    A specifier list in the form of `[(backend):](platform name)[;[(backend)]:(platform name)]...`. If a backend
-    specific specifier is present in the list and the test is running for that backend, the device with the given name
-    is chosen. Otherwise, it must match the name of the specifier from the list with no backend. Backend names are case-
-    insensitive, however platform names must match exactly.
-
-    For example, if the test device has multiple platforms and you want to run tests on the "ABC Corp" backend when
-    testing OpenCL and "XYZ Org" when testing level zero, you'd use `OPENCL:ABC Corp;LEVEL_ZERO:XYZ Org`. This form is
-    useful when running the `build` target with a build with multiple backends.
-
-    For testing only one platform, the backend can be omitted. For example, just `ABC Corp` is sufficient if the tests
-    are only going to be testing OpenCL.
-
-.. envvar:: UR_CTS_BACKEND
-
-    A (case insensitive) backend to force the test to use. For example, `opencl`, `level_zero`, `hip` and so on.
+   If ``ONEAPI_DEVICE_SELECTOR`` is set in the environment, only devices
+   matched by the selector will be tested.
 
 .. envvar:: UR_CTS_ALSO_RUN_KNOWN_FAILURES
 

scripts/ctest_parser.py

@@ -31,8 +31,9 @@ configure_file(lit.site.cfg.py.in lit.site.cfg.py)
 add_custom_target(deps_check-unified-runtime)
 if(UR_STANDALONE_BUILD)
   add_custom_target(check-unified-runtime
-    COMMAND "${URLIT_LIT_BINARY}" "${CMAKE_CURRENT_BINARY_DIR}" -v
+    COMMAND "${URLIT_LIT_BINARY}" "${CMAKE_CURRENT_BINARY_DIR}" -sv
     DEPENDS deps_check-unified-runtime
+    USES_TERMINAL
   )
 else()
   # Use the LLVM method to add the test suite - this also registers it under `check-all`
@@ -60,7 +61,8 @@ function(add_ur_lit_testsuite suite)
 
   if(UR_STANDALONE_BUILD)
     add_custom_target(${TARGET}
-      COMMAND "${URLIT_LIT_BINARY}" "${CMAKE_CURRENT_BINARY_DIR}" -v
+      COMMAND "${URLIT_LIT_BINARY}" "${CMAKE_CURRENT_BINARY_DIR}" -sv
+      USES_TERMINAL
     )
   else()
     add_lit_testsuite(${TARGET} "Running Unified Runtime ${suite} tests"