Merge pull request #546 from kbenzie/benie/experimental-features

[ur] Add experimental feature contribution docs

Author: callumfare

scripts/core/CONTRIB.rst

@@ -1,71 +1,205 @@
-
 <%
     OneApi=tags['$OneApi']
     x=tags['$x']
     X=x.upper()
 %>
-===================
+====================
  Contribution Guide
-===================
+====================
 
-This contribution guide explains how to contribute to the Unified Runtime 
-project and what processes you must follow in order to have your changes 
+This contribution guide explains how to contribute to the Unified Runtime
+project and what processes you *must* follow in order to have your changes
 accepted into the project.
 
+.. important::
+
+    Before making a contribution you *should* determine if the change should be
+    made directly to the core specification or introduced as an experimental
+    feature. The criteria we use to make this distinction are as follows:
+
+    *   The feature exists to enable an experimental feature in a parallel
+        language runtime being built on top of Unified Runtime.
+
+    *   The design phase of the feature is expected to span multiple oneAPI
+        releases.
+
+    *   A proof of concept implementation exists for a single adapter but
+        multiple adapters are intended to be supported. It is important to
+        consider as early as possible whether the feature is appropriate for
+        other adapters to evaluate its portability.
 
-Updating the Specification
-==========================
+    If the feature in question matches any of these criteria, please refer to
+    the `Experimental Features`_ section, otherwise refer to the `Core
+    Features`_ section. If you are unsure how to proceed please `create an
+    issue <https://github.com/oneapi-src/unified-runtime/issues/new>`_ asking
+    for clarification.
 
-The specification and many other components of Unified Runtime are generated 
-from a set of ``*.yml`` files located in the ``scripts/core/`` directory of the 
-project. To make a change to the specification you should modify only ``*.yml`` 
-files in this directory. To see the changes reflected in the specification you 
-`must` build the ``generate`` target. When making changes to the specification 
-you should commit all changes generated by the script as part of your change. 
-Refer to ``scripts/YaML.md`` on the yaml syntax as well as detailed descriptions 
-of required fields for each type of entity.
+    If you are unsure whether a feature can be supported by certain adapters
+    please seek the advice of an appropriate stakeholder or ask the Unified
+    Runtime team via the `GitHub issue tracker
+    <https://github.com/oneapi-src/unified-runtime/issues/new>`_.
 
-Naming Conventions
-~~~~~~~~~~~~~~~~~~
+Generating Source
+=================
 
-Additional to what is described in the Yaml syntax document. You should follow
-the following naming convention for argument names.
+The specification and many other components in the Unified Runtime repository
+are generated from a set of YAML_ files which are used as inputs to a Mako_
+based templating system. The YAML file syntax is defined in `YAML syntax`_. To
+generate the outputs of the Mako templates a build directory must be
+configured, instructions are available in the `README
+<https://github.com/oneapi-src/unified-runtime/blob/main/README.md>`_ file.
+Upon successfully configuring a build directory, generate the outputs with the
+following command (or suitable build system equivalent):
 
-    * Entry-Points use ``camelCase`` conventions i.e ``${x}DeviceGet``.
+.. code-block:: console
 
-    * Argument names are also `camelCase` and are prefixed with a type 
-      indicator:
+    $ cmake --build build --target generate
 
-        * Pointer arguments are prefixed with ``p`` or ``pp`` for double pointer 
-          i.e. ``pEvent``.
+.. _YAML: https://yaml.org/
+.. _Mako: https://www.makotemplates.org/
+.. _YAML syntax:
+   https://github.com/oneapi-src/unified-runtime/blob/main/scripts/YaML.md
 
-        * Handle arguments are prefixed with ``h`` i.e. ``hQueue``.
+Writing YAML
+============
 
+Please read the :ref:`core/INTRO:Naming Convention` section prior to making a
+contribution and refer to the `YAML syntax`_ for specifics of how to define the
+required constructs.
 
+When writing ``*.yml`` files and ``ur`` or ``UR`` should exist in the output
+use ``$${'x'}`` or ``$${'X'}`` respectively. These will be replaced while
+`Generating Source`_.
 
+Additionally, the following conventions *must* be followed for function
+arguments:
+
+*   Argument names are ``camelCase``.
+*   Arguments with pointer types are prefixed with ``p`` for each pointer in
+    the type i.e. ``char *pMessage``, ``char **ppMessage``, etc.
+*   Handle arguments are prefixed with ``h`` i.e. ``hQueue``.
+*   Pointer to handle arguments, such as out parameters, are prefixed with
+    ``ph`` i.e. ``phQueue``.
 
 Forks and Pull Requests
 =======================
 
-To submit a pull request to Unified Runtime, you must first create your own 
-personal fork of the project and submit your changes to a branch. By convention 
-we name our branches ``<your_name>/<short_description>``, where the description 
-indicates the intent of your change. You can then raise a pull request targeting 
-``oneapi-src/unified-runtime:main``.
+To submit a pull request to Unified Runtime, you must first create your own
+personal fork of the project and submit your changes to a branch. By convention
+we name our branches ``<your_name>/<short_description>``, where the description
+indicates the intent of your change. You can then raise a pull request
+targeting ``oneapi-src/unified-runtime:main``.
+
+When making changes to the specification you *must* commit all changes to files
+in the repository as a result of `Generating Source`_.
+
+Before your pull request is merged it *must* pass all jobs in the GitHub
+Actions workflow and *must* be reviewed by no less than two code owners.
+
+.. hint::
+
+    When rebasing a branch on top of ``main`` results in merged conflicts it is
+    recommended to resolve conflicts in the ``*.yml`` files then `Generating
+    Source`_. This will automatically resolve conflicts in the generated source
+    files, leaving only conflicts in non-generated source files to be resolved,
+    if any.
+
+Core Features
+=============
+
+A core feature *must* have a stable API/ABI and *should* strive to be supported
+across all adapters. However, core features *may* be optional and thus only
+supported in one or more adapters. A core feature *should* also strive to
+enable similar functionality in parallel language runtimes (such as SYCL,
+OpenMP, ...) where possible although this is a secondary concern.
+
+.. hint::
+
+    Optional features should be avoided as much as possible to maximize
+    portability across adapters and reduce the overhead required to make use of
+    features in parallel language runtimes.
+
+Core features are defined in the ``*.yml`` files in the `scripts/core
+<https://github.com/oneapi-src/unified-runtime/tree/main/scripts/core>`_
+directory. Most of the files are named after the API object who's interface is
+defined within them, with the following exceptions:
+
+*   `scripts/core/common.yml`_ defines symbols which are used by multiple
+    interfaces through the specification, e.g. macros, object handles, result
+    enumerations, and structure type enumerations.
+*   `scripts/core/enqueue.yml`_ defines commands which can be enqueued on a
+    queue object.
+*   `scripts/core/runtime.yml`_ defines global symbols pertaining to
+    initialization and tear down of the entire runtime.
+*   `scripts/core/registry.yml`_ contains an enumeration of all entry-points,
+    past and present, for use in the XPTI tracing framework. It is
+    automatically updated so shouldn't require manual editing.
+*   ``scripts/core/exp-<feature>.yml`` see `Experimental Features`_.
+
+.. _scripts/core/common.yml:
+   https://github.com/oneapi-src/unified-runtime/blob/main/scripts/core/common.yml
+.. _scripts/core/enqueue.yml:
+   https://github.com/oneapi-src/unified-runtime/blob/main/scripts/core/enqueue.yml
+.. _scripts/core/runtime.yml:
+   https://github.com/oneapi-src/unified-runtime/blob/main/scripts/core/runtime.yml
+.. _scripts/core/registry.yml:
+   https://github.com/oneapi-src/unified-runtime/blob/main/scripts/core/registry.yml
+
+Core Optional Features
+----------------------
+
+Optional core features *must* be supported in at least one adapter. Support for
+an optional core feature *must* be programmatically exposed to the user via
+boolean query call to ${x}DeviceGetInfo and a new enumerator of the form
+``UR_DEVICE_INFO_<FEATURE_NAME>_SUPPORT`` in ${x}_device_info_t.
+
+Conformance Testing
+-------------------
 
-Before your pull request is merged it `must` be reviewed by no less than two 
-code owners of the project `and` be passing all required CI jobs.
+For contributions to the core specification conformance tests *should* be
+included as part of your change. The conformance tests can be found
+under ``test/conformance/<component>``, where component refers to the API
+object an entry-point belongs to i.e. platform, enqueue, device.
 
+The conformance tests *should* ideally include end-to-end testing of all the
+changes to the specification if possible. At minimum, they *must* cover at
+least one test for each of the possible error codes returned, excluding any
+disaster cases like ${X}_RESULT_ERROR_OUT_OF_HOST_MEMORY or similar.
 
-Conformance Testing
-===================
-
-For contributions to the specification you should include appropriate 
-conformance tests as part of your change. The conformance tests can be found 
-under ``test/conformance/<component>``, where component refers to a set of 
-entry-points i.e. platform, enqueue, device. The conformance tests, at a 
-minimum, should cover at least one test for each of the possible error codes 
-returned, excluding any disaster cases like 
-``${X}_RESULT_ERROR_OUT_OF_HOST_MEMORY`` or similar. Conformance tests should not
-assume anything about the underlying adapter on which they are run against and
-should pass on any conformant adapter.
+Conformance tests *must* not make assumptions about the adapter under test.
+Tests fixtures or cases *must* query for support of optional features and skip
+testing if unsupported by the adapter.
+
+Experimental Features
+=====================
+
+.. warning::
+
+    Experimental features:
+
+    *   May be replaced, updated, or removed at any time.
+    *   Do not require maintaining API/ABI stability of their own additions
+        over time.
+    *   Do not require conformance testing of their own additions.
+
+Experimental features *must* be defined in two new files, where
+``<FEATURE>``/``<feature>`` are replaced with an appropriate name:
+
+*   ``scripts/core/EXP-<FEATURE>.rst`` document specifying the experimental
+    feature in natural language.
+*   ``scripts/core/exp-<feature>.yml`` defines the interface as an input to
+    `Generating Source`_.
+
+Naming Convention
+-----------------
+
+In addition to the requirements referred to in the `Writing YAML`_ section, and
+to easily differentiate experimental feature symbols, the following conventions
+*must* be adhered to when defining experimental features:
+
+## --validate=off
+*   All functions must use camel case ``${x}ObjectActionExp`` convention.
+*   All macros must use all caps ``${X}_NAME_EXP`` convention.
+*   All structures, enumerations, and other types must follow
+    ``${x}_exp_name_t`` name case convention.
+## --validate=on

scripts/core/INTRO.rst

@@ -47,22 +47,22 @@ Naming Convention
 The following naming conventions must be followed:
 
 ## --validate=off
-  - All functions must be prefixed with `${x}`
-  - All functions must use camel case `${x}ObjectAction` convention
-  - All macros must use all caps `${X}_NAME` convention
-  - All structures, enumerations and other types must follow `${x}_name_t` snake case convention
+  - All functions must be prefixed with ``${x}``
+  - All functions must use camel case ``${x}ObjectAction`` convention
+  - All macros must use all caps ``${X}_NAME`` convention
+  - All structures, enumerations and other types must follow ``${x}_name_t`` snake case convention
   - All structure members and function parameters must use camel case convention
-  - All enumerator values must use all caps `${X}_ENUM_ETOR_NAME` convention
-  - All handle types must end with `handle_t`
-  - All descriptor structures must end with `desc_t`
-  - All property structures must end with `properties_t`
-  - All flag enumerations must end with `flags_t`
+  - All enumerator values must use all caps ``${X}_ENUM_ETOR_NAME`` convention
+  - All handle types must end with ``handle_t``
+  - All descriptor structures must end with ``desc_t``
+  - All property structures must end with ``properties_t``
+  - All flag enumerations must end with ``flags_t``
 ## --validate=on
 
 The following coding conventions must be followed:
 
-  - All descriptor structures must be derived from `${x}_base_desc_t`
-  - All property structures must be derived from `${x}_base_properties_t`
+  - All descriptor structures must be derived from ${x}_base_desc_t
+  - All property structures must be derived from ${x}_base_properties_t
   - All function input parameters must precede output parameters
   - All functions must return ${x}_result_t
 

scripts/parse_specs.py

@@ -816,6 +816,7 @@ def parse(section, version, tags, meta, ref):
         line_nums = _get_line_nums(f)
 
         header = None
+        name = None
         objects = []
 
         for i, d in enumerate(docs):
@@ -833,14 +834,23 @@ def parse(section, version, tags, meta, ref):
                 header['ordinal'] = int(int(header.get('ordinal',"1000")) * float(header.get('version',"1.0")))
                 header['ordinal'] *= 1000 if re.match(r"extension", header.get('desc',"").lower()) else 1
                 header['ordinal'] *= 1000 if re.match(r"experimental", header.get('desc',"").lower()) else 1
+                basename = os.path.splitext(os.path.basename(f))[0]
+                if 'name' in header:
+                    name = header['name']
+                elif basename.startswith('exp-'):
+                    name = f'{basename[len("exp-"):]} (experimental)'
+                else:
+                    name = basename
+                for c in '_-':
+                    name = name.replace(c, ' ')
             elif header:
                 # for d in _make_versions(d, float(version)):
                 objects.append(d)
                 meta = _generate_meta(d, header['ordinal'], meta)
 
         if header:
             specs.append({
-                'name'      : os.path.splitext(os.path.basename(f))[0],
+                'name'      : name,
                 'header'    : header,
                 'objects'   : objects,
             })

scripts/templates/conf.py.mako

@@ -41,6 +41,7 @@ needs_sphinx = '1.8'
 # ones.
 extensions = [
     'sphinx.ext.autodoc',
+    'sphinx.ext.autosectionlabel',
     'sphinx.ext.doctest',
     'sphinx.ext.todo',
     'sphinx.ext.coverage',
@@ -52,6 +53,10 @@ extensions = [
     'breathe'
 ]
 
+# Configure sphinx.ext.autosectionlabel to require the document prefix when
+# cross-referencing document sections.
+autosectionlabel_prefix_document = True
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']