Multi-Version Drivers (#6163)

    MVD CP-52334 multi-version driver API/CLI

Provide an API/CLI for multi-version drivers. Two new classes represent
    them: Host_driver and Driver_variant. See

      * doc/content/toolstack/features/MVD/index.md

    for technical background and feature overview.

The current implementation mocks the missing drivertool that manipulates
the file system and observes driver state. We expect to integrate this
at a later point. Consequently, the current implementation is incomplete
    and exists to faciliate development of API and XE clients.

    On Xapi start and restart driver information is purged from the
database and re-built from scrach. This could be improved in the future
    to be incremental.

Author: lindig

doc/content/toolstack/features/MVD/index.md

@@ -0,0 +1,347 @@
++++
+title = "Multi-version drivers"
++++
+
+Linux loads device drivers on boot and every device driver exists in one
+version. XAPI extends this scheme such that device drivers may exist in
+multiple variants plus a mechanism to select the variant being loaded on
+boot. Such a driver is called a multi-version driver and we expect only
+a small subset of drivers, built and distributed by XenServer, to have
+this property. The following covers the background, API, and CLI for
+multi-version drivers in XAPI.
+
+## Variant vs. Version
+
+A driver comes in several variants, each of which has a version. A
+variant may be updated to a later version while retaining its identity.
+This makes variants and versions somewhat synonymous and is admittedly
+confusing.
+
+## Device Drivers in Linux and XAPI
+
+Drivers that are not compiled into the kernel are loaded dynamically
+from the file system. They are loaded from the hierarchy
+
+* `/lib/modules/<kernel-version>/`
+
+and we are particularly interested in the hierarchy
+
+* `/lib/modules/<kernel-version>/updates/`
+
+where vendor-supplied ("driver disk") drivers are located and where we
+want to support multiple versions. A driver has typically file extension
+`.ko` (kernel object).
+
+A presence in the file system does not mean that a driver is loaded as
+this happens only on demand. The actually loaded drivers
+(or modules, in Linux parlance) can be observed from
+
+* `/proc/modules`
+
+```
+netlink_diag 16384 0 - Live 0x0000000000000000
+udp_diag 16384 0 - Live 0x0000000000000000
+tcp_diag 16384 0 - Live 0x0000000000000000
+```
+
+which includes dependencies between modules (the `-` means no dependencies).
+
+## Driver Properties
+
+* A driver name is unique and a driver can be loaded only once. The fact
+  that kernel object files are located in a file system hierarchy means
+  that a driver may exist multiple times and in different version in the
+  file system. From the kernel's perspective a driver has a unique name
+  and is loaded at most once. We thus can talk about a driver using its
+  name and acknowledge it may exist in different versions in the file
+  system.
+
+* A driver that is loaded by the kernel we call *active*.
+
+* A driver file (`name.ko`) that is in a hierarchy searched by the
+  kernel is called *selected*. If the kernel needs the driver of that
+  name, it would load this object file.
+
+For a driver (`name.ko`) selection and activation are independent
+properties:
+
+* *inactive*, *deselected*: not loaded now and won't be loaded on next
+  boot.
+* *active*, *deselected*: currently loaded but won't be loaded on next
+  boot.
+* *inactive*, *selected*: not loaded now but will be loaded on demand.
+* *active*, *selected*: currently loaded and will be loaded on demand
+  after a reboot.
+
+For a driver to be selected it needs to be in the hierarchy searched by
+the kernel. By removing a driver from the hierarchy it can be
+de-selected. This is possible even for drivers that are already loaded.
+Hence, activation and selection are independent.
+
+## Multi-Version Drivers
+
+To support multi-version drivers, XenServer introduces a new
+hierarchy in Dom0. This is mostly technical background because a
+lower-level tool deals with this and not XAPI directly.
+
+* `/lib/modules/<kernel-version>/updates/` is searched by the kernel for
+  drivers.
+* The hierarchy is expected to contain symbolic links to the file
+  actually containing the driver:
+  `/lib/modules/<kernel-version>/xenserver/<driver>/<version>/<name>.ko`
+
+The `xenserver` hierarchy provides drivers in several versions. To
+select a particular version, we expect a symbolic link from
+`updates/<name>.ko` to `<driver>/<version>/<name>.ko`. At the next boot,
+the kernel will search the `updates/` entries and load the linked
+driver, which will become active.
+
+Example filesystem hierarchy:
+```
+/lib/
+└── modules
+    └── 4.19.0+1 ->
+        ├── updates
+        │   ├── aacraid.ko
+        │   ├── bnx2fc.ko -> ../xenserver/bnx2fc/2.12.13/bnx2fc.ko
+        │   ├── bnx2i.ko
+        │   ├── cxgb4i.ko
+        │   ├── cxgb4.ko
+        │   ├── dell_laptop.ko -> ../xenserver/dell_laptop/1.2.3/dell_laptop.ko
+        │   ├── e1000e.ko
+        │   ├── i40e.ko
+        │   ├── ice.ko -> ../xenserver/intel-ice/1.11.17.1/ice.ko
+        │   ├── igb.ko
+        │   ├── smartpqi.ko
+        │   └── tcm_qla2xxx.ko
+        └── xenserver
+            ├── bnx2fc
+            │   ├── 2.12.13
+            │   │   └── bnx2fc.ko
+            │   └── 2.12.20-dell
+            │       └── bnx2fc.ko
+            ├── dell_laptop
+            │   └── 1.2.3
+            │       └── dell_laptop.ko
+            └── intel-ice
+                ├── 1.11.17.1
+                │   └── ice.ko
+                └── 1.6.4
+                    └── ice.ko
+
+```
+
+Selection of a driver is synonymous with creating a symbolic link to the
+desired version.
+
+## Versions
+
+The version of a driver is encoded in the path to its object file but
+not in the name itself: for `xenserver/intel-ice/1.11.17.1/ice.ko` the
+driver name is `ice` and only its location hints at the version.
+
+The kernel does not reveal the location from where it loaded an active
+driver. Hence the name is not sufficient to observe the currently active
+version. For this, we use [ELF notes].
+
+The driver file (`name.ko`) is in ELF linker format and may contain
+custom [ELF notes]. These are binary annotations that can be compiled
+into the file. The kernel reveals these details for loaded drivers
+(i.e., modules) in:
+
+* `/sys/module/<name>/notes/`
+
+The directory contains files like
+
+* `/sys/module/xfs/notes/.note.gnu.build-id`
+
+with a specific name (`.note.xenserver`) for our purpose. Such a file contains
+in binary encoding a sequence of records, each containing:
+
+* A null-terminated name (string)
+* A type (integer)
+* A desc (see below)
+
+The format of the description is vendor specific and is used for
+a null-terminated string holding the version. The name is fixed to
+"XenServer". The exact format is described in [ELF notes].
+
+A note with the name "XenServer" and a particular type then has the version
+as a null-terminated string the `desc` field. Additional "XenServer" notes
+of a different type may be present.
+
+[ELF notes]: https://www.netbsd.org/docs/kernel/elf-notes.html
+
+## API
+
+XAPI has capabilities to inspect and select multi-version drivers.
+
+The API uses the terminology introduced above:
+
+* A driver is specific to a host.
+* A driver has a unique name; however, for API purposes a driver is
+  identified by a UUID (on the CLI) and reference (programmatically).
+* A driver has multiple variants; each variant has a version.
+  Programatically, variants are represented as objects (referenced by
+  UUID and a reference) but this is mostly hidden in the CLI for
+  convenience.
+* A driver variant is active if it is currently used by the kernel
+  (loaded).
+* A driver variant is selected if it will be considered by the kernel
+  (on next boot or when loading on demand).
+* Only one variant can be active, and only one variants can be selected.
+
+Inspection and selection of drivers is facilitated by a tool
+("drivertool") that is called by xapi. Hence, XAPI does not by itself
+manipulate the file system that implements driver selection.
+
+An example interaction with the API through xe:
+
+```
+[root@lcy2-dt110 log]# xe hostdriver-list uuid=c0fe459d-5f8a-3fb1-3fe5-3c602fafecc0 params=all
+uuid ( RO)                   : c0fe459d-5f8a-3fb1-3fe5-3c602fafecc0
+                   name ( RO): cisco-fnic
+                   type ( RO): network
+            description ( RO): cisco-fnic
+                   info ( RO): cisco-fnic
+              host-uuid ( RO): 6de288e7-0f82-4563-b071-bcdc083b0ffd
+         active-variant ( RO): <none>
+       selected-variant ( RO): <none>
+               variants ( RO): generic/1.2
+    variants-dev-status ( RO): generic=beta
+          variants-uuid ( RO): generic=abf5997b-f2ad-c0ef-b27f-3f8a37bf58a6
+    variants-hw-present ( RO): 
+```
+
+Selection of a variant by name (which is unique per driver); this
+variant would become active after reboot.
+
+```
+[root@lcy2-dt110 log]# xe hostdriver-select variant-name=generic uuid=c0fe459d-5f8a-3fb1-3fe5-3c602fafecc0
+[root@lcy2-dt110 log]# xe hostdriver-list uuid=c0fe459d-5f8a-3fb1-3fe5-3c602fafecc0 params=all
+uuid ( RO)                   : c0fe459d-5f8a-3fb1-3fe5-3c602fafecc0
+                   name ( RO): cisco-fnic
+                   type ( RO): network
+            description ( RO): cisco-fnic
+                   info ( RO): cisco-fnic
+              host-uuid ( RO): 6de288e7-0f82-4563-b071-bcdc083b0ffd
+         active-variant ( RO): <none>
+       selected-variant ( RO): generic
+               variants ( RO): generic/1.2
+    variants-dev-status ( RO): generic=beta
+          variants-uuid ( RO): generic=abf5997b-f2ad-c0ef-b27f-3f8a37bf58a6
+    variants-hw-present ( RO): 
+```
+
+The variant can be inspected, too, using it's UUID.
+
+```
+[root@lcy2-dt110 log]# xe hostdriver-variant-list uuid=abf5997b-f2ad-c0ef-b27f-3f8a37bf58a6
+uuid ( RO)           : abf5997b-f2ad-c0ef-b27f-3f8a37bf58a6
+           name ( RO): generic
+        version ( RO): 1.2
+         status ( RO): beta
+         active ( RO): false
+       selected ( RO): true
+    driver-uuid ( RO): c0fe459d-5f8a-3fb1-3fe5-3c602fafecc0
+    driver-name ( RO): cisco-fnic
+      host-uuid ( RO): 6de288e7-0f82-4563-b071-bcdc083b0ffd
+     hw-present ( RO): false
+```
+
+## Class Host_driver
+
+Class `Host_driver` represents an instance of a multi-version driver on
+a host. It references `Driver_variant` objects for the details of the
+available and active variants. A variant has a version.
+
+### Fields
+
+All fields are read-only and can't be set directly. Be aware that names
+in the CLI and the API may differ.
+
+* `host`: reference to the host where the driver is installed.
+* `name`: string; name of the driver without ".ko" extension.
+* `variants`: string set; set of variants available on the host for this
+  driver.  The name of each variant of a driver is unique and used in
+  the CLI for selecting it.
+* `selected_varinat`: variant, possibly empty. Variant that is selected,
+  i.e. the variant of the driver that will be considered by the kernel
+  when loading the driver the next time. May be null when none is
+  selected.
+* `active_variant`: variant, possibly empty. Variant that is currently
+  loaded by the kernel.
+* `type`, `info`, `description`: strings providing background
+  information.
+
+The CLI uses `hostdriver` and a dash instead of an underscore. The CLI
+also offers convenience fields. Whenever selected and
+active variant are not the same, a reboot is required to activate the
+selected driver/variant combination.
+
+(We are not using `host-driver` in the CLI to avoid the impression that
+this is part of a host object.)
+
+### Methods
+
+* All method invocations require `Pool_Operator` rights. "The Pool
+  Operator role manages host- and pool-wide resources, including setting
+  up storage, creating resource pools and managing patches, high
+  availability (HA) and workload balancing (WLB)"
+
+* `select (self, variant)`; select `variant` of driver `self`. Selecting
+  the variant (a reference) of an existing driver.
+
+* `deselect(self)`: this driver can't be loaded next time the kernel is
+  looking for a driver. This is a potentially dangerous operation, so it's
+  protected in the CLI with a `--force` flag.
+
+* `rescan (host)`: scan the host and update its driver information.
+   Called on toolstack restart and may be invoked from the CLI for
+   development.
+
+## Class `Driver_variant`
+
+An object of this class represents a variant of a driver on a host,
+i.e., it is specific to both.
+
+* `name`: unique name
+* `driver`: what host driver this belongs to
+* `version`: string; a driver variant has a version
+* `status`: string: development status, like "beta"
+* `hardware_present`: boolean, true if the host has the hardware
+   installed supported by this driver
+
+The only method available is `select(self)` to select a variant. It has
+the same effect as the `select` method on the `Host_driver` class.
+
+The CLI comes with corresponding `xe hostdriver-variant-*` commands to
+list and select a variant. 
+
+```
+[root@lcy2-dt110 log]# xe hostdriver-variant-list uuid=abf5997b-f2ad-c0ef-b27f-3f8a37bf58a6
+uuid ( RO)           : abf5997b-f2ad-c0ef-b27f-3f8a37bf58a6
+           name ( RO): generic
+        version ( RO): 1.2
+         status ( RO): beta
+         active ( RO): false
+       selected ( RO): true
+    driver-uuid ( RO): c0fe459d-5f8a-3fb1-3fe5-3c602fafecc0
+    driver-name ( RO): cisco-fnic
+      host-uuid ( RO): 6de288e7-0f82-4563-b071-bcdc083b0ffd
+     hw-present ( RO): false
+```
+
+### Database
+
+Each `Host_driver` and `Driver_variant` object is represented in the
+database and data is persisted over reboots. This means this data will
+be part of data collected in a `xen-bugtool` invocation.
+
+### Scan and Rescan
+
+On XAPI start-up, XAPI updates the `Host_driver` objects belonging to the
+host to reflect the actual situation. This can be initiated from the
+CLI, too, mostly for development.
+
+

ocaml/idl/datamodel.ml

@@ -10441,6 +10441,8 @@ let all_system =
   ; Datamodel_repository.t
   ; Datamodel_observer.t
   ; Datamodel_vm_group.t
+  ; Datamodel_host_driver.t
+  ; Datamodel_driver_variant.t
   ]
 
 (* If the relation is one-to-many, the "many" nodes (one edge each) must come before the "one" node (many edges) *)
@@ -10533,6 +10535,7 @@ let all_relations =
   ; ((_network_sriov, "logical_PIF"), (_pif, "sriov_logical_PIF_of"))
   ; ((_certificate, "host"), (_host, "certificates"))
   ; ((_vm, "groups"), (_vm_group, "VMs"))
+  ; ((_driver_variant, "driver"), (_host_driver, "variants"))
   ]
 
 let update_lifecycles =
@@ -10688,6 +10691,8 @@ let expose_get_all_messages_for =
   ; _repository
   ; _vtpm
   ; _observer
+  ; _host_driver
+  ; _driver_variant
   ]
 
 let no_task_id_for = [_task; (* _alert; *) _event]

ocaml/idl/datamodel_common.ml

@@ -311,6 +311,10 @@ let _repository = "Repository"
 
 let _observer = "Observer"
 
+let _host_driver = "Host_driver"
+
+let _driver_variant = "Driver_variant"
+
 let update_guidances =
   Enum
     ( "update_guidances"