Merge master into feature/perf and fix conflicts #6265
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ypes of devices, not only disks). Signed-off-by: Konstantina Chremmou <konstantina.chremmou@cloud.com>
Signed-off-by: Konstantina Chremmou <konstantina.chremmou@cloud.com>
The syslog feature allows to run a program and redirect its output/error stream to system log. It's triggered passing "syslog_stdout" argument to "Forkhelpers" functions like "execute_command_get_output". To test this feature use a small preload library to redirect syslog writing. This allows to test program without changing it. The log is redirected to /tmp/xyz instead of /dev/log. The name was chosen to allow for future static build redirection. The C program is used only for the test, so the code style take into account this (specifically it does not try to handle all redirect situation and all error paths). Signed-off-by: Frediano Ziglio <frediano.ziglio@cloud.com>
Provide an API/CLI for multi-version drivers. Two new classes represent them: Host_driver and Driver_variant. See * doc/content/toolstack/features/MVD/index.md for technical background and feature overview. The current implementation mocks the missing drivertool that manipulates the file system and observes driver state. We expect to integrate this at a later point. Consequently, the current implementation is incomplete and exists to faciliate development of API and XE clients. On Xapi start and restart driver information is purged from the database and re-built from scrach. This could be improved in the future to be incremental. Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
When scanning a host for drivers, don't remove and re-create them but instead update an entry if it already exists. Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
When scanning a host for drivers, don't remove and re-create them but instead update an entry if it already exists. Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
Address review comments; fix typos and parameter names. Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
This fixes a typo in list of fields we show in the CLI. Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
Re-generate datamodel_lifecycle.ml to reflect correctly the current version of the API. Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
Co-authored-by: Pau Ruiz Safont <psafont@users.noreply.github.com> Signed-off-by: Christian Lindig <christian.lindig@citrix.com>
Co-authored-by: Pau Ruiz Safont <psafont@users.noreply.github.com> Signed-off-by: Christian Lindig <christian.lindig@citrix.com>
Co-authored-by: Pau Ruiz Safont <psafont@users.noreply.github.com> Signed-off-by: Christian Lindig <christian.lindig@citrix.com>
Fix documentation for select() method. Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
Use more existing functions to format records. Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
…e.ml Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
…y the API. (xapi-project#6219) With regard to the error removal, these entries had been added to the SDK to account for specialised error messages needed by XenCenter, which was not the right thing to do; this should have been done on the implementing client side, not the SDK.
MVD CP-52334 multi-version driver API/CLI
Provide an API/CLI for multi-version drivers. Two new classes represent
them: Host_driver and Driver_variant. See
* doc/content/toolstack/features/MVD/index.md
for technical background and feature overview.
The current implementation mocks the missing drivertool that manipulates
the file system and observes driver state. We expect to integrate this
at a later point. Consequently, the current implementation is incomplete
and exists to faciliate development of API and XE clients.
On Xapi start and restart driver information is purged from the
database and re-built from scrach. This could be improved in the future
to be incremental.
Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
The previous feature processing logic was a bit ad hoc with various random conversion functions. Abstract them into one module and refactor a few functions to make their uses more explicit. Also try to encourage users, when processing features, to convert them into `Feature.t` and use the provided functions in this module rather than writing `List.mem` directly. Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
This is so that `Storage_migrate` can use this function, otherwise there would be a dependency cycle. This function also sounds like a utility function. Also add an mli file for `Storage_utils`. Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
This includes:
- VDI.compose for composing VDIs. Note the ordering of the parameter
`parent` and `child` is reversed, due to the inconsistency between
SMAPIv2 and SMAPIv3
- DP.attach_info which calls the idempotent DP.attach3 on a already
attached VDI to get the attach info again.
- VDI.{add_to,remove_from}sm_config for xapi to keep track of the mirror
vdi
- VDI.set_content_id for determining similarity between VDIs
Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
This API call prepares a nbd server that will be used for mirroring. This gets mapped to different logic on SMAPIv1 and SMAPIv3. - for SMAPIv1, this will reuse the old logic of xapi using Tapctl to retrieve this information from tapdisk - for SMAPIv3, this uses the new `import_activate` call to prepare an nbd server Also adjust the position of the module `DATA` to use the `DP` module. Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
Previously we hardcoded the export to be empty string when the nbd client negotiates with the server. This patch allows specifying this as a protocol parameter. This is useful for SXM of SMAPIv3 SRs when we copy the snapshot from source to destination, where the export name is needed when we copy using nbd. Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
This is the main commit that implements the tracking logic of SXM for inbound SMAPIv3 SRs. It mainly consists of the following changes: - Adding parameters to various SMAPIv2 calls to track the domain slice that is invoking the SMAPI calls. For SXM, we track two particular domains: the mirror domain and the copy domain, corresponding to two of the operations happening during migration: mirror of the VDI and copy of the base image. Although this was not needed for SMAPIv1 calls, it is now necessary for SMAPIv3 calls since it requires explicit tracking of the domain slices. - Extend various SMAPIv2 calls with the new domain parameter since SMAPIv3 has one qemu-dp process per domain, and therefore requires us to keep track of the domain parameter. - Invoking the `import_activate` call in the new `nbd_handler` on the receiving end to prepare an nbd server for mirroring. Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
SXM using qemu datapaths is done in two parts: inbound and outbound, hence the storage side has declared a `VDI_MIRROR_IN` features. Implement the checking logic for this feature here. This is done in two parts: 1. Check the relevant SM for the feature `VDI_MIRROR_IN`. This is done on the destination host. If this fails, then storage migration will be prevented from happening in the beginning. Consider this as a pre-check. 2. Check in xapi-storage-script that the chosen datapath supports mirroring. Technically mirroring is a datapath feature so it makes sense to check it there. But currently there is no easy way to register a datapath feature into the SM object in xapi database, so instead the volume plugins for SMAPIv3 SRs would also declare this feature. However, we still need a final check on the datapath features to make sure the selected datapath really supports this feature. Note this check happens while trying to set up the mirror, after storage migration has started, and will give a relatively cryptic error to the user. The end result of this is that if either the volume plugin does not declare `VDI_MIRROR_IN`/`VDI_MIRROR` or the dp plugin does not declare any of these two, SXM will fail (upfront/mid way). Note current SXM of SMAPIv1 SRs are not affected since `VDI_MIRROR` is a superset of `VDI_MIRROR_IN`. Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
Signed-off-by: Mark Syms <mark.syms@cloud.com>
xapi-project#6245) …le to XenServerTest only when specified. Making the internals visible to a friend assembly works only if both assemblies are signed with a strong name or both unsigned. The code won't compile in scenarios where only the source code needs to be built and signed with a .snk key, therefore I'm suggesting using a preprocessor directive that can be switched on and off by the build caller (the other option is to put both source and test projects in a solution and build and sign both at all times, but at the moment I think we can go with the easy option).
These were added for the SMAPIv3-inbound VM migrations Signed-off-by: Pau Ruiz Safont <pau.ruizsafont@cloud.com>
These were added for the SMAPIv3-inbound VM migrations
On all platforms tested the canonical location for dracut is `/usr/bin/dracut`. Older platforms symlink to it from `/usr/sbin/dracut`, whilst newer ones do not, thus on these platforms xapi is unable to find dracut and fails to start. Adjust the path to the canonical location to ensure it operates correctly on all platforms. Also adjust a reference in network_utils.ml (whilst newer platforms also seem to have the symlink from `/sbin/dracut`, it would seem to make sense to point to the canonical location regardless). Signed-off-by: Alex Brett <alex.brett@cloud.com>
Minor updates of the Hugo documentation: - The current Ubuntu snap package of Hugo is not supported by the docs. We should take a first minor step towards fixing this. - `doc/README.md` is outdated and should be updated. It says that the Ubuntu snap of Hugo works, but it does not anymore. Fix this by updating the outdated information. - An initial fix is to update the Relearn theme from 5.20.x to 5.23.0: - It does not introduce breaking changes. - It introduces more straightforward page links and deprecates older syntax. - Fix the warnings by updating relative links accordingly. Signed-off-by: Bernhard Kaindl <bernhard.kaindl@cloud.com>
On all platforms tested the canonical location for dracut is `/usr/bin/dracut`. Older platforms symlink to it from `/usr/sbin/dracut`, whilst newer ones do not, thus on these platforms xapi is unable to find dracut and fails to start. Adjust the path to the canonical location to ensure it operates correctly on all platforms. Also adjust a reference in network_utils.ml (whilst newer platforms also seem to have the symlink from `/sbin/dracut`, it would seem to make sense to point to the canonical location regardless).
Signed-off-by: Bernhard Kaindl <bernhard.kaindl@cloud.com>
Signed-off-by: Bernhard Kaindl <bernhard.kaindl@cloud.com>
…6251) Minor updates of the Hugo documentation: - The current Ubuntu `snap` package of Hugo is not supported by the docs. We should take a first minor step towards fixing this. - [`doc/README.md`](https://github.com/xapi-project/xen-api/blob/master/doc/README.md) is outdated and should be updated ([new version](https://github.com/xenserver-next/xen-api/blob/docs-update-README-and-hugo-relearn-to-5.23/doc/README.md)). It says that the Ubuntu snap of Hugo works, but it does not anymore. Fix this by updating the outdated information. - An initial fix is to update the Relearn theme from 5.20.x to 5.23.0: - It does not introduce breaking changes. - It introduces more straightforward page links and deprecates older syntax. - Fix the warnings by updating relative links accordingly. A preview is available on my site: https://xenserver-next.github.io/xen-api/index.html
Signed-off-by: Bernhard Kaindl <bernhard.kaindl@cloud.com>
…instead Signed-off-by: Bernhard Kaindl <bernhard.kaindl@cloud.com>
When the coordinator restarts. the no-other-masters check in the startup sequence does two things for each pool member: 1. It checks that the host agrees that it is are not the coordinator. 2. It unblocks the host's master_connection thread, which is likely waiting for a reconnection delay to expire, which may be up to 256 seconds (exponential backoff is used). The delay is interrupted to immediately unblock DB calls. Licensing initialisation comes earlier in the startup sequence, but under certain circumstance make calls to other host, in particular after an upgrade. A this time, hosts may still be blocked on the master_connection for up to 256 s, which adds an unnecessary delay to the coordinator's startup sequence and therefore the usability of the API. Address this by reversing the order of the two startup actions. Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
…instead (xapi-project#6255) For links to internal pages, `{{site.baseurl}}` (without the leading `.`) does not work in current Hugo, and it is better to replace it by using relative links: `{{.Site.BaseURL}}` is replaced in the rendered output, but it doesn't work when viewing and editing the files using Markdown editors and viewers like on GitHub/Lab as well as IDEs with improved support for Markdown editing(in editor and preview). As the relative links work in all cases like in the editors and previews, fix those (currently broken) links to use the relative links instead. While at those, also replace the link name "here" with a more specific link name. PS: The only place were `{{.Site.BaseURL}}` or similar is needed is the link in the `logo.html` template for `img="{{.Site.BaseURL}}/images/xapi-project.png"`
…oject#6257) When the coordinator restarts. the no-other-masters check in the startup sequence does two things for each pool member: 1. It checks that the host agrees that it is are not the coordinator. 2. It unblocks the host's master_connection thread, which is likely waiting for a reconnection delay to expire, which may be up to 256 seconds (exponential backoff is used). The delay is interrupted to immediately unblock DB calls. Licensing initialisation comes earlier in the startup sequence, but under certain circumstance make calls to other host, in particular after an upgrade. A this time, hosts may still be blocked on the master_connection for up to 256 s, which adds an unnecessary delay to the coordinator's startup sequence and therefore the usability of the API. Address this by reversing the order of the two startup actions.
The call currently calls `network.attach` on all networks, but this fails for PIFs that are not managed by xapi. Simply skip those. Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
…api-project#6259) The call currently calls `network.attach` on all networks, but this fails for PIFs that are not managed by xapi. Simply skip those.
This reverts commit 889dfa6. As there is no need to clean up the source VM earlier at all, VM_save already does the job of deactivating the source VM datapath. Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
Previously the `post_detach_hook` was run after the VDI is detached on the source VM, which is at the very end of the SXM, where the source VM is shutdown. However, the job of `post_detach_hook` is to call `Remote.receive_finalize` which will destroy the mirroring datapath. This should have been called as soon as we deactivate the datapath on the source VM, at which point the source VM will stop writing using that datapath. This commit changes `post_detach_hook` to `post_deactivate_hook` and moves its calling locations accordingly. Signed-off-by: Vincent Liu <shuntian.liu2@cloud.com>
…api-project#6254) Optional improvement for fast CI completion when only Hugo docs change on push/PRs: When only Hugo docs change on push and PRs, other workflows can be skipped using `paths-ignore:` for `docs/**` and `.github/workflows/hugo.yml` to not trigger other workflows if only these paths change when pushing to a branch / work on a PR. This is completely optional, but would be nice to have when only working on Hugo docs. It does not change scheduled CI runs. It changes only on-demand push/PR workflow runs.
…t_deactivate (xapi-project#6260) There are two parts to this PR: 1. Reverts the VM migration ordering change, which was previously thought to be necessary as I did not see `VM_save` would actually deactivate the source VM datapath. Thanks @edwintorok for pointing that out. 2. Change `post_detach_hook` to `post_deactivate_hook`, as this should have been called as soon as the datapath is deactivated, at which point all r/w using that datapath will be stopped. More details in the commit message.
Signed-off-by: Konstantina Chremmou <konstantina.chremmou@cloud.com>
Signed-off-by: Edwin Török <edwin.torok@cloud.com>
contificate
approved these changes
Jan 31, 2025
lindig
approved these changes
Jan 31, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The conflict resolution can be reviewed by
git log --remerge-diff -1: