Skip to content

Design proposal for supported image formats #6308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 11, 2025
+76 −0
Merged

Design proposal for supported image formats #6308

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions doc/content/design/sm-supported-image-formats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
title: Add supported image formats in sm-list
layout: default
design_doc: true
revision: 2
status: proposed
---

# Introduction

At XCP-ng, we are enhancing support for QCOW2 images in SMAPI. The primary
motivation for this change is to overcome the 2TB size limitation imposed
by the VHD format. By adding support for QCOW2, a Storage Repository (SR) will
be able to host disks in VHD and/or QCOW2 formats, depending on the SR type.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would we then need a field on the SR as well to indicate, which format is used? I assume that when creating an SR, you'll want to be able to choose a format? You could then, for example, create an "NFS QCOW2 SR" or an "NFS VHD SR". Or would a single SR be able to contain a mix of VDIs using different supported types, and if so, how would that work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A single SR will be able to contain a mix of VDIs. In our first version (it is an alpha version: there is a default format. So you have your current SR that uses VHD, then you install the package with qcow2 support. And now by default you will create qcow2 file. So you will still be able to use the VHD disks and at the same time you are now creating qcow2 file. In the future (but it was not the scope of my proposition but maybe it has to be) we will be able to choose the format when creating a VDI. So in the future (but it can be discussed here no worries) we will need to add a parametre to choose the kind of format when the VDI is created and the destination format in case of migration... It was not the scope of the version1 of this proposition but the more we discussed and the more it makes sense to add this part too.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A single SR can have multiple types of image format, like it can have RAW and VHD at the moment, it will just be QCOW2, RAW and VHD. With one preferred for default when not specifying which one to create with VDI creation.
For example: xe vdi-create sr-uuid=3ef2825a-f700-4224-ce72-91174581acc7 type=user sm-config:type=qcow2 virtual-size=1GiB name-label="Test QCOW2"

In the future, additional formats—such as VHDx—could also be supported.

We need a mechanism to expose to end users which image formats are supported
by a given SR. The proposal is to extend the SM API object with a new field
that clients (such as XenCenter, XenOrchestra, etc.) can use to determine the
available formats.

# Design Proposal

To expose the available image formats to clients (e.g., XenCenter, XenOrchestra, etc.),
we propose adding a new field called `supported-image-formats` to the Storage Manager (SM)
module. This field will be included in the output of the `SM.get_all_records` call.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The datamodel field must have underscores, so supported_image_formats, while the CLI normally uses hyphens.


The `supported-image-formats` field will be populated by retrieving information
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to check that I understand this field correctly, if a user wants to use qcow2 in a, say, NFS SR, all the VDIs created on that SR will now be qcow2? Will this affect the way the SMAPI drivers handle calls? For example, I imagine VDI.compose would work differently on VHD and QCOW2? Will this be handled by the forked SM?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes in the first version it is working like that. But it is not the scope of this proposition. Here the goal is just to expose supported format nothing more in reality. But you are right and behind that we are implementing the qcow2 support. So to answer we do that because when you enable the qcow2 support you will use qcow2 format on your SR (NFS is a good example). And the point is that you can still use your former VHD disks. So the transition is kind of transparent. And yes the SMAPI driver will be able to handle both format. I let @Wescoeur , @AnthoineB and @Nambrok answered to add details or correct me about this :)

from the SMAPI drivers. Specifically, each driver will update its `DRIVER_INFO`
dictionary with a new key, `supported_image_formats`, which will contain a list
of strings representing the supported image formats
(for example: `["vhd", "raw", "qcow2"]`).

The list designates the driver's preferred VDI format as its first entry. That
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The list" here refers to what is returned from the SMAPI. It looks like the intention in to put this straight into a xapi DB field, such that it can be queried through the XenAPI. However, the xapi datamodel uses sets rather than lists, so you cannot rely on a particular order. If you want to mark a particular format as preferred, you could add a separate field for that (e.g. SM.preferred_image_format).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I think we will have a version 2 of the proposition ;)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, is there a way for the user to specify his/her preference?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No not with this field. I think that if a user want to specify his/her preference it will be done by using for example a new parameter in vdi pool migrate. So it is not in the scope of this proposition. The scope of this proposition is just how to expose the format supported by a storage driver.

means that when migrating a VDI, the destination storage repository will
attempt to create a VDI in this preferred format. If the default format cannot
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the intention to automatically change the format of a VDI to the "preferred" format when doing a storage migration? Is there no choice or desire to keep the current format of the VDI?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently the "preferred" format is imposed by the storage driver. So we cannot change it. But yes in the next release it makes sense to at least be able to either keep the same format or use the qcow2 format.
So the current design does not take this into account but I think that effectively adding two fields for the destination format during the migration as well as a field for the format used when creating the VDI seems important finally.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just because of how migration work at the moment, creating a new VDI (with the default preferred format) and streaming the data of the old VDI in the new one.

be used (e.g., due to size limitations), an error will be generated.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this needs to be part of the precheck in migration, but I think the idea here is fine


If a driver does not provide this information (as is currently the case with existing
drivers), the default value will be an empty array. This signifies that it is the
driver that decides which format it will use. This ensures that the modification
remains compatible with both current and future drivers.

With this new information, listing all parameters of the SM object will return:

```bash
# xe sm-list params=all
```

will output something like:

```
uuid ( RO) : c6ae9a43-fff6-e482-42a9-8c3f8c533e36
name-label ( RO) : Local EXT3 VHD
name-description ( RO) : SR plugin representing disks as VHD files stored on a local EXT3 filesystem, created inside an LVM volume
type ( RO) : ext
vendor ( RO) : Citrix Systems Inc
copyright ( RO) : (C) 2008 Citrix Systems Inc
required-api-version ( RO) : 1.0
capabilities ( RO) [DEPRECATED] : SR_PROBE; SR_SUPPORTS_LOCAL_CACHING; SR_UPDATE; THIN_PROVISIONING; VDI_ACTIVATE; VDI_ATTACH; VDI_CLONE; VDI_CONFIG_CBT; VDI_CREATE; VDI_DEACTIVATE; VDI_DELETE; VDI_DETACH; VDI_GENERATE_CONFIG; VDI_MIRROR; VDI_READ_CACHING; VDI_RESET_ON_BOOT; VDI_RESIZE; VDI_SNAPSHOT; VDI_UPDATE
features (MRO) : SR_PROBE: 1; SR_SUPPORTS_LOCAL_CACHING: 1; SR_UPDATE: 1; THIN_PROVISIONING: 1; VDI_ACTIVATE: 1; VDI_ATTACH: 1; VDI_CLONE: 1; VDI_CONFIG_CBT: 1; VDI_CREATE: 1; VDI_DEACTIVATE: 1; VDI_DELETE: 1; VDI_DETACH: 1; VDI_GENERATE_CONFIG: 1; VDI_MIRROR: 1; VDI_READ_CACHING: 1; VDI_RESET_ON_BOOT: 2; VDI_RESIZE: 1; VDI_SNAPSHOT: 1; VDI_UPDATE: 1
configuration ( RO) : device: local device path (required) (e.g. /dev/sda3)
driver-filename ( RO) : /opt/xensource/sm/EXTSR
required-cluster-stack ( RO) :
supported-image-formats ( RO) : vhd, raw, qcow2
```

This change impacts the SM data model, and as such, the XAPI database version will
be incremented.

# Impact

- **Data Model:** A new field (`supported-image-formats`) is added to the SM records.
- **Client Awareness:** Clients like the `xe` CLI will now be able to query and display the supported image formats for a given SR.
- **Database Versioning:** The XAPI database version will be updated to reflect this change.

Loading