Skip to content

Commit d5f67ad

Browse files
authored
Design proposal to support import/export of Qcow2 VDI (#6318)
At XCP-ng, we are working on overcoming the 2TiB limitation for VM disks while preserving essential features. However, we currently cannot export a VDI to a Qcow2 file, nor import one. The purpose of this design proposal is to outline a solution for implementing VDI import/export in Qcow2 format.
2 parents fd52035 + 61c6cd5 commit d5f67ad

File tree

1 file changed

+121
-0
lines changed

1 file changed

+121
-0
lines changed
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
---
2+
title: Add qcow tool to allow VDI import/export
3+
layout: default
4+
design_doc: true
5+
revision: 1
6+
status: proposed
7+
---
8+
9+
# Introduction
10+
11+
At XCP-ng, we are working on overcoming the 2TiB limitation for VM disks while
12+
preserving essential features such as snapshots, copy-on-write capabilities, and
13+
live migration.
14+
15+
To achieve this, we are introducing Qcow2 support in SMAPI and the blktap driver.
16+
With the alpha release, we can:
17+
- Create a VDI
18+
- Snapshot it
19+
- Export and import it to/from XVA
20+
- Perform full backups
21+
22+
However, we currently cannot export a VDI to a Qcow2 file, nor import one.
23+
24+
The purpose of this design proposal is to outline a solution for implementing VDI
25+
import/export in Qcow2 format.
26+
27+
# Design Proposal
28+
29+
The import and export of VHD-based VDIs currently rely on *vhd-tool*, which is
30+
responsible for streaming data between a VDI and a file. It supports both Raw and
31+
VHD formats, but not Qcow2.
32+
33+
There is an existing tool called [qcow-tool](https://opam.ocaml.org/packages/qcow-tool/)
34+
originally packaged by MirageOS. It is no longer actively maintained, but it can
35+
produce Qcow files readable by QEMU.
36+
37+
Currently, *qcow-tool* does not support streaming, but we propose to add this
38+
capability. This means replicating the approach used in *vhd-tool*, where data is
39+
pushed to a socket.
40+
41+
We have contacted the original developer, David Scott, and there are no objections
42+
to us maintaining the tool if needed.
43+
44+
Therefore, the most appropriate way to enable Qcow2 import/export in XAPI is to
45+
add streaming support to `qcow-tool`.
46+
47+
# XenAPI changes
48+
49+
## The workflow
50+
51+
- The export and import of VDIs are handled by the XAPI HTTP server:
52+
- `GET /export_raw_vdi`
53+
- `PUT /import_raw_vdi`
54+
- The corresponding handlers are `Export_raw_vdi.handler` and
55+
`Import_raw_vdi.handler`.
56+
- Since the format is checked in the handler, we need to add support for `Qcow2`,
57+
as currently only `Raw`, `Tar`, and `Vhd` are supported.
58+
- This requires adding a new type in the `Importexport.Format` module and a new
59+
content type: `application/x-qemu-disk`.
60+
See [mime-types format](https://www.digipres.org/formats/mime-types/#application/x-qemu-disk).
61+
- This allows the format to be properly decoded. Currently, all formats use a
62+
wrapper called `Vhd_tool_wrapper`, which sets up parameters for `vhd-tool`.
63+
We need to add a new wrapper for the Qcow2 format, which will instead use
64+
`qcow-tool`, a tool that we will package (see the section below).
65+
- The new wrapper will be responsible for setting up parameters (source,
66+
destination, etc.). Since it only manages Qcow2 files, we don’t need to pass
67+
additional format information.
68+
- The format (`qcow2`) will be specified in the URI. For example:
69+
- `/import_raw_vdi?session_id=<OpaqueRef>&task_id=<OpaqueRef>&vdi=<OpaqueRef>&format=qcow2`
70+
71+
## Adding and modifying qcow-tool
72+
73+
- We need to package [qcow-tool](https://opam.ocaml.org/packages/qcow-tool).
74+
- This new tool will be called from `ocaml/xapi/qcow_tool_wrapper.ml`, as
75+
described in the previous section.
76+
77+
- To export a VDI to a Qcow2 file, we need to add functionality similar to
78+
`Vhd_tool_wrapper.send`, which calls `vhd-tool stream`.
79+
- It writes data from the source to a destination. Unlike `vhd-tool`, which
80+
supports multiple destinations, we will only support Qcow2 files.
81+
- Here is a typicall call to `vhd-tool stream`
82+
```sh
83+
/bin/vhd-tool stream \
84+
--source-protocol none \
85+
--source-format hybrid \
86+
--source /dev/sm/backend/ff1b27b1-3c35-972e-76ec-a56fe9f25e36/87711319-2b05-41a3-8ee0-3b63a2fc7035:/dev/VG_XenStorage-ff1b27b1-3c35-972e-76ec-a56fe9f25e36/VHD-87711319-2b05-41a3-8ee0-3b63a2fc7035 \
87+
--destination-protocol none \
88+
--destination-format vhd \
89+
--destination-fd 2585f988-7374-8131-5b66-77bbc239cbb2 \
90+
--tar-filename-prefix \
91+
--progress \
92+
--machine \
93+
--direct \
94+
--path /dev/mapper:.
95+
```
96+
97+
- To import a VDI from a Qcow2 file, we need to implement functionality similar
98+
to `Vhd_tool_wrapper.receive`, which calls `vhd-tool serve`.
99+
- This is the reverse of the export process. As with export, we will only
100+
support a single type of import: from a Qcow2 file.
101+
- Here is a typical call to `vhd-tool serve`
102+
```sh
103+
/bin/vhd-tool serve \
104+
--source-format raw \
105+
--source-protocol none \
106+
--source-fd 3451d7ed-9078-8b01-95bf-293d3bc53e7a \
107+
--tar-filename-prefix \
108+
--destination file:///dev/sm/backend/f939be89-5b9f-c7c7-e1e8-30c419ee5de6/4868ac1d-8321-4826-b058-952d37a29b82 \
109+
--destination-format raw \
110+
--progress \
111+
--machine \
112+
--direct \
113+
--destination-size 180405760 \
114+
--prezeroed
115+
```
116+
117+
- We don't need to propose different protocol and different format. As we will
118+
not support different formats we just to handle data copy from socket into file
119+
and from file to socket. Sockets and files will be managed into the
120+
`qcow_tool_wrapper`. The `forkhelpers.ml` manages the list of file descriptors
121+
and we will mimic what the vhd tool wrapper does to link a UUID to socket.

0 commit comments

Comments
 (0)