|
| 1 | +--- |
| 2 | +title: Add qcow tool to allow VDI import/export |
| 3 | +layout: default |
| 4 | +design_doc: true |
| 5 | +revision: 1 |
| 6 | +status: proposed |
| 7 | +--- |
| 8 | + |
| 9 | +# Introduction |
| 10 | + |
| 11 | +At XCP-ng, we are working on overcoming the 2TiB limitation for VM disks while |
| 12 | +preserving essential features such as snapshots, copy-on-write capabilities, and |
| 13 | +live migration. |
| 14 | + |
| 15 | +To achieve this, we are introducing Qcow2 support in SMAPI and the blktap driver. |
| 16 | +With the alpha release, we can: |
| 17 | + - Create a VDI |
| 18 | + - Snapshot it |
| 19 | + - Export and import it to/from XVA |
| 20 | + - Perform full backups |
| 21 | + |
| 22 | +However, we currently cannot export a VDI to a Qcow2 file, nor import one. |
| 23 | + |
| 24 | +The purpose of this design proposal is to outline a solution for implementing VDI |
| 25 | +import/export in Qcow2 format. |
| 26 | + |
| 27 | +# Design Proposal |
| 28 | + |
| 29 | +The import and export of VHD-based VDIs currently rely on *vhd-tool*, which is |
| 30 | +responsible for streaming data between a VDI and a file. It supports both Raw and |
| 31 | +VHD formats, but not Qcow2. |
| 32 | + |
| 33 | +There is an existing tool called [qcow-tool](https://opam.ocaml.org/packages/qcow-tool/) |
| 34 | +originally packaged by MirageOS. It is no longer actively maintained, but it can |
| 35 | +produce Qcow files readable by QEMU. |
| 36 | + |
| 37 | +Currently, *qcow-tool* does not support streaming, but we propose to add this |
| 38 | +capability. This means replicating the approach used in *vhd-tool*, where data is |
| 39 | +pushed to a socket. |
| 40 | + |
| 41 | +We have contacted the original developer, David Scott, and there are no objections |
| 42 | +to us maintaining the tool if needed. |
| 43 | + |
| 44 | +Therefore, the most appropriate way to enable Qcow2 import/export in XAPI is to |
| 45 | +add streaming support to `qcow-tool`. |
| 46 | + |
| 47 | +# XenAPI changes |
| 48 | + |
| 49 | +## The workflow |
| 50 | + |
| 51 | +- The export and import of VDIs are handled by the XAPI HTTP server: |
| 52 | + - `GET /export_raw_vdi` |
| 53 | + - `PUT /import_raw_vdi` |
| 54 | +- The corresponding handlers are `Export_raw_vdi.handler` and |
| 55 | + `Import_raw_vdi.handler`. |
| 56 | +- Since the format is checked in the handler, we need to add support for `Qcow2`, |
| 57 | + as currently only `Raw`, `Tar`, and `Vhd` are supported. |
| 58 | +- This requires adding a new type in the `Importexport.Format` module and a new |
| 59 | + content type: `application/x-qemu-disk`. |
| 60 | + See [mime-types format](https://www.digipres.org/formats/mime-types/#application/x-qemu-disk). |
| 61 | +- This allows the format to be properly decoded. Currently, all formats use a |
| 62 | + wrapper called `Vhd_tool_wrapper`, which sets up parameters for `vhd-tool`. |
| 63 | + We need to add a new wrapper for the Qcow2 format, which will instead use |
| 64 | + `qcow-tool`, a tool that we will package (see the section below). |
| 65 | +- The new wrapper will be responsible for setting up parameters (source, |
| 66 | + destination, etc.). Since it only manages Qcow2 files, we don’t need to pass |
| 67 | + additional format information. |
| 68 | +- The format (`qcow2`) will be specified in the URI. For example: |
| 69 | + - `/import_raw_vdi?session_id=<OpaqueRef>&task_id=<OpaqueRef>&vdi=<OpaqueRef>&format=qcow2` |
| 70 | + |
| 71 | +## Adding and modifying qcow-tool |
| 72 | + |
| 73 | +- We need to package [qcow-tool](https://opam.ocaml.org/packages/qcow-tool). |
| 74 | +- This new tool will be called from `ocaml/xapi/qcow_tool_wrapper.ml`, as |
| 75 | + described in the previous section. |
| 76 | + |
| 77 | +- To export a VDI to a Qcow2 file, we need to add functionality similar to |
| 78 | + `Vhd_tool_wrapper.send`, which calls `vhd-tool stream`. |
| 79 | + - It writes data from the source to a destination. Unlike `vhd-tool`, which |
| 80 | + supports multiple destinations, we will only support Qcow2 files. |
| 81 | + - Here is a typicall call to `vhd-tool stream` |
| 82 | +```sh |
| 83 | +/bin/vhd-tool stream \ |
| 84 | + --source-protocol none \ |
| 85 | + --source-format hybrid \ |
| 86 | + --source /dev/sm/backend/ff1b27b1-3c35-972e-76ec-a56fe9f25e36/87711319-2b05-41a3-8ee0-3b63a2fc7035:/dev/VG_XenStorage-ff1b27b1-3c35-972e-76ec-a56fe9f25e36/VHD-87711319-2b05-41a3-8ee0-3b63a2fc7035 \ |
| 87 | + --destination-protocol none \ |
| 88 | + --destination-format vhd \ |
| 89 | + --destination-fd 2585f988-7374-8131-5b66-77bbc239cbb2 \ |
| 90 | + --tar-filename-prefix \ |
| 91 | + --progress \ |
| 92 | + --machine \ |
| 93 | + --direct \ |
| 94 | + --path /dev/mapper:. |
| 95 | +``` |
| 96 | + |
| 97 | +- To import a VDI from a Qcow2 file, we need to implement functionality similar |
| 98 | + to `Vhd_tool_wrapper.receive`, which calls `vhd-tool serve`. |
| 99 | + - This is the reverse of the export process. As with export, we will only |
| 100 | + support a single type of import: from a Qcow2 file. |
| 101 | + - Here is a typical call to `vhd-tool serve` |
| 102 | +```sh |
| 103 | +/bin/vhd-tool serve \ |
| 104 | + --source-format raw \ |
| 105 | + --source-protocol none \ |
| 106 | + --source-fd 3451d7ed-9078-8b01-95bf-293d3bc53e7a \ |
| 107 | + --tar-filename-prefix \ |
| 108 | + --destination file:///dev/sm/backend/f939be89-5b9f-c7c7-e1e8-30c419ee5de6/4868ac1d-8321-4826-b058-952d37a29b82 \ |
| 109 | + --destination-format raw \ |
| 110 | + --progress \ |
| 111 | + --machine \ |
| 112 | + --direct \ |
| 113 | + --destination-size 180405760 \ |
| 114 | + --prezeroed |
| 115 | +``` |
| 116 | + |
| 117 | +- We don't need to propose different protocol and different format. As we will |
| 118 | +not support different formats we just to handle data copy from socket into file |
| 119 | +and from file to socket. Sockets and files will be managed into the |
| 120 | +`qcow_tool_wrapper`. The `forkhelpers.ml` manages the list of file descriptors |
| 121 | +and we will mimic what the vhd tool wrapper does to link a UUID to socket. |
0 commit comments