Cloud hypervisor

Cloud Hypervisor on Altra

Build cloud-hypervisor

Follow doc: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/building.md

Build dynamic linked binary

cargo build --release

file target/release/cloud-hypervisor
target/release/cloud-hypervisor: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=6188673dff9f9a438cc8bbef34eb769fb3d74575, for GNU/Linux 3.7.0, stripped

Build static linked binary

Need to use musl as C library. The original doc does not work. On ubunut-2004:

rustup target add aarch64-unknown-linux-musl
cargo build --release --target=aarch64-unknown-linux-musl --all
error occurred: Failed to find tool. Is `aarch64-linux-musl-gcc` installed?

Do bellow:

$ sudo apt install musl-tools
$ CC=musl-gcc cargo build --release --target=aarch64-unknown-linux-musl --all

$ file target/aarch64-unknown-linux-musl/release/cloud-hypervisor
target/aarch64-unknown-linux-musl/release/cloud-hypervisor: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=36e2a5f9869eb13fe004323c26794f1cbe2f6842, stripped

It is also possible to use docker build cloud hypervisor, e.g:

$ ./scripts/dev_cli.sh build --release --libc musl
[Cloud Hypervisor] Binaries placed under /home/adam/cloud_hyp/cloud-hypervisor/cloud-hypervisor/build/cargo_target/aarch64-unknown-linux-musl/release

$ size build/cargo_target/aarch64-unknown-linux-musl/release/cloud-hypervisor
   text    data     bss     dec     hex filename
2856088  382716    7168 3245972  318794 build/cargo_target/aarch64-unknown-linux-musl/release/cloud-hypervisor

Run cloud-hypervisor

./target/release/cloud-hypervisor \
          --kernel ../linux-cloud-hypervisor/arch/arm64/boot/Image  \
          --disk path=$ROOTFS path=/tmp/ubuntu-cloudinit.img  \
          --cmdline "console=hvc0 root=/dev/vda1 rw" \
          --cpus boot=4   \
          --memory size=0,shared=on  \
          --memory-zone id=mem0,size=1G,shared=on,host_numa_node=0 \
          --net "tap=,mac=,ip=,mask="

Control VM with ch-remote

We can use ch-remote tool to control an running Virtual Machine. See doc: https://www.cloudhypervisor.org/docs/prologue/commands/#ch-remote-binary

Start a VM with --api-socket option

sudo ./cloud-hypervisor ...  --api-socket /tmp/cloud-hypervisor.sock

sudo ./ch-remote --api-socket /tmp/cloud-hypervisor.sock reboot

Firmware boot

Rust Hypervisor Firmware

TBD - https://github.com/cloud-hypervisor/rust-hypervisor-firmware/issues/198, pretty new.

用hypervisor-fw_arm64,配置focal-server-cloudimg-arm64.raw,启动后马上报错
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: /EFI/BOOT/BOOTAA64.EFI

用hypervisor-fw_arm64和bionic-server-cloudimg-arm64.raw,可以成功启动vm
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: /EFI/BOOT/BOOTAA64.EFI
Executable loaded
error: no suitable video mode found.
error: no such device: root.

Press any key to continue...

uname -a
Linux cloud 4.15.0-106-generic

EDK2 Firmware

CH issue:The latest CLOUDHV_EFI.fd can not start aarch64 VM

TBD - https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/uefi.md#building-uefi-firmware-for-aarch64

使用CLOUDHV_EFI.fd启动VM时卡住,配合--serial tty --console off可以启动GRUB,但启动ubuntu时报错:

Synchronous Exception at 0x000000006B8D5624

Reference cloud-hypervisor/scripts/common-aarch64.sh,build cloudhv_efi.fd with old code,then VM can work:

Reset edk2 to 46b4606ba23498d3d0e66b53e498eb3d5d592586,reset edk2-platforms to 8227e9e9f6a8aefbd772b40138f835121ccb2307,
reset acpica to b9c69f81a05c45611c91ea9cbce8756078d76233.
Rebuild CLOUDHV_EFI.fd.

I have created a issue:https://github.com/cloud-hypervisor/cloud-hypervisor/issues/5202

Network

How to use macvtap:

https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/macvtap-bridge.md

More details:
https://github.com/cloud-hypervisor/cloud-hypervisor/discussions/5084

How does a virtual machine automatically obtain an IP address:

test_data/cloud-init/ubuntu/user-data
-        echo -n "@DEFAULT_TCP_LISTENER_MESSAGE" > /dev/tcp/@HOST_IP/@TCP_LISTENER_PORT
+        echo -n "booted" > /dev/tcp/192.168.2.1/8001

./scripts/create-cloud-init.sh
cp /root/workloads/focal-server-cloudimg-arm64-custom-20210929-0.raw /root/ch_files/osdisk.img
target/aarch64-unknown-linux-gnu/release/cloud-hypervisor \
        --api-socket /tmp/ch0 \
        --event-monitor path=/tmp/event.json \
        --cpus boot=4 \
        --memory size=4G,hotplug_method=virtio-mem,hotplug_size=32G \
        --balloon size=0 \
        --kernel /root/workloads/Image \
        --disk path=/root/ch_files/osdisk.img \
        --disk path=/tmp/ubuntu-cloudinit.img,iommu=on \
        --net id=net123,tap=,mac=12:34:56:78:90:ab,ip=192.168.2.1,mask=255.255.255.0 \
        --vsock cid=3,socket=/tmp/vsock \
        --cmdline "console=hvc0 root=/dev/vda1 rw systemd.journald.forward_to_console=1"

Run "nc -l 8001" on host,after receive "booted",I can connect the VM by `ssh cloud@192.168.2.2`

More details:https://github.com/cloud-hypervisor/cloud-hypervisor/discussions/5274

Q&A:

Q:What does "--net id=net123,tap=,mac=12:34:56:78:90:ab,ip=192.168.2.1,mask=255.255.255.0" mean?
A:We can find id0 in test_data/cloud-init/ubuntu/network-config.
  id0:
    match:
      macaddress: 12:34:56:78:90:ab
    addresses:
    - 192.168.2.2/24
    gateway4: 192.168.2.1
And some other ids with different mac and ip address.

Cloud-init detects the mac address "12:34:56:78:90:ab" and sets the specified ip address "192.168.2.2" to VM.

We can find "192.168.2.1" and "aa:16:ea:a7:03:93" after call ifconfig on host.

Please find more details in https://cloudinit.readthedocs.io/en/latest/

Performance test

Details:Cloud-Hypervisor Performance Metrics

VMM features

Details:Features-and-functionality

Errors and fix

Details:Errors-and-fix