This repository was archived by the owner on Jul 16, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 18
Cloud hypervisor
Dom edited this page Feb 28, 2023
·
77 revisions
Follow doc: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/building.md
cargo build --release
file target/release/cloud-hypervisor
target/release/cloud-hypervisor: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=6188673dff9f9a438cc8bbef34eb769fb3d74575, for GNU/Linux 3.7.0, stripped
Need to use musl as C library. The original doc does not work. On ubunut-2004:
rustup target add aarch64-unknown-linux-musl
cargo build --release --target=aarch64-unknown-linux-musl --all
error occurred: Failed to find tool. Is `aarch64-linux-musl-gcc` installed?
Do bellow:
$ sudo apt install musl-tools
$ CC=musl-gcc cargo build --release --target=aarch64-unknown-linux-musl --all
$ file target/aarch64-unknown-linux-musl/release/cloud-hypervisor
target/aarch64-unknown-linux-musl/release/cloud-hypervisor: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=36e2a5f9869eb13fe004323c26794f1cbe2f6842, stripped
It is also possible to use docker build cloud hypervisor, e.g:
$ ./scripts/dev_cli.sh build --release --libc musl
[Cloud Hypervisor] Binaries placed under /home/adam/cloud_hyp/cloud-hypervisor/cloud-hypervisor/build/cargo_target/aarch64-unknown-linux-musl/release
$ size build/cargo_target/aarch64-unknown-linux-musl/release/cloud-hypervisor
text data bss dec hex filename
2856088 382716 7168 3245972 318794 build/cargo_target/aarch64-unknown-linux-musl/release/cloud-hypervisor
./target/release/cloud-hypervisor \
--kernel ../linux-cloud-hypervisor/arch/arm64/boot/Image \
--disk path=$ROOTFS path=/tmp/ubuntu-cloudinit.img \
--cmdline "console=hvc0 root=/dev/vda1 rw" \
--cpus boot=4 \
--memory size=0,shared=on \
--memory-zone id=mem0,size=1G,shared=on,host_numa_node=0 \
--net "tap=,mac=,ip=,mask="
We can use ch-remote tool to control an running Virtual Machine. See doc: https://www.cloudhypervisor.org/docs/prologue/commands/#ch-remote-binary
Start a VM with --api-socket option
sudo ./cloud-hypervisor ... --api-socket /tmp/cloud-hypervisor.sock
sudo ./ch-remote --api-socket /tmp/cloud-hypervisor.sock reboot
TBD - https://github.com/cloud-hypervisor/rust-hypervisor-firmware/issues/198, pretty new.
用hypervisor-fw_arm64,配置focal-server-cloudimg-arm64.raw,启动后马上报错
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: /EFI/BOOT/BOOTAA64.EFI
用hypervisor-fw_arm64和bionic-server-cloudimg-arm64.raw,可以成功启动vm
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: /EFI/BOOT/BOOTAA64.EFI
Executable loaded
error: no suitable video mode found.
error: no such device: root.
Press any key to continue...
uname -a
Linux cloud 4.15.0-106-generic
使用CLOUDHV_EFI.fd启动VM时卡住,配合--serial tty --console off可以启动GRUB,但启动ubuntu时报错:
Synchronous Exception at 0x000000006B8D5624
Reference cloud-hypervisor/scripts/common-aarch64.sh,build cloudhv_efi.fd with old code,then VM can work:
Reset edk2 to 46b4606ba23498d3d0e66b53e498eb3d5d592586,reset edk2-platforms to 8227e9e9f6a8aefbd772b40138f835121ccb2307,
reset acpica to b9c69f81a05c45611c91ea9cbce8756078d76233.
Rebuild CLOUDHV_EFI.fd.
I have created a issue:https://github.com/cloud-hypervisor/cloud-hypervisor/issues/5202
Please check Cloud-Hypervisor Performance Metrics
TBD: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/snapshot_restore.md
Not ready for ARM64. Refer to: https://github.com/cloud-hypervisor/linux/pull/13
All pass
--cpus boot=4,topology=1:1:1:1
Error parsing config: Error validating configuration: Product of CPU topology parts does not match maximum vCPUs
--cpus features=amx
Error parsing config: Invalid feature in --cpus features list: amx
All pass.
sudo ./ch-remote --api-socket /tmp/ch-socket3 resize --cpus 8
Error running command: Server responded with an error: InternalServerError: ApiError(VmResize(CpuManager(VcpuCreate(Failed to create Vcpu: Device or resource busy (os error 16)
sudo ./ch-remote --api-socket /tmp/ch-socket3 add-fs tag=myfs,socket=/tmp/virtiofs.sock
Error running command: Server responded with an error: InternalServerError: ApiError(VmAddFs(ConfigValidation(VhostUserRequiresSharedMemory)))
sudo ./ch-remote --api-socket /tmp/ch-socket3 add-net tap=chtap0
{"id":"_net2","bdf":"0000:00:05.0"}
但VM没有新增虚拟网卡
sudo ./ch-remote --api-socket /tmp/ch-socket3 add-pmem file=/tmp/ubuntu-cloudinit_2.img
{"id":"_pmem4","bdf":"0000:00:06.0"}
sudo ./ch-remote --api-socket /tmp/ch-socket3 add-vsock cid=3,socket=/tmp/vsock.sock
命令运行成功,但不知道在VM中如何验证
--memory size=8G,hugepages=on 不识别hugepages=on,看help是支持的
Error booting VM: VmBoot(MemoryManager(SharedFileSetLen(Os { code: 22, kind: InvalidInput, message: "Invalid argument" })))
No test case.
--net ip=192.168.101.1
[ OK ] Started Network Time Synchronization.
[ OK ] Reached target System Time Set.
[ OK ] Reached target System Time Synchronized.
It will be blocked for a few minutes to wait for the network.
The document require to run cloud-hypervisor in VM,but it return error:
./cloud-hypervisor: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ./cloud-hypervisor)
./cloud-hypervisor: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./cloud-hypervisor)
./cloud-hypervisor: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./cloud-hypervisor)
TBD
Please reference Macvtap-bridge-for-cloud-hypvervisor
这两条命令在VM中的内存总量是1.5G,其他的都是1G
--memory size=1G,hotplug_method=virtio-mem,hotplug_size=1G,hotplugged_size=512M
--memory size=0,hotplug_method=virtio-mem --memory-zone id=mem0,size=1G,hotplug_size=1G,hotplugged_size=512M
--memory size=0 --memory-zone id=mem0,size=1G,hotplug_size=1G
cloud-hypervisor: 4.763259ms: <vmm> ERROR:vmm/src/memory_manager.rs:741 -- Invalid to set ACPI hotplug method for memory zones
Error booting VM: VmBoot(MemoryManager(InvalidHotplugMethodWithMemoryZones))
正确配置:
--memory size=0,hotplug_method=virtio-mem --memory-zone id=mem0,size=1G,hotplug_size=1G
--numa guest_numa_id=0
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', vmm/src/config.rs:1897:67
--numa guest_numa_id=0,cpus=[1-3,7] guest_numa_id=1,cpus=[0,4-6]
Unrecognized argument: guest_numa_id=1,cpus=[0,4-6]
--memory size=0 --memory-zone id=mem0,size=1G --memory-zone id=mem1,size=1G --memory-zone id=mem2,size=1G --numa guest_numa_id=0,memory_zones=[mem0,mem2] --numa guest_numa_id=1, memory_zones=mem1
thread 'vmm' panicked at 'called `Option::unwrap()` on a `None` value', arch/src/aarch64/fdt.rs:729:22
--sgx-epc id=epc0,size=32M id=epc1,size=64M id=epc2,size=32M
--numa guest_numa_id=0,sgx_epc_sections=epc1 --numa guest_numa_id=1,sgx_epc_sections=[epc0,epc2]
Arm不支持sgx
sudo ./cloud-hypervisor --api-socket /tmp/ch0 --restore source_url=file:///home/dom/cloud-hypervisor/bin/dir
Blocked,output nothing.
# First terminal
sudo ./cloud-hypervisor --api-socket /tmp/ch0
# Second terminal
sudo ./ch-remote --api-socket /tmp/ch0 restore source_url=file:///home/dom/cloud-hypervisor/bin/dir/
Return immediately,output nothing.
Please reference edk2-firmware
确认/sys/bus/pci/devices/0003:04:00.0/下有多个文件
用 --device path=/sys/bus/pci/devices/0003:04:00.0/启动vm
Error booting VM: VmBoot(DeviceManager(VfioCreate(OpenGroup(Os { code: 2, kind: NotFound, message: "No such file or directory" }, "44"))))
--device 'path=/sys/bus/pci/devices/0003\:04\:00.0/
Error booting VM: VmBoot(DeviceManager(VfioCreate(InvalidPath)))
--user-device socket=/tmp/vfio-user.sock
cloud-hypervisor: 97.668894ms: <vmm> WARN:vfio_user/src/lib.rs:627 -- Ignoring unsupported vfio region capability (id = '2')
thread 'vmm' panicked at 'not implemented', pci/src/vfio.rs:641:21
./scripts/rpc.py nvmf_subsystem_add_listener nqn.2019-07.io.spdk:cnode -t VFIOUSER -a /tmp/nvme-vfio-user/ -s 0
[2023-02-08 02:33:57.142873] vfio_user.c:4409:nvmf_vfio_user_listen: *ERROR*: /tmp/nvme-vfio-user/: error to mmap file /tmp/nvme-vfio-user//bar0: Invalid argument.
[2023-02-08 02:33:57.142990] nvmf.c: 676:spdk_nvmf_tgt_listen_ext: *ERROR*: Unable to listen on address '/tmp/nvme-vfio-user/'
--user-device socket=/tmp/nvme-vfio-user/cntrl
Error booting VM: VmBoot(DeviceManager(VfioUserCreateClient(Connect(Os { code: 2, kind: NotFound, message: "No such file or directory" }))))
cntrl不存在,不知道如何获取
VM可以启动,但linux无法启动,一直卡住
https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/virtiofs-root.md#to-create-the-vm-rootfs
for循环中所有提到的文件在host上都找不到.
TBD
- virtualbox已经启动,此时再启动vm,此处提示设备忙
thread 'vmm' panicked at 'called `Result::unwrap()` on an `Err` value: VmCreate(Device or resource busy (os error 16))', vmm/src/vm.rs:863:41
- 启动vm里的--disk所指定的文件不存在时报错:
Error booting VM: VmBoot(DeviceManager(Disk(Os { code: 2, kind: NotFound, message: "No such file or directory" })))
- VM netowrk还没有调通,参考
https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/macvtap-bridge.md
在启动vm时使用--net "tap=virbr,mac=52:54:00:12:34:56",
Error booting VM: VmBoot(DeviceManager(CreateVirtioNet(OpenTap(TapOpen(ConfigureTap(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }))))))
使用--net fd=3,mac=$mac 3<>$"$tapdevice"
Error booting VM: VmBoot(DeviceManager(CreateVirtioNet(TapError(IoctlError(2147767506, Os { code: 25, kind: Uncategorized, message: "Inappropriate ioctl for device" })))))
以下方法仅能在x86平台上验证pass,命令./ch-remote --api-socket=/tmp/ch-socket3 add-net tap=chtap1无法在arm VM上生成ens5. 另外,之前的命令中有一条brctl stp br0 on,导致交换机的端口直接关闭,任何设备接入该端口都没有网络连接,慎用.
sudo ./cloud-hypervisor \
--kernel ./hypervisor-fw \
--console off \
--serial tty \
--disk path=./focal-server-cloudimg-amd64.raw \
--cmdline "root=/dev/vda1 console=hvc0" \
--cpus boot=4 \
--memory size=1024M \
--api-socket=/tmp/ch-socket3
#Create chtap1 on host and create ens5 on VM
sudo ./ch-remote --api-socket=/tmp/ch-socket3 add-net tap=chtap1
## in the host machine
sudo brctl addbr br0 #add bridge
sudo brctl addif br0 eno2 #add an interface to bridge
sudo ifconfig eno2 0 up #The Ethernet physical interface becomes a logical port on the network bridge
#and a part of the logical bridge device, so the IP address is no longer required. Release these IP addresses
sudo dhclient br0 #get IP
route -n
sudo ip link set dev chtap1 up #enable chtap1
sudo brctl addif br0 chtap1 #add another interface to bridge
sudo iptables -P FORWARD ACCEPT #it is needed to connect to internet
## in the guest
sudo ip link set dev ens3 up
sudo dhclient ens3 # (or set static ip using this command: sudo ip addr add ***.***.***.***/24 dev ens3)
4.Proxy配置
[7] Couldn't connect to server (Failed to connect to 127.0.0.1 port 4781 after 0 ms: Connection refused); class=Net (12)
config.json 配置代理要用host的IP,而不是127.0.0.1
/root/.docker/config.json
{
"proxies":
{
"default":
{
"httpProxy": "socks5://10.0.0.41:4781",
"httpsProxy": "socks5://10.0.0.41:4781",
"noProxy": "*.test.example.com,.example2.com,127.0.0.0/8"
}
}
}
5.重启网卡
ifconfig eno2 0
会把eno2关闭,下次再打开,输入ifconfig eno2 up但此时没有获取IP,需要再执行dhclient eno2
6.wget不支持socks5代理
wget --quiet
该命令除了不打印下载过程,连报错也不打印,把quiet删了后才打印出error:
Error parsing proxy URL socks5://10.0.0.41:4781: Unsupported scheme 'socks5'.
7.wget配置http代理
Error parsing proxy URL socks5://10.0.0.41:4781: Unsupported scheme 'socks5'.
把/root/.docker/config.json里的socks5://10.0.0.41:4781换成 http://10.0.0.41:4780 就好了
cargo build --no-default-features --features kvm,mshv --all --release --target aarch64-unknown-linux-gnu
use of undeclared crate or module `mshv`
mshv是微软贡献 Linux 内核代码，可运行多个 Windows
mshv: 增加对检测嵌套的 hypervisor 的支持
hv, mshv : 改变嵌套 root 分区的中断向量
Test 'boot_time_ms' running .. (control: test_timeout = 2s, test_iterations = 10, overrides: )
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', test_infra/src/lib.rs:1232:18
把脚本改成cargo build --all --release --target $BUILD_TARGET
成功了几个测试,但还是失败了
Test 'boot_time_ms' running .. (control: test_timeout = 2s, test_iterations = 10, overrides: )
Test 'boot_time_ms' .. ok: mean = 413.76223999999996, std_dev = 28.015316271611148
Test 'boot_time_pmem_ms' running .. (control: test_timeout = 2s, test_iterations = 10, overrides: )
Test 'boot_time_pmem_ms' .. ok: mean = 18.412709999999997, std_dev = 0.21074413135364106
Test 'boot_time_16_vcpus_ms' running .. (control: test_timeout = 2s, test_iterations = 10, overrides: )
Test 'boot_time_16_vcpus_ms' .. ok: mean = 749.25443, std_dev = 144.4245624768658
Test 'boot_time_16_vcpus_pmem_ms' running .. (control: test_timeout = 2s, test_iterations = 10, overrides: )
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
left: `1`,
right: `2`: Expecting two matching lines for 'Debug I/O port: Kernel code'', performance-metrics/src/performance_tests.rs:176:9
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
- uart error 启动vm命令:
sudo ./target/release/cloud-hypervisor \
--kernel ../linux-cloud-hypervisor/arch/arm64/boot/Image \
--disk path=$ROOTFS --disk path=/tmp/ubuntu-cloudinit.img \
--cmdline "console=hvc0 root=/dev/vda1 rw" \
--cpus boot=4 \
--memory size=0,shared=on \
--memory-zone id=mem0,size=1G,shared=on,host_numa_node=0 \
--net "tap=,mac=,ip=,mask=" \
--serial tty \
--console off
vm启动几秒就卡住
[ 1.200267] EXT4-fs (vda1): re-mounted. Opts: (null). Quota mode: disabled.
[ 1.888490] squashfs: SQUASHFS error: Xattrs in filesystem, these will be ignored
[ 1.890036] unable to read xattr id index table
or
[ 0.001761] Console: colour dummy device 80x25
[ 0.002461] printk: console [tty0] enabled
[ 0.003070] printk: bootconsole [pl11] disabled
cloud-hypervisor: 698.505152ms: <vcpu2> WARN:devices/src/legacy/uart_pl011.rs:358 -- [Debug I/O port: Kernel code: 0x41] 0.695055 seconds
通过kill进程的方式停止vm,可以打印以下错误:
VMM thread exited with error: Error shutting down VMM: SetTerminalCanon(Error(5))
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error(5)', src/main.rs:594:50
相关代码如下:
589 // SAFETY: trivially safe
590 let on_tty = unsafe { libc::isatty(libc::STDIN_FILENO) } != 0;
591 if on_tty {
592 // Don't forget to set the terminal in canonical mode
593 // before to exit.
594 std::io::stdin().lock().set_canon_mode().unwrap();
595 }