Macvtap bridge for cloud hypvervisor

CLH VM network setup

macvtap

Follow doc: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/macvtap-bridge.md

net fd=[fd],mac=[MAC] [fd]<>/dev/tapX

Make sure do NOT run CLH with sudo, otherwise there will be error: Error booting VM: VmBoot(DeviceManager(CreateVirtioNet(TapError(IoctlError(2147767506, Os { code: 25, kind: Uncategorized, message: "Inappropriate ioctl for device" })))))

Refer to: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/4917#issuecomment-1442834740

[setup macvtap on host]
# The MAC address must be attached to the macvtap and be used inside the guest
mac="c2:67:4f:53:29:cb"
# Host network adapter to bridge the guest onto
host_net="enP4p4s0"

# Create the macvtap0 as a new virtual MAC associated with the host network
sudo ip link add link "$host_net" name macvtap0 type macvtap
sudo ip link set macvtap0 address "$mac" up
sudo ip link show macvtap0

# A new character device is created for this interface
tapindex=$(< /sys/class/net/macvtap0/ifindex)
tapdevice="/dev/tap$tapindex"

# Ensure that we can access this device
sudo chown "$UID.$UID" "$tapdevice"

[login as root, not sudo]
./target/release/cloud-hypervisor --kernel ../linux-cloud-hypervisor/arch/arm64/boot/Image --disk path=../focal-server-cloudimg-arm64.raw,iommu=on --disk path=/tmp/ubuntu-cloudinit.img --cmdline 'console=hvc0 root=/dev/vda1 rw' --cpus boot=4 --memory size=0,shared=on --memory-zone id=mem0,size=1G,shared=on,host_numa_node=0 --api-socket /tmp/cloud-hypervisor.sock --net fd=3,mac=c2:67:4f:53:29:cb 3<>/dev/tap9

On VM run dhcp to get ip address:

ip addr
sudo dhclient ens4p0

Hotplug macvtap:

Details in: https://github.com/cloud-hypervisor/cloud-hypervisor/discussions/5084#discussioncomment-4866815 Make sure boot VM with UEFI:

While ACPI requires UEFI booting. That means you must boot the VM from a firmware, not from the direct kernel. This restriction is only for AArch64. On X86, the VM booting from direct kernel can also play ACPI (and hotplug of course).

CLH integration test includes test_macvtap_hotplug:

./scripts/dev_cli.sh tests --integration -- --test-filter test_macvtap_hotplug

It calls:

docker run --workdir /cloud-hypervisor --rm --privileged --security-opt seccomp=unconfined --ipc=host --net=bridge --mount type=tmpfs,destination=/tmp --volume /dev:/dev --volume /home/dom/cloud-hypervisor/cloud-hypervisor:/cloud-hypervisor --volume /root/workloads:/root/workloads --env USER=root --env CH_LIBC=gnu cloudhypervisor/dev:20230116-0 ./scripts/run_integration_tests_aarch64.sh --hypervisor kvm --test-filter test_macvtap_hotplug

Or invoke the test directly:

USER=root CH_LIBC=gnu ./scripts/run_integration_tests_aarch64.sh --hypervisor kvm --test-filter test_macvtap_hotplug 2>&1 | tee tlog


`cargo test common_parallel::test_macvtap_hotplug --target aarch64-unknown-linux-gnu -- --nocapture 2>&1 | tee tlog_cargo`

command = sudo ip link delete guestmacvtap1 command = sudo ip link delete hostmacvtap1 command = sudo ip link add link enP9p3s0 name guestmacvtap1 type macvtap mod bridge command = sudo ip link set guestmacvtap1 address 12:34:56:78:90:01 up command = sudo ip link show guestmacvtap1 command = sudo chown $UID.$UID /dev/tap107 command = sudo ip link add link enP9p3s0 name hostmacvtap1 type macvtap mod bridge command = sudo ip address add 192.168.1.1/24 dev hostmacvtap1 command = sudo ip link set dev hostmacvtap1 up

target/aarch64-unknown-linux-gnu/debug/cloud-hypervisor" "--cpus" "boot=2" "--memory" "size=512M" "--kernel" "/root/workloads/CLOUDHV_EFI.fd" "--cmdline" "root=/dev/vda1 console=hvc0 rw systemd.journald.forward_to_console=1" "--disk" "path=/tmp/chtLUHf9/osdisk.img" "--disk" "path=/tmp/chtLUHf9/cloudinit" "--api-socket" "/tmp/chtLUHf9/cloud-hypervisor.sock" "-v" "-v"

Hotplug network device with ch-remote:

ch-remote /tmp/chtLUHf9/cloud-hypervisor.sock add-net fd=[3,4],mac=12:34:56:78:90:01,num_queues=4 guest:remote_command_w_output,{"id":"_net0","bdf":"0000:00:05.0"} guest:wait_vm_boot guest:ssh_command ip -o link child kill command = sudo ip link del guestmacvtap1 command = sudo ip link del hostmacvtap1

## Use network bridge instead of macvtap
Reference https://github.com/cloud-hypervisor/cloud-hypervisor/issues/4917

On host: sudo ip tuntap add mytap0 mode tap #Create tap sudo brctl addbr br0 #Create bridge sudo brctl addif br0 mytap0 enP9p3s0 #Connect tap and netcard with bridge sudo ifconfig mytap0 0 up #Clear IP of tap sudo ifconfig enP9p3s0 0 up #Clear IP of netcard sudo dhclient br0 #Get IP for bridge sudo dhclient enP9p3s0 #Get IP for netcard

Run cloud-hypervisor with `--net "tap=mytap0"`

On VM: sudo dhclient enp0s5 #You can find enp0s5 after run "ip addr"