Skip to content

A QEMU dual-GPU setup guide that works with some anti-VM software, and is more performant than other guides

License

Notifications You must be signed in to change notification settings

felikcat/qemu-hidden

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 

Repository files navigation

About

This guide’s goal is to be the easiest to follow, and to bypass some VM detections without slowing down performance. The guide built on CachyOS using systemd-boot and KDE Plasma desktop.

There will be no custom compiling QEMU in attempts to hide it. Later on, there may be an accompanying bootkit that I wrote to hide as many QEMU traces as I can muster. This’ll take a long time, probably a year from now, so don’t expect it any time soon or at all.

The host GPU is from AMD, while the guest (VM / Virtual Machine / VFIO) GPU is from NVIDIA.

ℹ️
These instructions also work for GPUs from Intel or AMD, but there’s no advice on resolving the reset bug or other problems they may have.

The operating system of choice is Windows 10 LTSC 2019, get it from the MyDigitalLife forums. Please don’t use Windows 11, it will not work without purposefully sacrificing your performance after the June 2025 updates.

1. GPU passthrough

Programs such as OBS Studio will keep trying to use NVIDIA drivers instead of the AMD drivers if the LIBVA_DRIVER_NAME is still set to nvidia, in which the libva-nvidia-driver package enforces; for OBS, this means you can’t record using GPU hardware acceleration:
sudo pacman -R libva-nvidia-driver

Create the directory if it doesn’t exist:
mkdir -p ~/.config/environment.d

Edit ~/.config/environment.d/99-amdgpu.conf

LIBVA_DRIVER_NAME=radeonsi

sudo rm /etc/profile.d/nvidia-vaapi.sh

This will fix the AMD GPU issue with OBS Studio after a reboot (don’t reboot now).

Find the IDs to pass to vfio-pci for your guest GPU:
lspci -vnn

For this example, you would passthrough the IDs 10de:2705 and 10de:22bb:

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD103 [GeForce RTX 4070 Ti SUPER] [10de:2705] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:413d]
        Flags: bus master, fast devsel, latency 0, IRQ 101, IOMMU group 21
        Memory at dd000000 (32-bit, non-prefetchable) [size=16M]
        Memory at f800000000 (64-bit, prefetchable) [size=16G]
        Memory at fc00000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000 [size=128]
        Expansion ROM at de000000 [virtual] [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nouveau, nvidia_drm, nvidia

05:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22bb] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:413d]
        Flags: bus master, fast devsel, latency 0, IRQ 169, IOMMU group 21
        Memory at de080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

Sudo edit: /etc/modprobe.d/vfio.conf

softdep drm pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci

# Change these PCIe IDs to yours!
options vfio-pci ids=10de:2705,10de:22bb

Sudo edit: /etc/mkinitcpio.conf

# Replace MODULES=() with the following:
# MODULES=(vfio_pci vfio vfio_iommu_type1)
# If you already have modules in there, put the new vfio entries before those modules.

# Check to see if 'modconf' is already included in HOOKS=(), it's necessary.

We’re going to do a couple extra changes here instead of later on to prevent one extra reboot.

Sudo edit: /etc/sysctl.d/99-qemu.conf

# Prevents GPU timeouts or other VM hanging while running video games.
kernel.split_lock_mitigate=0

# Replicates the value set by default in SteamOS. Not strictly necessary but here just incase.
vm.max_map_count = 2147483642

Sudo edit: /etc/modprobe.d/qemu.conf

# Prevents Windows BSODs and performance decreases in instances of MSRs faults.
options kvm ignore_msrs=Y report_ignored_msrs=N kvmclock_periodic_sync=N

# Improves VM performance and lowers DPC latency drastically.
options kvm_amd npt=1 avic=1

# Pause Loop Exit is useful when the CPU is overcommitted (with how a gaming VM is setup, it won't be), such as multiple VMs accessing the same CPU affinities; this lowers DPC latency, which is important for gaming.
options kvm_intel ple_gap=0 ple_window=0

Sudo edit: /etc/sdboot-manage.conf.d/99-qemu.conf

# Remove intel_iommu=on if using an AMD CPU.
LINUX_OPTIONS="intel_iommu=on iommu=pt tsc=reliable no_timer_check"

Apply the bootloader kernel option changes:
sudo sdboot-manage gen

Apply the new modprobe.d and mkinitcpio.conf changes:
sudo mkinitcpio -P

Reboot your PC:
reboot

2. Virtual machine requisites

Install the necessary packages:
sudo pacman -S qemu-desktop virt-manager iptables-nft dnsmasq virglrenderer hwloc dmidecode usbutils swtpm samba

Assign your user to the correct groups to prevent issues in the future:
sudo usermod -a -G qemu,video,kvm,libvirt,libvirt-qemu "$USER"

Sudo edit: /etc/libvirt/network.conf, and change firewall_backend to iptables. This is necessary since we’re using iptables-nft and not nftables standalone.

Enable and start the required services to use Virtual Machine Manager:
sudo systemctl enable --now libvirtd.service

3. Virtual machine setup

Enable XML editing in Virtual Machine Manager before doing anything else: Edit → Preferences → Check "Enable XML editing", then close.

Ensure the disk file location you’ll use for the VM is mounted to /mnt instead of /run/media, or you could pass-through an entire NVMe disk. Up to you.

When creating a disk file, ensure its Format is 'raw' instead of 'qcow', and use this calculator to get exact disk size measurements. VirtualBox is the only good option for snapshots in my opinion, so if you do malware analysis, that’s the way to go.

Add an additional SATA CDROM to the VM, so you can load the latest virtio-win-*.iso into it; get the ISO from here. At first this is used so you have disk drivers, but later on you can install all of its drivers inside of the VM by running virtio-win-guest-tools (after Windows is installed).

Add the PCI Host Device for your NVIDIA GPU and its accompanying audio device.

Install Windows 10 LTSC 2019 as it is the best edition to use for performance and stability.

⚠️
Make sure the disk type is virtio before installing Windows 10, otherwise you have to follow these steps to correct this mistake later for higher performance: https://superuser.com/a/1253728

4. Looking Glass setup

paru -S looking-glass looking-glass-module-dkms obs-plugin-looking-glass

Follow through the official documentation but skip past the "Installing" step as you’ve already done that the Arch way.

ℹ️
Run sudo chmod 0660 /dev/kvmfr0 if you’re getting permission errors for /dev/kvmfr0 from QEMU.

Install the Looking Glass Host to the Windows VM, and run it.

Be sure to plug in a display port into the passed through GPU, otherwise Looking Glass will not work. It could be a real HDMI or DisplayPort plug (recommended), or a dummy plug.

To stop the annoyance of getting asked to allow the microphone and hide the microphone icon, edit: ~/.config/looking-glass/client.ini:

[audio]
micDefault=allow
micShowIndicator=no

Ensure the display scaling in KDE Plasma is set to intervals of 25%, otherwise Looking Glass will look blurry (such as on 115% scaling instead of 125% scaling).

5. Optimizing VM performance and hiding the VM from some software

Looking Glass has its own recommendations, follow those in addition to what my guide recommends below.

For the XML changes below, here is the topology of the 9800X3D CPU used (relevant for the CPU pinning):
lstopo

Show the hardware topology to understand what your CPU’s pinning would look like, but make sure to press f if there appears to be missing CPU cores:
lstopo

Set or change the following in your VM XML:

  # Put under </currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
    <allocation mode="immediate"/>
  </memoryBacking>

  # Put inside <clock>; gets past RDTSC exit checks by faking a 0.6GHz CPU frequency.
  <timer name="tsc" frequency="600000000"/>

  # Change the cores to the amount allocated to the VM; 12 cores would be cores="6".
  # Remove "svm" if using an Intel CPU, otherwise remove "vmx" if using an AMD CPU.
  <cpu mode="host-passthrough" check="none" migratable="off">
    <topology sockets="1" dies="1" clusters="1" cores="7" threads="2"/>
    <cache mode="passthrough"/>
    <feature policy="require" name="topoext"/>
    <feature policy="require" name="invtsc"/>
    <feature policy="require" name="tsc-deadline"/>
    <feature policy="disable" name="svm"/>
    <feature policy="disable" name="vmx"/>
  </cpu>

  # Put under </vcpu>
  # Note that setting a vcpu scheduler can cause your PC to lockup
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='9'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='10'/>
    <vcpupin vcpu='6' cpuset='3'/>
    <vcpupin vcpu='7' cpuset='11'/>
    <vcpupin vcpu='8' cpuset='4'/>
    <vcpupin vcpu='9' cpuset='12'/>
    <vcpupin vcpu='10' cpuset='5'/>
    <vcpupin vcpu='11' cpuset='13'/>
    <vcpupin vcpu='12' cpuset='6'/>
    <vcpupin vcpu='13' cpuset='14'/>
    <vcpusched vcpus='0-13' scheduler='rr' priority='1'/>
    <iothreadsched iothreads='1' scheduler='fifo' priority='98'/>
  </cputune>

  # Put inside <features>; if you have issues, take out each line inside hyperv one by one to test (except vendor_id)
  # If using an Intel CPU: change the vendor_id to "GenuineIntel"
    <pmu state="off"/>
    <kvm>
      <hidden state="on"/>
      <hint-dedicated state="on"/>
      <poll-control state="on"/>
      <pv-ipi state="off"/>
      <dirty-ring state="on" size="4096"/>
    </kvm>
    <ioapic driver='kvm'/>
    <hyperv mode="custom">
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="4096"/>
      <vpindex state="on"/>
      <runtime state="on"/>
      <synic state="on"/>
      <stimer state="on">
        <direct state="on"/>
      </stimer>
      <reset state="on"/>
      <vendor_id state="on" value="AuthenticAMD"/>
      <frequencies state="on"/>
      <reenlightenment state="on"/>
      <tlbflush state="on">
        <direct state="on"/>
        <extended state="on"/>
      </tlbflush>
      <ipi state="on"/>
      <evmcs state="on"/>
      <emsr_bitmap state="on"/>
      <xmm_input state="on"/>
    </hyperv>

  # Put inside <os>
  <smbios mode="host"/>

  # Put inside <devices>
  <memballoon model="none"/>

  # Put inside <qemu:commandline>
  <qemu:arg value="-overcommit"/>
  <qemu:arg value="cpu-pm=on"/>

  # More optimal settings for virtio on NVMe drives
  <disk type="file" device="disk">
    <driver name="qemu" type="raw" cache="none" io="native" discard="unmap" iothread="1" queues="8"/>
    <source file="/mnt/nvme/win11.img"/>
    <target dev="vda" bus="virtio"/>
  </disk>

  # Under <interface type="network">, ensure the model type is "virtio"

Create the automatic hooks directory:
sudo mkdir -p /etc/libvirt/hooks

Sudo edit: /etc/libvirt/hooks/qemu; change the vm_running and vm_not_running core numbers to what’s applicable to your CPU.

#!/bin/sh

command=$2
# It's based on Logical (L#) cores, not Physical (P#) cores.
vm_running="14-15"
vm_not_running="0-15"

if [ "$command" = "started" ]; then
    systemctl set-property --runtime -- system.slice AllowedCPUs=${vm_running}
    systemctl set-property --runtime -- user.slice AllowedCPUs=${vm_running}
    systemctl set-property --runtime -- init.scope AllowedCPUs=${vm_running}
elif [ "$command" = "release" ]; then
    systemctl set-property --runtime -- system.slice AllowedCPUs=${vm_not_running}
    systemctl set-property --runtime -- user.slice AllowedCPUs=${vm_not_running}
    systemctl set-property --runtime -- init.scope AllowedCPUs=${vm_not_running}
fi
Go into the Windows VM and do the following:
  • Run "Edit group policy". Go to Computer Configuration → Administrative Templates → System → Device Guard → Turn On Virtualization Based Security, and set it to "Enabled". Ensure "Select Platform Security Level" is set to "Secure Boot", and the rest of the options are left as "Not Configured".

  • Run "Turn Windows features on or off". Ensure that "Guarded Host", "Hyper-V", "Virtual Machine Platform", "Windows Hypervisor Platform", "Windows Sandbox", and "Windows Subsystem for Linux" is left unchecked as these features will destroy performance.

6. Sharing files to the Windows VM without enabling shared memory (for better performance)

Edit: /etc/samba/smb.conf

[global]
# Security
client min protocol = SMB3
## SMB3_11 is also faster than previous versions.
server min protocol = SMB3
## Allow local IPs.
hosts allow = 192.168.0.0/16
## Deny all other IPs.
hosts deny = 0.0.0.0/0
restrict anonymous = 2
disable netbios = Yes
dns proxy = No
# Performance
use sendfile = Yes
## Don't use outside local IPs!
smb encrypt = No
# Other
server role = standalone server
# Disable printer support
disable spoolss = Yes
load printers = No
printcap name = /dev/null
show add printer wizard = No
printing = bsd

# 'share1' is what Windows 10 will see in its file manager.
[share1]
path = /directory/to/folder
read only = No
## If the user is not 'admin', rename the group and user.
force group = admin
force user = admin

Validate the SMB server config, it should return no errors:
testparm

Add an SMB login for your username. It’s recommended to use a different password than your real Linux password:
sudo smbpasswd -a $USER

Allow the SMB ports through the firewall:
sudo ufw allow 445; sudo ufw allow 139

Enable and start the SaMBa service:
sudo systemctl enable --now smb.service

Find the correct local IP address to connect to inside the Windows VM for the file sharing; for me the interface was "enp14s0":
ip a

Open the 'Run' program in the Windows VM, and run: \\192.168.50.179 (replace with your local IP that was shown earlier).

About

A QEMU dual-GPU setup guide that works with some anti-VM software, and is more performant than other guides

Topics

Resources

License

Stars

Watchers

Forks