PCI Passthrough with Thunderbolt

Due to the hardware/software I'm using, I'll only be covering how to get Intel-based machines with an Nvidia graphics card working on Arch.

What these BIOS options do is enable Direct Memeory Access (DMA) for hardware devices to specific memory regions only, isolating it from the rest of the system. You'll need to enable this so that you can set up and allocate the resources used by virtual machines.

IOMMU stands for Input-Output Memory Management Unit and it normally controls the access of hardware resources to regions in memory automatically by mapping virtual addresses that the device can see to physical memory regions. What we need to do is set our kernel to give us control over the IOMMU so that we can isolate the hardware we plan to use for our guest VM.

sudo vim /etc/defaults/grub/
# Modify GRUB_CMDLINE_LINUX_DEFAULT by appending "intel_iommu=on iommu=pt"
sudo grub-mkconfig -o /boot/grub/grub.cfg
sudo reboot now

This will add the necessary kernel parameters to enable control of the IOMMU and also generate a new configuration file for GRUB with the updated changes.

Now that we've enabled control of the IOMMU, we need to isolate our GPU and ensure that it's using the correct kernel drivers for virtualization. We'll do this with the following:

# this displays all connected PCI devices along with their vendor-device ID pairs
lspci -knn
# Copy the device ID (format is [10de:xxxx]) for the graphics card and audio

sudo echo 'options vfio-pci ids=<gpu-id>,<audio-id>' >> /etc/modprobe.d/vfio.conf

sudo vim /etc/mkinitcpio.conf
# modify MODULES by adding 'vfio_pci vfio vfio_iommu_type1 vfio_virqfd' to the start
# modfiy HOOKS by adding 'modconf' if it's not already present

What we've done now is load the virtualization drivers (vfio-pci) and set them to be the drivers used by our target PCI devices (the GPU). We now need to regenerate our kernel image, which we can do with:

sudo mkinitcpio -P
sudo reboot now

If everything worked as expected, you should be able to run lspci -knn and see that your graphics card and audio are now using vfio-pci drivers.

You'll first need to download some packages using:

sudo pacman -Syu libvirt virt-manager ovmf qemu

Libvirt is a wrapper around QEMU for managing platform virtualization and giving us an easier API to work with. Virt-manager is a GUI that makes setting up some of the next specs a bit easier. OVMF stands for Open Virtual Machine Firmware and gives virtual machines the ability to run a UEFI (Unifed Extensible Firmware Interface). We'll need this for our VM to detect and interface with our GPU when we pass it in. Lastly, QEMU is an open-source emulator that can convert instructions between architectures and emulate hardware devices for VMs using host hardware.

Once the packages are done installing, we'll need to add the path to our OVMF firmware image to libvirt:

sudo echo 'nvram = [
	"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"
]' >> /etc/libvirt/qemu.conf

Then we'll enable libvirt and add give our user permission to use libvirt with:

sudo systemctl start libvirtd.service
sudo systemctl start virtlogd.service
sudo systemctl enable libvirtd.service
sudo systemctl enable virtlogd.service
sudo usermod -a -G libvirt $USER

Now our environment is ready for creating a VM that can interface with our GPU.

You'll want to enter virt-manager & to start a GUI for VM management. Press the button for creating a new VM and select the Windows 10 ISO you need to download from here. Then click through the install wizard until you reach the last screen and make sure to check the option for customizing your VM before you install. When the GUI with your VM configuration opens, make sure to go to the "Overview" section and select to use UEFI with OVMF from the "Firmware" dropdown and "Q35" for chipset.

Now we'll make some general changes to the VM to make sure it's functional.

• Remove the default IDE disk and create a new SCSI disk

• download the virtio drivers so the guest VM recognizes our hardware here and add it to a new SATA CD-ROM

• Add the PCI devices corresponding to the graphics card, audio, and an external mouse+keyboard

  (do not add any thunderbolt devices to the VM or the host will not be able to communicate with them and the GPU will fail)

At this point the VM is all set up and ready to be run. If when you run the VM your external monitor does not turn on with the UEFI screen then something must be wrong and you'll need to troubleshoot.

In order to now install Windows on the SCSI disk created before, you'll need to do the following:

• On the UEFI screen type "exit" to enter into the BIOS

• From the BIOS go to "boot manager" and select the CD-ROM holding the Windows 10 ISO

• Go through the Windows installation steps, but select "custom install" when given the prompt

• From this screen select "load driver", then select the CD holding the virtio drivers->vioscsi->w10->amd64

• From the previous step the SCSI disk should now be visible to the VM. Select it and continue with the Windows installation

Once Windows is installed you'll notice that in device manager your graphics card cannot be recognized. To fix this you'll need to install the necessary drivers for your graphics card. If that doesn't fix the problem then see the troubleshooting options below.

I hope setting all of this up went better for you than it did for me. If it didn't, I've included as much troubleshooting advice as I can offer below to make it easier to find. I had to scour the web for quite a while to find a lot of this so hopefully this saves you some time.