Host Configuration

Host Configuration#

Prerequisites#

Before starting this guide, ensure you have:

Getting Started with Virtualization - Understanding of MxGPU concepts

This section provides step-by-step instructions for configuring SR-IOV to enable device assignment to QEMU/KVM guests. Proper configuration is essential for ensuring that the GPU device can be passed through to VMs effectively.

System BIOS Setting#

To enable virtualization extensions, you need to set up System BIOS. For example (depending on BIOS version and vendor), sample System BIOS settings will look like this:

  • SR-IOV Support: Enable this option in the Advanced → PCI Subsystem Settings page.

  • Above 4G Decoding: Enable this option in the Advanced → PCI Subsystem Settings page.

  • PCIe ARI Support: Enable this option in the Advanced → PCI Subsystem Settings page.

  • IOMMU: Enable this option in the Advanced → NB Configuration page.

  • ACS Enabled: Enable this option in the Advanced → NB Configuration page.

Note: AER must be enabled for ACS enablement to work. PCI AER Support can be enabled in the Advanced → ACPI Settings page.

GRUB File Update#

After configuring the BIOS settings, you need to modify the GRUB configuration to apply the necessary changes.

To assign a device to a QEMU/KVM guest, the device needs to be managed by a VFIO (Virtual Function I/O) kernel driver. However, by default, the host device binds to its native driver, which is not a VFIO driver. Therefore, the device must be unbound from its native driver (blacklist amdgpu driver on OS booting up) before it can be passed to the libvirt to assign it to the guest.

Additionally, to enable PCI SRIOV functionality, you need to enable virtualization extensions, IOMMU, etc. All these configurations can be set in /etc/default/grub and then applied to the OS boot grub configuration.

Use the following commands to update the GRUB settings:

  1. Edit GRUB Configuration File:

Use a text editor to modify the /etc/default/grub file (Following example uses “nano” text editor). Open the terminal and run the following command:

# sudo nano /etc/default/grub

  1. Modify the GRUB_CMDLINE_LINUX Line:

Look for the line that begins with GRUB_CMDLINE_LINUX. It should look like this initially:

GRUB_CMDLINE_LINUX=""

Modify it to include following parameters:

GRUB_CMDLINE_LINUX="modprobe.blacklist=amdgpu iommu=on amd_iommu=on"

If there are already parameters in the quotes, append your new parameters separated by spaces.

  1. Save the changes by pressing CTRL + O, press Enter, then exit with CTRL + X.

  2. After modifying the configuration file, you need to update the GRUB settings by running the following command:

Ubuntu:

# sudo update-grub

RHEL:

# sudo grub2-mkconfig -o /boot/grub2/grub.cfg

  1. Reboot Your System:

For the changes to take effect, reboot your system using the following command:

# sudo reboot

  1. Verifying changes:

After the system reboots, confirm that the GRUB parameters were applied successfully by running:

# cat /proc/cmdline

When you run the command above, you should see a line that includes:

modprobe.blacklist=amdgpu iommu=on amd_iommu=on

This indicates that your changes have been applied correctly.

Note: In case host machine is running Intel CPU, replace amd_iommu with intel_iommu.

Installing Libvirt, KVM, QEMU Packages on Host#

Follow the steps below to install Libvirt, KVM, QEMU Packages on Ubuntu Host:

  1. Install the required packages

Ubuntu:

# sudo apt update
# sudo apt install qemu-kvm virtinst libvirt-daemon virt-manager -y

RHEL:

# sudo dnf install @virt Management -y
# sudo systemctl start libvirtd  
# sudo systemctl enable libvirtd

  1. Configure non-root user to access virt-manager and add user account to the libvirt group:

Ubuntu:

# sudo groupadd --system libvirt
# sudo usermod -a -G libvirt $(whoami)
# newgrp libvirt

RHEL:

# sudo usermod -a -G libvirt $(whoami)

Log out of your current user session. Then, reopen the terminal or log back into your user account. This refreshes your session, ensuring the group changes take effect for access to libvirt resources.

  1. Edit libvirtd configuration file:

# sudo nano /etc/libvirt/libvirtd.conf

  1. Set the UNIX domain socket group ownership to libvirt, (around line 81):

unix_sock_group = "libvirt"

  1. Set the UNIX socket permissions for the R/W socket (around line 104):

unix_sock_rw_perms = "0770"

  1. Restart libvirt daemon after making the change.

# sudo systemctl restart libvirtd.service

Host Driver Setup#

To enable the AMD GPUs to operate in SR-IOV mode, the MxGPU PF driver is essential. This guide presents two effective approaches for installing the PF driver: via a pre-packaged .deb/.rpm file or by building from source code.

Using the package offers a convenient and streamlined installation process, making it accessible even for users with limited technical experience. This method allows for easy dependency management but may not always include the most up-to-date features available in the source code.

On the other hand, building the PF driver from source provides users with the flexibility to customize their driver installation and access the latest enhancements. This approach, however, requires more technical expertise and manual handling of dependencies, which might present a steeper learning curve for some users.

Both methods will effectively prepare your system for GPU virtualization, and from version 8.3.0.K onwards, both will automatically install AMD SMI, providing robust management and monitoring capabilities for your GPUs. The choice between the two approaches depends on your specific needs and familiarity with the installation process. The following sections will detail each approach to guide you through the setup.

Installing the PF Driver via Package#

On Ubuntu systems, PF driver can be installed using the latest .deb package.

On RHEL systems, PF driver can be installed using the latest .rpm package.

Here are the steps:

  1. Install dependencies:

Ubuntu:

# sudo apt update
# sudo apt install build-essential dkms autoconf automake

RHEL:

# sudo dnf install @development-tools kernel-devel dkms autoconf automake

  1. To install the package, use the following command:

Ubuntu:

# sudo dpkg -i gim_driver_package.deb

RHEL:

# sudo rpm -ivh gim_driver_package.rpm

  1. Once the PF driver is installed, reboot the system and load the driver after reboot by using the following command:

# sudo modprobe gim

The PF driver is loaded with one VF by default, which means that if we use a platform with eight GPUs, each of the GPUs can support one VF. This results in a total of eight VFs.

  1. To ensure that the PF driver has loaded correctly, check if the GPU VF devices are visible by running:

MI210X:

# lspci -d 1002:7410

MI300X:

# lspci -d 1002:74b5

MI325X:

# lspci -d 1002:74b9

MI350X:

# lspci -d 1002:75b0

Radeon Pro V710:

# lspci -d 1002:7461

When the PF driver is loaded the command will output an entry for each VF.

  1. Way to check if driver loaded successfully:

# sudo dmesg | grep GIM.*Running

Building the PF Driver from Source#

Alternatively, you can build the PF driver from the source code if you prefer to have the latest features or customize the installation. Here are the steps:

  1. Install dependencies:

Ubuntu:

# sudo apt update
# sudo apt install build-essential autoconf

RHEL:

# sudo dnf groupinstall "Development Tools"  
# sudo dnf install kernel-devel autoconf

  1. Obtain the Source Code: Clone or download the source code for the PF driver.

  2. Navigate to the Source Directory: Change to the directory containing the PF driver source code.

  3. Build the PF Driver:

To build and install the driver from the source, run the following commands:

# make clean  
# make all -j
# sudo make install

  1. Insert the Driver: Once the PF driver is installed, load it using the following command

# sudo modprobe gim

To verify that the PF driver is loaded properly, use the same verification commands mentioned in the previous section.

Note: Newer Ubuntu kernel versions require gcc and g++ version 12 installed. If needed install gcc-12, g+±12 and update the alternatives system to point to the correct version:

# sudo apt install gcc-12 g++-12 
# sudo update-alternatives --config gcc   
# sudo update-alternatives --config g++

Next Steps#

After completing host configuration, you have two paths:

Simple Setup#

Virtual Machine Setup - Configure your VMs with default VF settings

Advanced Setup (Custom configurations)#

Choose one or both advanced guides before VM setup: