Enabling USB Hotplug support with USB Hubs in Proxmox

Currently, Proxmox already supports USB redirection for individual devices, identified either by vendor / device ID or USB bus and port. There is even hotplug support, so I could disconnect a device, reconnect it later, and it would be assigned directly to the VM. This implementation of redirection is already sufficient for most use cases, but I needed something more flexible. In my case, I have a VM set up to act pretty much like a physical desktop PC: I am passing through a GPU with external monitor attached for video and graphics acceleration, together with the aforementioned USB redirection to attach a mouse, keyboard and any other USB devices I might want to use. Among them are also various microcontrollers, with device IDs that I do not know in advance. So, my plan was to take a USB hub for all of the devices, which I attach to a USB port on my host to redirect that port to the VM.

As it turns out, USB redirection simply does not work that way: By specifying a port ID in qemu, I can only pass through a single device attached to that port directly. I asked google for a solution to the problem, and it seems that the common solution simply consists of a PCIe USB controller, which can be attached via PCI passthrough. This is definitely a valid solution (and probably a bit less fiddly than mine), but I really did not want to spend money and occupy a PCIe slot while there are enough perfectly good USB ports at the back of the motherboard. After a bit more googling, I found usb-libvirt-hotplug: Someone else had a similar problem before, and build a solution using udev rules and the libvirt API. While the script does not work for my use case - it is using libvirt and does not treat hubs the way I want - it inspired me to write my own based on the same ideas.

Talking to QEMU #

The official way of adding a device at runtime in proxmox is the use of the device_add command in the qemu monitor, which can be accessed either from the web UI or the qm monitor command. With this command, we can add devices by their vendor / device ID or physical location, just like we can do in the config file or qemu command line. Since I didn't want to pipe the commands into the terminal, I went through the code of the qm script to find out how it interacts with qemu. As it turns out, the script talks to the VM over a so-called QMP Socket, which offers a script-friendly command interface based on JSON. By looking at the QEMU QMP reference, we can see that the device_add command from the monitor command line directly translates to a QMP command. In addition, there is also a device_del command, which allows us to detach a USB device.

Understanding and enumerating USB Hubs #

Now that we have a way to attach and detach USB devices from the VM on demand, we need to know the actual devices that we want to attach. In plain english, this is easy enough to express:

"We have a designated physical USB port at the back of our server, and want to pass through every USB device that is somehow attached to that port (including devices on hubs on that port)"

To translate that requirement into actual USB device identifiers, we have to understand how the USB port and device numbering on Linux works.

Here and Here and Here are some resource that I used for reference. If you don't understand something from my descriptions, they may be worth reading.

QEMU expects us to specify a physical USB port by a bus ID (hostbus parameter) and a port sequence (hostport parameter). Together, both specify a physical USB port. Most Linux tools specify a Port with a string of the form ${hostbus}-${hostport} - for example, physical port 4 on the root hub of Bus 3 would be listed as 3-4. The ID of a USB hub is composed recursively of two parts: First, the port sequence that identifies the port where the hub itself is plugged into, then the port itself separated by a dot. So, the string 3-4.1 would identify a device on Port 1 of a hub, which itself is plugged in on Port 4 of the root hub of bus 3 (which is likely a port directly on the motherboard). You can see the port numbers for yourself by typing lsusb -t, albeit in a slightly different format. Here is the output of my system:

kristian@ad.krisnet.de@kellerserver:~$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 4: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 22, If 0, Class=Hub, Driver=hub/5p, 480M
            |__ Port 1: Dev 23, If 0, Class=Human Interface Device, Driver=usbfs, 480M
            |__ Port 4: Dev 40, If 0, Class=Mass Storage, Driver=usbfs, 480M
            |__ Port 2: Dev 25, If 1, Class=Human Interface Device, Driver=usbfs, 12M
            |__ Port 2: Dev 25, If 0, Class=Human Interface Device, Driver=usbfs, 12M
        |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbfs, 1.5M
        |__ Port 4: Dev 4, If 1, Class=Human Interface Device, Driver=usbfs, 1.5M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M

I also made a small diagram, with the USB hub devices removed:

In my case, Bus 03, Port 4 is the port on the mainboard that I would like to pass through to the VM. We want to attach devices from this port at two different places in our script: First, we want to listen for device attachment / detachment event on the system and add / remove the device from the VM when it is on the correct port. With the aforementioned port numbering scheme, this is actually delightfully simple: We can simply check whether the identifier string starts with (or matches) 3-4 - that's it. In addition, we need to enumerate all devices on the port when the VM starts up. This sounds a little trickier, but it is still relatively achievable thanks to sysfs: For every port on the system that has a device attached, there is a corresponding directory nested somewhere under /sys/devices/. To organize all USB ports in one place, there is a folder at /sys/bus/usb/devices which contains symlinks to the directories for all active ports on the system:

root@kellerserver:/sys/bus/usb/devices# ls
1-0:1.0  2-0:1.0  3-0:1.0  3-4  3-4.1  3-4:1.0  3-4.1.1  3-4.1:1.0  3-4.1.1:1.0  3-4.1.2  3-4.1.2:1.0  3-4.1.2:1.1  3-4.1.4  3-4.1.4:1.0  3-4.4  3-4.4:1.0  3-4.4:1.1  4-0:1.0  usb1  usb2  usb3  usb4

Let's cd to the directory for 3-4 to see how we can use the sysfs for enumeration:

root@kellerserver:/sys/bus/usb/devices# file 3-4
3-4: symbolic link to ../../../devices/pci0000:00/0000:00:07.1/0000:08:00.3/usb3/3-4
root@kellerserver:/sys/bus/usb/devices# cd ../../../devices/pci0000:00/0000:00:07.1/0000:08:00.3/usb3/3-4
root@kellerserver:/sys/devices/pci0000:00/0000:00:07.1/0000:08:00.3/usb3/3-4# ls
3-4.1    authorized         bConfigurationValue  bDeviceSubClass  bMaxPower           busnum         dev      driver         idProduct    maxchild           power      remove    subsystem  urbnum
3-4:1.0  avoid_reset_quirk  bDeviceClass         bmAttributes     bNumConfigurations  configuration  devnum   ep_00          idVendor     physical_location  quirks     rx_lanes  tx_lanes   version
3-4.4    bcdDevice          bDeviceProtocol      bMaxPacketSize0  bNumInterfaces      descriptors    devpath  firmware_node  ltm_capable  port               removable  speed     uevent

As we can see by the path the symlink points to, all directories in sysfs are organized to resemble a device tree: Port 3-4 is a port on USB Bus 3, which itself is a PCI device. In the directory itself, there are subdirectories 3-4.1 and 3-4.4 for the child ports. There is also another folder called 3-4:1.0, which can be translated as Device on Bus 3, Port 4 in Configuration 1, Interface 0. This is probably some sort of interface to control the hub itself, but we don't care about it here. One last thing we'd like to know for enumeration is whether the device on a given port is a Hub or a regular USB device that we can redirect. To get an answer to this question, we can look at the bDeviceClass file, which contains the USB device class; Hubs have a designated device class of 09.

Creating a script and udev hook #

Now that we have understood USB numbering and sysfs, we are armed with everything we need to write a script to solve our problem. I decided to write the script in Python, simply because I am very familiar with it and think that it's well readable. As it is to be expected, there is already a package in python for interacting with QMP, called qemu.qmp, which can save us some boring code for interacting with the QMP socket.

The full script is attached in the blog below, and should be mostly self-explanatory with the comments inside:

#!/usr/bin/python3
# Script to implement USB hotplugging in Proxmox
#
# This script may be called from two places:
# - To enable the hotplug functionality, this script is called from a udev rule
#   In this case, the script is executed without arguments, and all parameters are passed
#   by udev as environment variables. In this case, we can get the device from the env,
#   and we will look through the config files to see which VM is interested in it.
#
# - To enable all devices at VM start, this script is also intended to be called as a proxmox
#   hook script. In this case, it will parse the VM config file, and attach all devices
#   that match the device pattern within. In the case of a hookscript, this script is called with two
#   command line parameters: first the VMID, then the phase the VM is in. See `/usr/share/pve-docs/examples/guest-example-hookscript.pl`
#   for an overview of the phases.
#

import asyncio
from qemu.qmp import QMPClient
import os
import logging
import sys
import itertools
import re

logger = logging.getLogger('qmp_device_add')
logger.setLevel(logging.DEBUG)
fh = logging.FileHandler('/var/log/qmp_device_add.log')
fh.setLevel(logging.INFO)
logger.addHandler(fh)


# prepare a dictionary of VMs to their corresponding device patterns
usb_config_files = [ f.path for f in os.scandir('/etc/pve/qemu-server') if f.is_file() and re.match(r'^\d+.usb.conf$', os.path.basename(f.path))]
usb_config = {}

for usb_config_file in usb_config_files:
    vmid = os.path.basename(usb_config_file).split('.')[0]
    with open(usb_config_file, 'r') as usb_config_file_handle:
        usb_config[vmid] = usb_config_file_handle.read().splitlines()

def get_vm_for_port(devid):
    candidate_vms = []
    for vmid in usb_config.keys():
        if any(map((lambda devspec: devid.startswith(devspec)), usb_config[vmid])):
            candidate_vms.append(vmid)
    if len(candidate_vms) > 1:
        raise Exception(f'Multiple VMs {candidate_vms} could take USB device {devid} - will not attach it anywhere. Fix the config')
    if len(candidate_vms) == 0:
        return None
    return candidate_vms[0]

def get_devices(devdir):
    if not os.path.exists(os.path.join(devdir, 'bDeviceClass')):
        return []
    with open(os.path.join(devdir, 'bDeviceClass'), 'r') as bDeviceClass_file:
        bDeviceClass = bDeviceClass_file.read().strip()
    if bDeviceClass == '09':    # We are a USB hub, don't include us in the hierarchy but enumerate
        subdevice_folders = [ f.path for f in os.scandir(devdir) if 
            # regex essentially matches USB port names of the form 1-2.3.4
            # - we don't match anything with a :, so no matching of device configurations and interfaces
            # - we don't allow a 0 for the first digit after /, since this can correspond to a root hub USB device
            f.is_dir() and re.match(r'^\d-[1-9](\.\d+)*$',os.path.basename(devdir))
        ]
        return list(itertools.chain(*[get_devices(subdevice_folder) for subdevice_folder in subdevice_folders]))
    else:                       # we are something else, return us as a device
        return [devdir]
       

# catches all exceptions and logs them (exception helper)
# since async seems to swallow exceptions
async def exhelper():
    try:
        await main()
    except:
        logger.error('An exception occurred in async main:', exc_info=True)
        exit(-1)

async def main():
    if len(sys.argv) > 1:
        vmid = sys.argv[1]
        
        # check that we are in the correct phase, otherwise we can't attach usb devices
        if sys.argv[2] != 'post-start':
            logger.info(f'Phase {sys.argv[2]} != "post-start", script will not run')
            return

        device_patterns = usb_config[sys.argv[1]]
        devices = set()
        for device_pattern in device_patterns:
            devices.update(get_devices(os.path.join('/sys/bus/usb/devices', device_pattern)))
        logger.info(f'List of devices: {devices}')
        
        for device in devices:
            device_usbtree_path = os.path.basename(device)
            hostbus, hostport = device_usbtree_path.split('-')
            await add_device(vmid,hostbus, hostport)

    else:
        # pull in udev parameters
        devpath = os.environ['DEVPATH']
        action = os.environ['ACTION']

        
        logger.debug(os.environ)

        device_usbtree_path = os.path.basename(devpath)
        hostbus, hostport = device_usbtree_path.split('-')

        logger.info(f'udev action: {action}')
        logger.info(f'device USB tree path: {device_usbtree_path}')

        vmid = get_vm_for_port(device_usbtree_path)
        if vmid == None:
            logger.info('Found no device for vm, leaving it for host')
            return
        logger.info(f'VM to attach to: {vmid}')

        if re.match(r'^\d-[1-9](\.\d+)*$', device_usbtree_path): 
            if action == 'add':
                await add_device(vmid, hostbus, hostport)
            elif action == 'remove':
                await remove_device(vmid, hostbus, hostport)
            else:
                logger.error(f'Action "{action}" unknown')
        else:
            logger.info('Not full USB device, ignoring it')

async def add_device(vmid, hostbus, hostport):
    qmp = QMPClient('vm')
    await qmp.connect(f'/var/run/qemu-server/{vmid}.qmp')

    result = await qmp.execute('device_add', arguments={
        'driver' : 'usb-host',
        'hostbus': hostbus,
        'hostport': hostport,
        'id': f'usbhost{hostbus}-{hostport}'
        })
    logger.info(f'QEMU Result: {result}')
 
    await qmp.disconnect()
async def remove_device(vmid, hostbus, hostport):
    qmp = QMPClient('vm')
    await qmp.connect(f'/var/run/qemu-server/{vmid}.qmp')
 
    
    result = await qmp.execute('device_del', arguments={
        'id': f'usbhost{hostbus}-{hostport}'
    })
    logger.info(f'QEMU result: {result}')
    await qmp.disconnect()

asyncio.run(exhelper())

Script Installation #

Proxmox Hookscript #

To attach all devices as the VM boots up, we have to register the script as a Proxmox hookscript. To allow Proxmox to find the script, it should be placed in the snippets subdirectory of any Proxmox-integrated storage. Usually, Proxmox mounts its storage under /mnt/pve/${STORAGENAME}, so we have to place the script at /mnt/pve/${STORAGENAME}/snippets/qmp_device_add.py. Also, remember to chmod +x the script file.

Next, the hookscript has to be activated for every VM that requires it. In the config file under /etc/pve/qemu-server/${VMID}.conf, add the following line:

hookscript: ${STORAGENAME}:snippets/qm_device_add.py

Note: ${STORAGENAME} is just a placeholder, you have to substitute it with the actual name of your storage.

udev rule #

For hotplugging, a udev rule is needed to call the script whenever a USB device is attached / detached. Create a new file at /etc/udev/rules.d/90-vm-usb-hotplug.rule, and enter the following content:

SUBSYSTEM=="usb",RUN+="/mnt/pve/${STORAGENAME}/snippets/qmp_device_add.py"

Note: The udev rule is a simple catch-all to listen for all USB device changes. Filtering is done by the script.

VM config file #

To configure the ports that are passed through to any VM, the script expects a config file under /etc/pve/qemu-server/${VMID}.usb.conf. The file should contain a list of physical USB ports on the host, separated by newlines. In our example, where we only pass through port 3-4, the file contents will be:

3-4

When a VM is configured for passthrough with the script, you should remove all static USB passthrough configurations in the VM config file to avoid conflicts.

Testing #

To verify that the script works, we can use the monitor function in Proxmox:

1. With the VM turned off, attach a USB device to the port you specified.
2. Start the VM in the web UI. If the start job fails, it may be related to an error in the hookscript - take a look at the task log, it will contain the python logs
3. Open the "Monitor" tab on the left and enter info usb. You should see an output like this:

# info usb
  Device 0.1, Port 1, Speed 480 Mb/s, Product QEMU USB Tablet, ID: tablet
  Device 1.1, Port 1, Speed 1.5 Mb/s, Product HDMI KVM, ID: usbhost3-4.4
  Device 1.1, Port 2, Speed 480 Mb/s, Product USB HID, ID: usbhost3-4.1.1
  Device 1.2, Port 3, Speed 480 Mb/s, Product Cruzer Blade, ID: usbhost3-4.1.4
  Device 1.1, Port 4, Speed 12 Mb/s, Product 2.4G Wireless Receiver, ID: usbhost3-4.1.2

In my case, I have multiple devices attached to hubs on port 3-4. If you don't see any devices with IDs starting with usbhost, then something is not working with the hookscript

4. Try to detach a device, then enter info usb again. You should see the device disappear. Otherwise, something is wrong with the udev rule or removal code
5. Try to reattach the device to the port, then enter info usb again. The device should pop up again and be usable from the VM.

If this small test routine worked out for you, then you should be all set! Otherwise, you may want to observe the logs written by the script at /var/log/qmp_device_add.py.

Comments

If you have any questions or comments, please feel free to reach out to me by sending an email to blog(at)krisnet.de.

• Next: Investigation: Why does an Arduino clone not work as a USB-Serial
• Previous: Quick Tip: Running GitLab (Omnibus Package) with a nonstandard SSH Port