Re: Proposal: Delegating Exceptions from VS-mode or VU-mode to U-mode

Anup Patel

Hi Yifei,


Both full emulated I/O devices (e.g. UART, RTC, PCIe devices, etc) and paravirtual I/O devices (e.g. VirtIO devices, XenPV devices etc) have MMIO registers which are trap-n-emulated by Hypervisors. The paravirtual I/O devices are carefully designed devices where MMIO register programming is minimum so that MMIO trap-n-emulate overhead is minimum across hypervisors.


In KVM hypervisor, the full emulated I/O devices are usually emulated in KVM user-space whereas paravirtual I/O devices and pass-through devices are usually provided from KVM kernel-space (Refer, VHost, IRQFD, VFIO, etc). The interrupt-controller and timer are the only critical full emulated I/O devices so for most architectures we have KVM in-kernel emulation of interrupt-controller and timer. In case of KVM RISC-V, the timer emulation is totally in-kernel whereas we are waiting for new interrupt-controller spec for KVM RISC-V in-kernel interrupt-controller emulation.


Over decades, the focus across architectures (x86, ARM, PowerPC, etc) and across hypervisors (KVM, Xen, HyperV, VMWare, etc) have been to maximize use of paravirtual I/O devices (i.e. VirtIO, XenPV, etc) and pass-through I/O devices for performance. Modern Guest/VMs usually have mix of pass-through I/O devices, paravirtual I/O devices and full emulated I/O devices where the full emulated I/O devices are only used for less critical things such as UART, RTC, etc. Due to this, none of the architectures have focused on accelerating user-space emulation full emulate I/O devices for KVM (Type2) hypervisors.


If the focus of this proposal is to only improve trap-n-emulate performance of less-critical full emulated I/O devices in KVM user-space then it won’t have any significant impact because:

  1. Full emulated I/O devices in KVM user-space are generally in-frequently accessed and less critical devices so overall performance improvement will be less.
  2. We will be having critical full emulated I/O devices in KVM kernel-space (e.g. interrupt-controller) so routing all Guest/VM MMIO traps to KVM user-space will not allow full emulated I/O devices in KVM kernel-space.
  3. This proposal is too focused on user-space MMIO emulation of KVM hypervisor and in my opinion won’t benefit Type1 hypervisor (e.g. Xen, VMWare ESX, HyperV, Xvisor, etc) because these hypervisors do lot of MMIO emulation in their respective kernel spaces.





From: tech-privileged@... <tech-privileged@...> On Behalf Of Yifei Jiang via
Sent: 21 September 2020 19:21
To: tech-privileged@...
Subject: Re: [RISC-V] [tech-privileged] Proposal: Delegating Exceptions from VS-mode or VU-mode to U-mode


Hi Anup,


Sorry for the confusing description of the background in this proposal. To be clear, we here divide the implementation of trap-and-emulated I/O devices into full emulated I/O devices and paravirtual I/O devices. This proposal only focuses on improving the performance of full emulated I/O devices implemented in userspace, such as UART. Besides, we believe that other mmio emulation devices which require Guest OS to trap and exit to userspace, such as rtc, can also benefit from our proposal.


Paravirtual I/O devices, which require the GuestOS to trap and send in-kernel interrupts to interact with VirtIO backend, i.e., vHost you mentioned, are not considered in this proposal. 




Join { to automatically receive all group messages.