Sometimes, a paper is more than just a paper. Around late 2005 or early 2006 I started working on direct device assignment, a useful approach for I/O virtualization where you give a virtual machine direct access to an I/O device so that it can read and write the physical machine’s memory without hypervisor involvement. The main reason to use direct device assignment is performance: since you bypass the hypervisor on the I/O path, it stands to reason that for I/O intensive workloads — the hardest workloads to virtualize — direct device assignment would provide bare-metal performance. Right?
Wrong. Since 2006, we’ve seen again and again that even with direct device assignment virtual machines performance falls far short of bare-metal performance for the same workload. Sometime in 2009, we realized that after you solve all other problems, one particular thorny issue remains: interrupts. The interrupt delivery and completion architectural mechanisms in contemporary x86 machines, even with the latest virtualization support, were not designed for delivering interrupts directly to untrusted virtual machines. Instead, every hypervisor programs the interrupt controllers to deliver all interrupts directly to the hypervisor, which then injects the relevant interrupts to each virtual machine. For interrupt-intensive virtualized workloads, these exits to the hypervisor can lead to a massive drop in performance.
Although it is possible to work around the interrupt issue by modifying the virtual machine’s device drivers to use polling, as we did in the Turtles paper and in the Tamarin paper that will be presented in FAST ’12, it always annoyed me that the promise of bare-metal performance for virtual machines remained unreachable for unmodified virtual machines. That is, until now.
Through the amazing work of a combined IBM and Technion team, we came up with an approach — called ELI, for Exitless Interrupts — that allows direct and secure handling of interrupts directly in virtual machines — without any changes to the underlying hardware. With ELI, direct device assignment can finally do what it was always meant to do: provide virtual machines with bare-metal performance. It is nice to look back at the research over the last five or six years that lead us to this point; it will be even nicer, when we present this work in ASPLOS in London in a couple of months, to ponder what other breakthroughs the next few years hold.
“ELI: Bare-Metal Performance for I/O Virtualization”, by Abel Gordon, Nadav Amit, Nadav Har’El, Muli Ben-Yehuda, Alex Landau, Assaf Schuster and Dan Tsafrir. In ASPLOS ’12: Seventeenth International Conference on Architectural Support for Programming Languages and Operating Systems.
Direct device assignment enhances the performance of guest virtual machines by allowing them to communicate with I/O devices without host involvement. But even with device assignment, guests are still unable to approach bare-metal performance, because the host intercepts all interrupts, including those interrupts generated by assigned devices to signal to guests the completion of their I/O requests. The host involvement induces multiple unwarranted guest/host context switches, which significantly hamper the performance of I/O intensive workloads. To solve this problem, we present ELI (ExitLess Interrupts), a software-only approach for handling interrupts within guest virtual machines directly and securely. By removing the host from the interrupt handling path, ELI manages to improve the throughput and latency of unmodified, untrusted guests by 1.3x — 1.6x, allowing them to reach 97%–100% of bare-metal performance even for the most demanding I/O-intensive workloads.
Muli Ben-Yehuda's journal 