A number of early efforts explored virtualization on ARM. An early effort to port Xen to ARM [47] supported ARM CPUs without hardware virtualization support and used paravirtualization and sig- nificantly modified the guest OS to run on the non-virtualizable ARM architecture. This type of paravirtualization requires intimate knowledge of the guest OS source code and is difficult to main- tain. For example, this particular attempt at ARM virtualization modified more than 4500 lines of code by hand in Linux, and these changes were never supported in mainline Linux. Eventually this effort never materialized with a healthy development ecosystem, no guest OS other than a Linux 2.6.11 kernel was supported, and the effort was abandoned in favor of a new Xen implementation for ARM. In contrast to the classic full paravirtualization approach, we introduced lightweight par- avirtualization which requires no knowledge of the guest operating system, is fully automated, and can be applied to any guest operating system with access to its source code or binary executable image format. While our early implementation of KVM for ARM did not get directly merged up- stream in Linux or was commercially deployed, the code base was used as a foundation for what eventually became KVM/ARM, the Linux ARM hypervisor. This was in large part due to KVM for ARM initially being based on Linux, and being able to leverage the Linux kernel development community early in the process.
Heiser and Leslie [44] discuss the differences between hypervisors and microkernels based on their experiences building OKL4, a microkernel and hypervisor hybrid for ARM. Their work is done before hardware virtualization support was added to the ARM architecture, and is based on some form of paravirtualization. Unfortunately, not many details regarding their approach to ARM virtualization is presented, but instead their work focuses on theoretically comparing the function- ality and abstractions provided by microkernels and hypervisors. OKL4 specifically targeted the embedded market and required custom modified guest operating systems. In contrast, our work on KVM for ARM focused on providing a general purpose hypervisor with few, simple, and auto- mated modifications to guest operating systems. Additionally, our work focused on the interaction between the hypervisor software and the computer architecture, where their work focused on the relationship between hypervisors and microkernels.
tion to support virtual machines on ARM processors without hardware support for virtualization. Their hosted hypervisor design resembled the VMware workstation design [77] in which the VMM executes in a separate environment from the host kernel and only loads a small driver in the host OS being able to switch between the VMM and host execution contexts and interface with support from the host OS, for example from hardware drivers. Additionally, the authors focus on a very specific use case, the Bring Your Own Device (BYOD) model where VMs provide separate mobile computing environments on a phone. For example, they provide an analysis of the best file sys- tems to use for NAND flash memory and SD cards typically present in mobile phones, and how to support network and telephony in a virtual mobile phone environment. In contrast, our early work on KVM for ARM focuses on the general challenges in providing virtualization on ARM without architectural support for virtualization. Furthermore, KVM for ARM was integrated with the host kernel and did not have to construct a separate execution environment for the VMM.
Cells [6, 30] used operating system virtualization similar to containers to support Virtual Phones (VPs) using kernel namespaces. While Cells did not support virtual machines in the classic sense, it provided very efficient virtualization on ARM processors without hardware virtualization support. Furthermore, Cells benefited from its integration with Linux and device drivers to provide a smooth user experience with fully accelerated graphics in all VPs, something which was not supported by the commercial MVP approach. Where our work on KVM focused on providing a general purpose hypervisor that provided full system virtualization, our work on Cells focused on providing multiple virtual phone environments with full device support on real smartphones with as little performance overhead as possible.
Ding et al. [39] expanded on our work and used the same techniques as presented in this dis- sertation, but further optimized the implementation and presented some performance results. Some commercial virtual solutions for ARM CPUs without architectural support for virtualization [42, 67] are also available, primarily targeting the embedded market, but little is known about their specific designs or experiences.
Pinto et al. [65] describe how to run multiple instances of custom OSes using ARM TrustZone technology on ARM systems without architectural support for virtualization. Their solution stat- ically partitions the system, and does not offer flexible assignment of resources such as memory, which is typically provided by virtualization solutions. CPU, memory, and I/O devices are not vir-
tualized and multiplexed between VMs, but are statically partitioned at boot time and assigned to specific software instances. Their solution requires enlightened guests that are aware of a parti- tioned physical memory map, and they rely on an optional peripheral, a TrustZone address space controller, to ensure memory isolation between multiple VMs.