Memory Virtualization

Virtual memory is already a property of the hardware, but virtual memory is managed by the mem- ory management unit (MMU) on modern architectures. To maintain the VM equivalence property, hypervisors must properly virtualize the MMU and multiplex the MMU functionality between mul- tiple VMs. Often, this is referred to as memory virtualization in the context of running virtual machines, however ambiguous the term is.

Unmodified guest OSes written to manage virtual and physical memory on a real hardware system must also execute correctly in the functionally equivalent VM. The guest OS must also be protected from guest applications, similar to real hardware. Second, the hypervisor must be protected from the guest OS and applications to satisfy the isolation VM property. KVM for ARM uses a well-known technique from the x86 world known as shadow page tables [24] which leverages the hardware MMU to virtualize memory for the VM.

Since the VM is isolated from the underlying physical machine, and the physical memory re- source is multiplexed between multiple virtual machines, the guest’s view of physical memory is entirely separate from the actual physical memory layout. The guest view of physical memory is called the Guest Physical Address (GPA) space. The guest operating system maps from Virtual Addresses (VAs) to GPAs using its own page table structures, but these page tables are not used by the physical MMU. Instead, the hypervisor creates shadow page tables, which translate from the guest VAs to real Physical Addresses (PAs). Figure 1.1 illustrates the various address spaces and their relationship.

Guest pages are mapped using the shadow page tables on demand, when the hypervisor handles page faults from VMs. An entry is created in the shadow page table for the faulted virtual address. Shadow page table entries translate directly from VAs to PAs. The VA is translated to a GPA by walking guest page tables in software. KVM for ARM can translate any GPA into a PA by using its own data structures. Each shadow page table entry is therefore the result of combining two

KVM process Virtual Memory

Guest physical Guest virtual

Host kernel Guest memory

Machine memory

Figure 1.1: Address Space Mappings

translation stages into a single stage, and translates directly from VAs to PAs and is then walked by the hardware MMU when running the VM.

Two special entries are always mapped by the hypervisor in the shadow page tables and re- serve part of the guest’s VA space, the hypervisor exception page and the shared page. When the hypervisor enters the VM, the entire virtual address space is completely changed from the hyper- visor and host OS’s view to the guest OS’s view. Both the virtual address space and registers are changed when moving from host to VM execution context, and these two contexts are essentially in two different worlds. Therefore, we refer to the transition between the hypervisor and the VM as a world switch. However, when moving from one page table to the other, the ARM architecture requires that this is done from a page which is shared between the two page tables. KVM for ARM therefore also has to support a shared page, which is a page mapped at the same VA in both the host and VM’s virtual address space. The shared page contains two sequences of code, one to move from the hypervisor to the VM, and one to move from the VM to the hypervisor, and also includes a small amount of data, for example to store the host’s page table base address, which is used to return to the host OS from the VM. If the VM tries to write to any of these two pages it will generate a permission fault to the hypervisor and the hypervisor moves the location of the special page and simply chooses a new address in the guest’s VA space for the special page.

The hypervisor must keep the shadow page tables up to date with any changes to either of the two translation stages combined into the shadow page table entries. GPA to PA translations are maintained by the hypervisor itself and are therefore trivial to propagate into the shadow page table entries, but changes to guest page tables are more difficult to track. For example, if the guest kernel uses copy-on-write, it will modify its own page table mappings on the first write to a COW memory region. Fortunately, such a change in address mappings requires the guest to also invalidate any TLB entries potentially caching the old mapping, and TLB invalidation is a privileged operation that traps to the hypervisor. The hypervisor therefore detects that the guest has modified all or part of its address space and invalidates corresponding shadow page table entries as needed.

Since the guest OS must run in the non-privileged user mode to protect the hypervisor and the rest of the system, and since ARMv7 allows a page to have different permissions for privileged and non-privileged accesses, KVM for ARM must virtualize the access permissions. For example, if the guest page table entry for a particular page allows access in privileged mode but no access in user mode, and the VM is running in virtual kernel mode (but actually running in user mode), then the shadow page table permissions must allow non-privileged access. When the VM changes its emulated mode to user mode, the shadow page tables must not allow any accesses. The hypervisor therefore has to update access permissions on the shadow page table when the virtual CPU changes its virtual mode between privileged and non-privileged execution and vice versa.

ARMv7 and earlier architecture revisions define domains, a mechanism to overwrite page access controls. Domains configure 1 MB virtual address regions to either be completely non-accessible, follow access permissions in the page tables, or always allow full read/write access regardless of the CPU mode. If the guest configures a region that covers one of the two special pages, the shared page or the exception page, to allow all accesses to the region, then the hypervisor must reconfigure the domain of that region in the shadow page tables to instead use the access permissions in the page table, and configure the special pages to only allow privileged access and configure the remaining guest pages for non-privileged access.