Timer Virtualization

Systems software needs some mechanism to keep time. Both telling the passing of time and pro- gramming timers to generate interrupts after some time has passed are essential in providing OS services to applications. For example, a graphical user interface (GUI) typically programs a timer to blink the cursor when editing text, and OS kernels typically check time spent at various check- points to schedule tasks according to a scheduling policy. When running VMs, the hypervisor has similar requirements for both time keeping mechanisms, and the guest OS running inside the VM also need to keep time. However, the definition of time can depend on the underlying hardware and can range from telling wall-clock time to CPU cycles or using other constant timers which may stop counting or count at a slower rate depending on the power state of the hardware system. Further- more, when running VMs, the VM may not be running continuously because the hypervisor may run other tasks instead of the VM, without the VM being notified, and as a consequence, wall clock time may not relate virtual processing time the same way it relates to native processing time.

The ARM Generic Timer Architecture, also known as the architected timers, includes support for timer virtualization. The goal is to provide VMs with access to their own timekeeping hardware, so that VMs can tell time and program timers without trapping to the hypervisor, and to allow VMs to tell the difference between some concept of virtual time and physical time [68]. To meet that goal, ARM provides three separate timers; a hypervisor timer, a physical timer, and a virtual timer. The idea is that the hypervisor timer represents the actual passing of time and is owned exclusively by the hypervisor, where both the physical and virtual timers are used by a VM. The physical timer also represents actual passing of time, where the virtual timer can be adjusted by the hypervisor to provide a measure of virtual time, for example based on when the VM has actually been allowed to run.

More specifically, each timer on ARM comprises both a counter and a timer. The counter is simply a device that counts up, and the timer is a small logical circuit, which can be programmed to generate an interrupt when the counter reaches a programmable value. The hypervisor timer is only accessible from Hyp mode, and any attempt to program it from kernel or user mode generates an exception to kernel mode. The hypervisor can configure if accesses to the physical timer trap when executed in the VM or if the physical timer is accessible by the VM. The virtual timer is

always accessible in the VM. The hypervisor can program a special virtual counter offset register, only accessible in Hyp mode, which changes the counter value of the virtual timer relative to the physical timer. A hypervisor can use this, for example, to subtract time from the virtual counter when the VM has not been running, and the VM can use this information to adjust its scheduling policy or report stolen time, time when the virtual CPU wasn’t actually running, to the user.

While the virtualization support for timers allows a VM to directly program a timer without trapping, the hypervisor is still involved in signaling interrupts to a VM when a timer fires. When timers fire, they generate hardware interrupts. Such hardware interrupts are no different from other hardware interrupts on the system, regardless of the timer that signals them, and therefore follow the configuration of the system to either be delivered directly to kernel mode, when executing a native OS, or trap to the hypervisor when executing a VM. When the hypervisor handles an interrupt from a timer device which is assigned to a VM, for example the virtual timer, it will typically program a virtual interrupt using the GIC VE to signal to the VM that the timer has expired. However, since the timer output signal is level-triggered, the timer output will remain asserted until the VM, which has direct access to the timer hardware, cancels the timer or reprograms it to fire in the future. If the hypervisor deactivates the interrupt on behalf of the VM, the GIC will re-sample the output line from the timer, and since it is still asserted, it will immediate signal the new pending interrupt to the CPU, and the CPU will not make any forward progress, because it will be busy handling a continuous stream of interrupts. The ARM architecture is specifically designed to integrate the functionality of the timers with the GIC, by leaving the physical interrupt from a timer assigned to a VM active, which prevents further signaling of interrupts, and inject a virtual interrupt for the timer to VM, which is tied to the underlying physical interrupt by setting the HW bit in the list register. In that way, when the VM runs and observes the pending virtual timer, it will run the Interrupt Service Routine (ISR) for the timer, cancel the timer which lowers the timer output signal, and will finally deactivate the interrupt, for example by writing to the EOI register on the virtual CPU interface. Deactivating the virtual interrupt deactivates both the physical and virtual interrupt because the HW bit is set, and when the timer fires again later, it will trap to the hypervisor.

Each CPU has its own private set of timers, and it is the hypervisor’s responsibility to properly migrate and synchronize timers when migrating virtual CPUs across physical CPUs.