Dynamic binary translation

In this subsection, we will give the results of actually nesting the virtual machines in- side a L1 hypervisor based on dynamic binary translation, as discussed in section 4.1. The nested hypervisors should not need modifications. Only the nested hypervisors based on hardware support for virtualization need a virtual processor that contains the hardware extensions. The L1 hypervisors are VirtualBox and VMware Worksta- tion using dynamic binary translation for virtualizing guests. Since we test two L1 hypervisors, this subsection describes 14 setups. These setups are described in the following paragraphs categorized by their technique for the L2 hypervisor. The first paragraphs elaborate on the setups that use dynamic binary translation on top of dynamic binary translation. The next paragraph presents the setups that use para- virtualization as the L2 hypervisor, followed by a paragraph that presents the setups that use hardware support as the L2 hypervisor. The last paragraph concludes this subsection with an overview.

Dynamic binary translation: Each setup that used dynamic binary trans-

lation as both the L1 and L2 hypervisor resulted in failure. The setups either hung or crashed when starting the inner guest. In the two setups where VMware Work- station was nested inside VMware Workstation and VirtualBox was nested inside VirtualBox, the L2 guest became unresponsive when started. After a few hours, the nested guests were still trying to start, so these setups could be marked as failures. In both setups the L1 and L2 hypervisors were the same, the developers know what instructions and functionality is used by the nested hypervisor and may have fore- seen this situation. However, the double layer of dynamic binary translation seems to be inoperative or too slow for a working nested setup with the same hypervisor for both L1 and L2 hypervisors.

The other two setups, where VMware Workstation is nested in VirtualBox and VirtualBox is nested in VMware Workstation, resulted in a crash. In the former setup the L1 VirtualBox guest crashed which indicates that the L2 guest tried to use functionality that is not fully supported by VirtualBox. This can be functionality that was left out in order to improve performance or a simple bug. In the other setup, with VMware Workstation as the L1 hypervisor and VirtualBox as the L2, the VirtualBox guest crashed but the VMware Workstation guest stayed operational. The L2 guest noticed that some conditions are not met and crashes with an assertion failure. In both setups, it seems that the L2 guest does not see a fully virtualized environment and one of the guests, in particularly VirtualBox, reports a crash. More information about the reported crash is given in section B.1. A possible reason that in both cases VirtualBox reports the crash is that VirtualBox is open source and can allow more information to be viewed by its users.

Paravirtualization: Of the two setups that use paravirtualization on top of

dynamic binary translation, one worked and the other crashed. Figure 5.1 shows the layers of these setups, where the L1 guest and the L2 hypervisor are represented by the same layer. The setup with VMware Workstation as the L1 hypervisor allowed

5.1. SOFTWARE SOLUTIONS 37

Figure 5.1: Layers for nested paravirtualization in dynamic binary translation.

a Xen guest to boot successfully. In the other setup, using VirtualBox, the L1 guest crashed and reported a message similar to the setup with VMware Workstation

inside VirtualBox (see section B.1). The result, one setup that works and one

that does not, gives some insight in the implementation of VMware Workstation and VirtualBox. The latter contains one or more bugs which make the L1 guest crash when a nested hypervisor starts a guest. The functionality could be left out deliberately because such a situation might not be very common. Leaving out these exceptional situations allows developers to focus on more important functionality for allowing virtualization. On the other hand, VMware Workstation does provide the functionality and could be considered more mature for nested virtualization using dynamic binary translation as the L1 hypervisor.

Hardware supported virtualization: VirtualBox and VMware Worksta-

tion do not provide the x86 virtualization processor extensions to their guests. This means that there is no hardware support available in the guests, neither for the processor, nor the memory management. Since four of the hypervisors are based on hardware support, there are eight setups that contain such a hypervisor. The lack of hardware support causes the failure of these eight setups. Implementing the hardware support in the L1 hypervisor using software, without underlying support from the processor, could result in bad performance. However, if performance is not an issue, such a setup could be useful to simulate a processor with hardware support on an incompatible processor.

Only one out of 14 setups worked with dynamic binary translation as the L1 hypervisor. The successful combination is the Xen hypervisor using paravirtuali- zation within the VMware Workstation hypervisor. Other setups hung or crashed and VirtualBox reported the most crashes. VirtualBox seems to contain some bugs that VMware Workstation does not have, resulting in crashes in the guest being virtualized by VirtualBox. Hardware support for virtualization is not present in the L1 guest using VMware Workstation or VirtualBox, which eliminates the eight setups with a nested hypervisor that needs the hardware extensions. Table 5.2 gives a summary of the setups described in this subsection. The columns represent the

5.1. SOFTWARE SOLUTIONS 38 Virtual- Box VMware DBT DBT VirtualBox DBT × × HV × × VMware DBT × × HV × × Xen PV × X HV × × KVM HV × ×

Table 5.2: The nesting setups with dynamic binary translation as the L1 hypervisor technique. DBT stands for dynamic binary translation, PV for paravirtualization and HV for hardware virtualization.

L1 hypervisors and the rows represent the L2 hypervisors.

5.1.2 Paravirtualization

Previous subsection described the setups that use dynamic binary translation as the L1 hypervisor. The following paragraphs elaborate on the use of a L1 hypervisor based on paravirtualization. In section 4.2, we concluded that nested virtualization with paravirtualization as a bottom layer needs modifications to the nested hyper- visor. The L1 hypervisor used for the tests is Xen. In all the nested setups, the L2 hypervisor should be modified to use the paravirtual interfaces offered by Xen instead of executing ring 0 code. We discuss the problems for each hypervisor tech- nique in the following paragraphs, together with what the setup would look like if the nested virtualization works. The last paragraph summarizes the setups described in this subsection.

Paravirtualization: The paravirtualized guest does not allow the start of a

Xen hypervisor within the guest. The kernel loaded in the paravirtualized guest is a kernel adapted for paravirtualization. The Xen hypervisor is not adapted to use the provided interface and hence the paravirtualized guest removes the other kernels from the bootloader. The complete setup, see figure 5.2, consists of Xen as the L1 hypervisor which automatically starts domain 0. This domain 0 is a L1 privileged guest. Another domain would run the nested hypervisor, which in turn would run its automatically started domain 0 and a nested virtual machine.

Dynamic binary translation: The hypervisor of VMware Workstation and

VirtualBox based on dynamic binary translation could not be loaded in the paravir- tualized guest. The reason is that the ring 0 code is not adapted for the paravir- tualization. In practice this expresses itself as the inability to compile the driver or module that needs to be loaded. It should be compiled against the kernel headers

5.1. SOFTWARE SOLUTIONS 39

Figure 5.2: Layers for nested Xen paravirtualization.

but fails to compile since it does not recognize the version of the adapted kernel and its headers. The setup for dynamic binary translation as technique for the nested hypervisor (see figure 5.3) differs from the previous setup (figure 5.2) in that the L2 hypervisor is on top of a guest operating system. Xen is a native, bare-metal hypervisor which runs directly on the hardware, i.e. in this case the virtual hard- ware. VMware Workstation and VirtualBox are hosted hypervisors and do need an operating system between the hypervisor and the virtual hardware.

Figure 5.3: Layers for nested dynamic binary translation in paravirtualization.

Hardware supported virtualization: The other four setups, where a nested

hypervisor based on hardware support is used, have the same problem. None of the hypervisors are modified to run in a paravirtualized environment. In addition, the virtualization extensions are not provided in the paravirtualized guest. Even if the hypervisors were adapted for the paravirtualization, they would still need these extensions. These setups look like figure 5.2 or figure 5.3, depending on whether the nested hypervisor is hosted or native, bare-metal.

None of the seven setups with paravirtualization as bottom layer worked. The results of the setups are shown in table 5.3. The column with the header “Xen” represents the L1 hypervisor. The main problem is the adaptation of the hypervisors.

5.2. FIRST GENERATION HARDWARE SUPPORT 40 XEN PV VirtualBox DBT × HV × VMware DBT × HV × Xen PV × HV × KVM HV ×

Table 5.3: The nesting setups with paravirtualization as the L1 hypervisor technique.

Unless these hypervisors are modified, paravirtualization is not a good choice as L1 hypervisor technique. It will always depend on the adaptation of the hypervisor and one could only use that hypervisor. When using paravirtualization, the best one could do is hope that developers adapt their hypervisors or modify the hypervisor oneself.