4.3 Hardware supported virtualization
5.1.3 Overview software solutions
Previous subsections explain the results of nested virtualization with software solu- tions for the bottom layer hypervisor. This subsection gives an overview of all the possible setups described in the previous subsections. All these setups are gathered in table 5.4. The columns of the table represent the setups belonging to the same L1 hypervisor. The rows in the table indicate a different nested hypervisor, i.e. the hypervisor represented by the row is nested inside the hypervisor represented by the column.
Nested x86 virtualization using a L1 hypervisor based on a software solution is not successful. Out of the 21 setups that were tested, only one setup allows to successfully boot a L2 guest: nesting Xen inside VMware Workstation. Note that 12 setups are unsuccessful simply because hardware support for x86 virtualization is not available in the L1 guest.
5.2
First generation hardware support
The setups with a bottom layer hypervisor based on hardware support are described in this section. The theoretical possibilities and requirements needed for these se- tups are discussed in section 4.3. The conclusion was that it should be possible to nest virtual machines without modifying the guest operating systems, given that the physical processor provides the hardware extensions for x86 virtualization. In
5.2. FIRST GENERATION HARDWARE SUPPORT 41
Virtual-
Box VMware XEN
DBT DBT PV Subsection 5.1.1 5.1.1 5.1.2 VirtualBox DBT × × × HV × × × VMware DBT × × × HV × × × Xen PV × X × HV × × × KVM HV × × ×
Table 5.4: Overview of the nesting setups with a software solution as the L1 hyper- visor technique.
chapter 3, the hardware support for x86 virtualization was divided into the first gen- eration and second generation hardware support. The second generation hardware support adds a hardware supported memory management unit so that the hyper- visor does not need to maintain shadow tables. The original research was done on a processor1 that did not have second generation hardware support. Detailed
information about the hypervisor versions is listed in section B.3. To make a com- parison between first generation and second generation hardware support for x86 virtualization, the setups were also tested on a newer processor2 that does provide
the hardware supported MMU. The results of the tests on the newer processor are given in section 5.3.
The tested L1 hypervisors using the hardware extensions for virtualization are VirtualBox, VMware Workstation, Xen and KVM. We nested the seven hypervisors (see table 3.1) within these four hypervisors, resulting in 28 setups. In the first subsection the nested hypervisor is based on dynamic binary translation. The second subsection described the setups with Xen paravirtualization as the L2 hypervisor. The last subsection handles the setups with a nested hypervisor based on hardware support for x86 virtualization.
1
Setups with a L1 hypervisor based on first generation hardware support for x86 virtualization were tested on an Intel R
CoreTM2 Quad Q9550 processor. 2
Setups with a L1 hypervisor based on second generation hardware support for x86 virtualization were tested on an Intel R
5.2. FIRST GENERATION HARDWARE SUPPORT 42
5.2.1 Dynamic binary translation
Using dynamic binary translation as the nested hypervisor technique, there are eight setups. Three of these setups are able to successfully boot and run a nested virtual machine. The layout of these setups can be seen in figure 5.4 where the L1 hypervisor is based on hardware support and the L2 hypervisor is based on dynamic binary translation. When Xen is used as the L1 hypervisor, the host OS layer can be left out and a domain 0 is started next to VM1, which still uses hardware support for its virtualization.
Figure 5.4: Layers for nested dynamic binary translation in a hypervisor based on hardware support.
VirtualBox: When VirtualBox based on hardware support is used as the bot-
tom layer hypervisor, none of the setups worked. Nesting VirtualBox inside Virtual- Box resulted in the L2 guest becoming unresponsive. The same result happened when VirtualBox was nested in VirtualBox but used dynamic binary translation for both levels. When trying to nest a VMware Workstation guest inside VirtualBox, the configuration of that setup is very unstable so that each minor change resulted in a setup that refuses to start the L2 guest. There was one working configuration which we listed in section B.3.
VMware Workstation: If the L1 hypervisor in figure 5.4 is VMware Work-
station, the setups were successful in nesting virtual machines. Both VirtualBox and VMware Workstation as nested hypervisors based on dynamic binary transla- tion were able to start the L2 guest which booted and ran correctly.
Xen: VMware Workstation3 checks whether there is an underlying hypervisor
running. It noticed that Xen was running and refused to start a nested guest. This prevents a L2 VMware guest from starting within a Xen guest. In the other setup, where VirtualBox is used as inner hypervisor, the L2 again became unresponsive
5.2. FIRST GENERATION HARDWARE SUPPORT 43
after starting. There is no crash, error message or warning which might indicate that the L2 guest booted at a very slow pace.
KVM: The third and last working setup for nesting a hypervisor based on
dynamic binary translation within one based on hardware support is nesting VMware Workstation inside KVM. In newer versions of VMware Workstation4, a check for an underlying hypervisor noticed that KVM was running and refused to boot a nested guest. The setup with VirtualBox as the nested hypervisor crashed while booting. The L2 guest showed an error indicating a kernel panic because it could not synchronize. The guest became unresponsive after displaying the message.
Virtual-
Box VMware XEN KVM
HV HV HV HV
VirtualBox DBT × X × ×
VMware DBT ∼ X × X
Table 5.5: The nesting setups with first generation hardware support as the L1 hypervisor technique and DBT as the L2 hypervisor technique.
Table 5.5 gives a summary of the eight setups discussed in this subsection. VM- ware Workstation is the best option since it allows nesting other hypervisors based on dynamic binary translation, but it will also most likely work when used as nested hypervisor based on dynamic binary translation. In comparison to nesting inside a software solution, VirtualBox is able to nest within VMware Workstation when using hardware support for the L1 hypervisor. VirtualBox is still not able to nest within KVM, Xen and within itself, while VMware Workstation is able to nest within KVM and itself. It is regretful that VMware Workstation checks for an underlying hypervisor, other than VMware itself, to prevent the use of VMware Workstation within other hypervisors.
5.2.2 Paravirtualization
In this subsection, we discuss the setups that nest a paravirtualized guest inside a guest virtualized using hardware support. Figure 5.5 shows the layers in these setups. The main differences with the setups in the previous subsection are that the L1 guest and the L2 hypervisor are represented by the same layer and that Xen automatically starts domain 0.
There are just four setups tested in this subsection since only Xen is nested within the four hypervisors based on hardware support. All four setups could successfully nest a paravirtualized guest inside the L1 guest. However, the setup where Xen is nested inside VirtualBox was not very stable. Sometimes during the start-up of the privileged domain several segmentation faults occurred. Domain 0 was able to boot and run successfully but the creation of another paravirtualized guest was sometimes
4
5.2. FIRST GENERATION HARDWARE SUPPORT 44
Figure 5.5: Layers for nested paravirtualization in a hypervisor based on hardware support.
impossible. Xen reported that the guest is created, however, it did not show up in the list of virtual machines indicating that the guest crashed immediately.
Virtual-
Box VMware XEN KVM
HV HV HV HV
Xen PV ∼ X X X
Table 5.6: The nesting setups with first generation hardware support as the L1 hypervisor technique and PV as the L2 hypervisor technique.
An overview of the four setups is shown in table 5.6. It is clear that using paravirtualization as technique for the nested hypervisor can be recommended. The only setup that does not completely work is the one with VirtualBox. Since the other three setups work and since previous conclusions were also not in favor of VirtualBox, VirtualBox is probably the reason for the instability.
5.2.3 Hardware supported virtualization
The remaining setups, which attempt to nest a hypervisor based on hardware sup- ported virtualization, are discussed in this subsection. Nesting the four hypervisors based on hardware support within each other results in 16 setups. The layout of the setups is equal to figure 5.4 and figure 5.5, depending on which hypervisor is used. None of the hypervisors provide the x86 virtualization processor extensions to their guests indicating that none of the setups will work.
Developers of both KVM and Xen are working on support for nested hardware support. Detailed information can be found in section 5.4. KVM has already released initial patches for nested hardware support on AMD processors and is working on patches for the nested support on Intel processors. Xen is also researching the ability
5.2. FIRST GENERATION HARDWARE SUPPORT 45
to nest the hardware support so that nested virtual machines can use the hardware extensions.
Virtual-
Box VMware XEN KVM
HV HV HV HV
VirtualBox HV × × × ×
VMware HV × × × ×
Xen HV × × × ×
KVM HV × × × ×
Table 5.7: The nesting setups with first generation hardware support as the L1 and L2 hypervisor technique.
The results of this subsection are summarized in table 5.7. It is regretful that currently none of the setups work because the L1 hypervisors do not yet provide the hardware support for virtualization to the guests. Nonetheless, it is hopeful that KVM and Xen are doing research and work in this area. Their work can motivate managers and developers of other hypervisors to provide these hardware extensions to their guests as well.
We would like to note that VMware and VirtualBox guests with a 64 bit oper- ating system need hardware support to execute. If we would use a 64 bit operating system for the nested guest, the result would be the same as the results in this section since there is currently no nested hardware support.