Identifying Weaknesses in VM/Hypervisor Interfaces

(1)

Identifying Weaknesses in VM/Hypervisor Interfaces

Lucas McDaniel Department of Computer Science

University of Alaska Fairbanks

Kara Nance

Department of Computer Science University of Alaska Fairbanks

Abstract

As cloud and virtualized environments become more widely used to solve challenges faced by companies of all sizes, it is increasingly likely that this infrastructure will be a common focus of attacks in the years to come. Successful attacks against this infrastructure could allow an attacker to “break out” of the virtual environment and gain control of the physical infrastructure effectively compromising the entire system. Given the recent surge in the development and deployment of these environments, it is reasonable to expect that these systems have not undergone the same amount of testing that comes with age and wide acceptance, and which we often require for critical services. Therefore, in-depth analysis of the attack surfaces exposed by these environments is necessary to ensure the security of these systems.

This paper describes a Cyber Fast Track (CFT) project to create a testing framework to analyze the interfaces exposed by several hypervisors to potentially untrusted users (inside VMs) for vulnerabilities. The methods used consist of random input testing of emulated devices by intercepting and modifying valid device I/O with a state-aware system. These techniques are general enough that they may be extended to test many interfaces across a wide range of virtualization systems. This project then uses this tool against current versions of several virtualization systems with the ultimate goal to inform developers and system administrators alike about potential vulnerabilities in these systems.

Keywords:

Virtualization, Cloud Security,

Hypervisor, Automatic Test Generation

1 INTRODUCTION

Cloud computing and virtualization systems are quickly becoming standard additions to many company’s computing infrastructures as these types of systems offer very tangible benefits in the forms of reduced cost,

elasticity, increased efficiency, and unparalleled mobility. Critical systems can be virtualized to offer an increased level of resilience and reliability at minimal technical costs. The ability to dynamically modify the hardware of a virtual server as well as relocate it to a different datacenter with practically zero downtime are features that can easily be utilized by any company. It is easy to see how these advantages can overshadow the risks associated with the virtualization of critical systems. It can lead to adoption of cloud-based solutions which may not be well understood by system administrators and managers weighing the benefits of migrating to virtual systems or even the developers and architects of virtualization systems.

It is naïve to treat these virtualization components as simply another physical hard drive or server without fully analyzing the risks inherent to this approach. Virtualization at its core is designed to add a layer of abstraction between an application and an execution environment ideally to gain some tangible benefit such as greater security [1], portability [2], or ability to share resources [3]. However, this layer of abstraction provides an additional attack surface and a potential break through this layer of abstraction is a factor that needs to be taken into consideration. One of the goals of this project is to present a tool which can be used to give system administrators and managers a better understanding of these risks through empirical means.

Attacks against the virtualization layer are a real threat [4, 5, 6, 7]. Since the virtualization layer interacts with virtual machines and the underlying hardware, the interface between the hardware and VM as well as between two VMs is potentially a target for attack. Such interfaces could allow an attacker to “break out” of the virtual environment and gain control of the virtualization layer, allowing them (direct and hard to detect) access to the physical infrastructure and all VMs running on it. Virtualization suites are complex software systems, and are extremely likely to contain multiple vulnerabilities that have yet to be discovered (at least publicly).

The purpose of this research effort is to address the asymmetry in which attackers can target this relatively

2013 46th Hawaii International Conference on System Sciences

5087

2013 46th Hawaii International Conference on System Sciences

(2)

unexplored layer to find vulnerabilities and develop exploits, while defenders are generally left to patch systems after the fact when vulnerabilities are made public. By providing an automated testing framework that can be used to proactively identify vulnerabilities in the common VM/hypervisor interfaces, we can allow system administrators to more easily defend their systems, system managers to more accurately determine the risks of virtualization and cloud computing, and the developers and architects of virtualization systems to identify vulnerable code segments and more clearly understand the implications of their design choices.

Currently, testing of these interfaces currently tends to focus on functionality, and as such the priority tends to be ensuring that valid and predictable commands produce valid and predictable outputs [8]. The proposed method will take the approach more typically taken by those seeking to compromise systems (i.e., determining the response of a system to invalid or unexpected inputs or operations). In short, this project will build a testing framework aimed at the VM/hypervisor interfaces.

The primary interfaces that were targeted during this initial effort were device drivers for several commonly found emulated virtual devices within VMs (e.g., a device driver for a virtual network card). However, the techniques developed are expected to be generalizable to other interfaces as necessary (e.g. VM/hypervisor communication and optimization interfaces such as VMware tools).

2 BACKGROUND

Understanding current trends for use and testing of virtualization environments is the key motivating factor for this project. This project extends previous work done to test emulations in similar manners and provides a framework for performing these types of tests within the kernel.

2.1 Virtualization

Virtualization is certainly a mainstream component of enterprise IT server environments, and is also seeing increasing use in the workstation arena. It is also a fundamental driver of cloud computing, which is becoming a common fixture across the spectrum from home to enterprise computing environments. In light of that, the primary use case is one of providing virtualization system developers, architects, and administrators with insight into the weaknesses in both implementations and architectures. Because virtualization is at the heart of many of today’s IT systems (again, touching all levels of computing from home to enterprise, including the cloud), vulnerabilities in the virtualization layer can have significant impacts on the security of much of today’s systems. For example, a business that deploys

infrastructure in the cloud, and the cloud-provider making that system available, may understand the risks of traditional external attack against that system, and make take steps to appropriate mitigate that risk (such as OS and application level configurations). However, it is likely that neither the business nor the cloud-provider currently understands the risk associated with attacks against the virtualization layer itself, and as such may be providing quite inadequate protection against the threats they are actually facing [9].

At the highest level, the technical objective of this effort was to design and implement a test suite that can identify vulnerable components of the various VM/hypervisor interfaces. Such interfaces include a wide range of device drivers for emulated/virtual hardware devices, including network devices, storage devices, audio devices, and graphics devices, as well as the virtualized PCI implementation. These attack surfaces are exposed to users of a VM itself, who can, in the case of public clouds such as EC2 or compromised VMs in enterprise infrastructure, be attackers looking to gain control over the underlying physical infrastructure (and through that access to all VMs running on it). The interfaces can also potentially be exposed to external users and systems (e.g., an external system can interact with device drivers on a VM to some extent by sending network traffic to that VM). In order to test for vulnerabilities, we need a very granular method of monitoring and modifying the data sent and received from these interfaces. Instead of testing a virtual environment by sending completely random and unstructured data to the targets this project implemented a mechanism that could intercept and modify valid and structured data to allow for a more complete testing footprint. In order to accomplish this the interface for the corresponding targets was modified by inserting a wrapper around the function(s) for each testable operation. A set of at least 15-20 commonly used interfaces across several virtualization packages were modified and analyzed throughout this project.

Even after this level of control is established, it is beneficial to also control when the data will be altered, as certain vulnerabilities may be presented only after the device has been properly initialized and configured. This was achieved by dynamically generating rules customizable by a tester that describe when the data sent to or from an operation will be altered. By including this as the primary mechanism for altering the environment, not only can general rules be used across a wide range of targets, but also sets of rules can be honed to test a specific section of a target.

2.2 Related Work

Random input testing is a common method of analyzing software, and it has even been used to test

5088

(3)

virtualization packages [10, 13]. However, previous work testing emulated devices has been limited in the interfaces tested. Additionally, existing testing suites for virtualization packages largely focus on ensuring functionality is maintained across patches and updates, instead of identifying exploitable vulnerabilities and ensuring hardened systems [8]. This tool was designed to provide an automated method of analyzing new versions of virtualization packages that can be integrated into current testing suites.

Random input testing has proven to be highly successful at finding bugs within virtualization packages over the years:

• Cloudburst (2009). Discovered by researchers at Immunity, Inc., Cloudburst was a series of attacks discovered in VMware’s SVGA II emulation some of which allowed for arbitrary code injection on the host. [5]

• Virtunoid (2011). Discovered by Nelson Elhage, virtunoid an attack on KVM’s hot plug checking routine during device removals that allowed for arbitrary code injection. [6]

• In 2007, Tavis Ormandy produced a paper describing a series of bugs across many virtualization platforms generated by using crashme and a tool designed to simulate random IO port activity. [4]

This is but a small list of the vulnerabilities independent researchers have discovered in virtualization systems and the magnitude of these attacks should be indicative of the possibility of non-publically available attacks of similar scale.

3 RESEARCH

A preliminary project was conducted using similar techniques used in previous projects to gain a thorough understanding of how these methods held up against modern systems. This focused largely on analyzing the attack surface exposed by device communication methods. For the sake of simplicity, the follow statements regarding this communication process are only in regards to the x86 architecture.

3.1 Device Communication

There are two main communication methods that were analyzed during the preliminary project: Memory Mapped IO (MMIO) and IO ports. IO ports share many similarities with network sockets in that there are 65536 IO ports and values can be written to and read from them. The main difference is with the size of the values that may be sent that are limited to 8, 16, or 32 bit values and only one at any given time. MMIO, on the other hand, acts very similar to standard memory as the name implies

since it uses a region of the corresponding address bus. Both of these memory regions are be accessed through a series of kernel wrapper functions.

These mappings are initialized when the device is being configured after the appropriate driver has been found and loaded into the kernel [11]. For PCI devices, which consist of the majority of emulations tested during this project, this is done during the driver’s corresponding PCI probe function. When these mappings have been setup, they become the primary method for the driver to communicate with the device. While it is possible to access both MMIO and IO ports (after IO ports have been remapped to memory addresses) by dereferencing a void pointer, standard dictates that such communication should take place through a series of kernel wrapper functions:

• MMIO: io(read|write)(8|16|32) • IO ports: (in|out)(b|w|d)

For example, outb(0x378, 0xff) would write the value 0xff to IO port 0x378, and ioread8(0xF0001000) would return the byte value at memory address 0xF0001000).

It deserves to be mentioned that these methods of device access are common across many bus architectures; however, there are alternate methods to achieve this end. Direct Memory Access (DMA) is a very popular option for passing data between the driver and emulation because it does not require the CPU to facilitate this transfer. Once a DMA controller has been initialized (and the device has been set to be bus master), it is common for devices to transfer large blocks of data (e.g. frame buffers, network packets) through the controller since it doesn’t need to call kernel functions and thus doesn’t require the CPU to transfer the data. Since this method lacks appropriate functions to wrap, there isn’t a direct method of altering data sent through DMA controllers.

3.2 Preliminary Testing

Preliminary testing consisted largely of writing random values to the shared MMIO or IO ports to determine how the driver and emulated device would handle such alterations. Linux has nice interfaces to these areas exposed directly to userspace: /dev/port and /dev/mem. It should be noted that many distributions disallow access to /dev/mem by compiling the kernel with STRICT_DEVMEM enabled to prevent against some types of attack [12]. In order to use these interfaces for testing, the appropriate mappings need to be known. This information can be extracted through /proc/ioports, /proc/iomem, and lspci.

Preliminary testing first required creating several VM configurations using as many types of emulations as possible, and repeating for several virtualization platforms. A basic shell script was used to dd sections of

5089

(4)

/dev/urandom to sections of mapped memory and IO ports. While such methods lack any sophisticated knowledge of the device state and structure of the data contained in these regions, this technique is a remarkably useful at finding emulations that crash and reduced the list of emulations to check for the main project. A brief overview of these results can be found in the Results section.

4 CHALLENGES

There were several challenges identified prior to and during the execution of this CFT project. These include technical challenges, attack surface impact, and adversary perspective.

4.1 Technical Challenges

There are several technical challenges for this project, which include modifying a range of interfaces correctly and in a manner that can be generalized to a wide range of targets, and potentially having other components of the VM, including the kernel, intercept altered data on its way to the devices (meaning that we are in effect testing other components of the system instead of the VM/hypervisor interfaces). It is simply not practical to gain a thorough understanding of all the internal intricacies of a large number of drivers, for example; however, the testing utility is designed to allow analysis to be performed without requiring this high level of understanding. All analysis is performed by modifying the VM/hypervisor interface functions (e.g., driver read or write functions) to alter the data inline based on the defined test rules. Furthermore, when necessary the interface states can in most cases be determined by observing the common target structures (e.g., the pci_driver struct). While inadvertently targeting other components of the VM, such as the kernel, to some extent is probably unavoidable, modifying the data as close as possible to its use will reduce this risk. We can further reduce the risk of unintentionally putting the kernel into an inoperable state by restarting the VM with a new set of rules at predefined time intervals.

4.2 Impact

Attack surfaces are neither directly enlarged nor shrunken as a direct result of this tool (as it will not run on a VM we are attempting to protect). However, what will change is our understanding of the attack surfaces that already exist. Such understanding will allow the developers and architects of virtualized systems to identify weaknesses and engineer more resilient systems, for system administrators to defend their systems more completely (e.g., by blocking inbound traffic that may be valid for the network, but hazardous to the networking

stack of a VM), and for managers to more completely evaluate the risks of virtualized (and cloud-based) systems.

4.3 Adversary

This project doesn't change the playing field by making attacks on the VM/hypervisor interfaces infeasible, nor does it alter how such attacks could be performed. Instead this project gives the developers and the security community the tools to analyze implementations in the same manner that an attacker would search for vulnerabilities. By providing an easy to use and automated method of analyzing implementations, we can identify areas of weakness in both implementations and architectures, and thereby ensure that current deployments can be updated or reconfigured to address existing vulnerabilities, and that future versions can be designed based on the lessons learned from this analysis.

It is possible that this tool may empower at attacker to discover more vulnerabilities in the system. Since this tool is user agnostic, it is effective for both the defender and the adversary. By adding interfaces (e.g., additional device drivers) to the test suite for this tool, the adversary may be able to identify new vulnerabilities. As we will focus on implementing the most commonly used interfaces in this project, we expect that this benefit for the adversary will be limited to less common configurations (and will decline as more interfaces are added to the tool over time).

5 DESIGN

The testing environment is setup with multiple hypervisors each with multiple VMs all connected to each other via a local area network. For this project, we focused on KVM, Xen and ESXi, but several additional hypervisors were chosen for the preliminary testing. The controller, also on this network, manages the rule generation and logging for the entire system. The controller starts by turning on all relevant VMs and sending their rule sets via serial ports that are connected on the local network. Each driver loads its set of rules and performs standard operations always checking to see if a rule has been triggered and how the triggered rule should react. All logging is sent back to the controller through the same communication channel and stored in a database. After a fixed length of time has passed, the VM’s state is checked to ensure it has not crashed, and then all VMs are power cycled and given a new rule set to use.

5090

(5)

In order to appropriately modify reads and writes to the device, each driver needs to be modified to wrap all such functions. The project was designed with this in mind, so that very little additions are required to be placed into the driver which will be testing the emulation, and the bulk of the management comes from an additional kernel module which takes care of logging and rule checking. The structure can be broken down into how each driver was modified, how these modifications interacted with the backend, and how rules could be applied to alter the operations.

5.1 Driver Modifications

Each driver was modified to allow for two types of tests to be run which are referred to as Common Interface, and Atomic Operations. The drivers chosen for modification consisted of a selection of network, disk, audio, and graphics drivers used to control various emulated devices.

5.1.1 Common Interface

The goal of the Common Interface is to monitor internal state information of the driver and wrap each device access function so that they can be altered when a certain state has been reached. The “common” term is used since a tester who creates a set of rules to test one emulation may use the same set of rules to test another emulation provided they share similar states. The internal state information is saved in a structure located in the pci_dev struct, and external functions are called to determine if a rule has been met.

5.1.2 Atomic Operations

Atomic Operations are defined as a series of read or writes to the device (or a series of functions that are called

to perform such actions) that are deterministic and reliable in execution such as a network driver enabling TX buffers before RX buffers. By adding a few jump labels as well as a control block to manage when these labels should be jumped to allows for this reliance to be tested by changing the order of certain functions, and repeating or ignoring sections of the driver. The “atomic” term is used to signify that at the driver level, these operations are indivisible sets of reads and writes that once started must always be completed in the same order.

5.2 Driver Backed

The backend consists of another driver (normally compiled into the kernel) which manages loading rules, logging, storing state information, determining if a rule has been met, and how the data should be altered given a rule has been met. This backend consists of several exported functions that are called by the modified drivers anytime a read or write is to be executed. These functions first check the current state of the driver and determine if a rule matching that state has been received. If a rule has been met, then the system modifies the access according to the rule’s specification, otherwise it calls the original kernel function and passes the return values back.

Prior to any operation being performed, the current state of the driver and any rule that has been met is logged via serial ports sent to the logging system listening on the other end that saves this information in a database. This provides an easy metric for determining if rules are properly being evaluated.

The rules get loaded during the PCI probe operation. At this time, modified drivers make a request out through the serial port for their rule set, which are stored within the pci_dev structure. This communication also contains information regarding the logging level, memory mappings to expect, and an assortment of other information to assist with correctly identifying states.

5.3 Rule Generation

Peach Fuzzer is the framework being used for the backend because of its highly modular design and test case generation managed through XML files referred to as Peach Pits [13]. All information regarding the test is stored in these Peach Pits including the rules to be used, which sets of VMs to be cycled for each test, and how long the VMs should be left running; all of this makes it remarkably easy to anyone to modify and create new rules to test emulations with.

For each test, the VMs are powered on and these rules are then set via the network to the serial ports on the VMs being tested. Monitors determine if the VM has been prematurely shutdown as the result of a test and this information is logged. Given a crash, log files are extracted from the hypervisor as well as the last rule triggered before the crash to assist a tester with

5091

(6)

identifying the reason for the crash. Since these tests are largely automated, the entire system was designed to be capable of being integrated inside an existing testing environment to increase the testing footprint of such tests.

6 RESULTS

This project is a DARPA-funded Cyber Fast Track Project and ended in August 2012. Since this paper focuses on the framework developed, comprehensive analysis of the crashes produced and the extent into which they may be exploited will not be explored. Instead, we seek to highlight a few crashes to demonstrate the successfulness of testing modern hypervisor interfaces with the proposed method.

The tool was used to test a wide range of configurations across several hypervisors. Emphasis was placed on Xen, KVM, and ESXi as they were specifically required for the CFT but a few others were also chosen. These configurations were tested over a two-week period and a subset of the interesting crashes is provided below.

6.1 VMware Workstation

Using the previously mention methods of testing these emulation, we’ve identified an emulation that reliably crashes giving access violation error:

This was found in a recent version of VMWare Workstation (8.0.4 build-744019).

6.2 KVM

An emulation in KVM / QEMU reliably produces a double free error:

Also found using a recent version of KVM (qemu-kvm-0.12.5).

6.3 ESXi 5.0

An emulation in a recent build of ESXi 5.0

produces a purple screen of death (PSOD).

7 FUTURE RESEARCH

While this project has modified over a dozen commonly used device drivers, this is a small number of the drivers controlling potential emulated devices or interfaces between the VM and hypervisor. The most clear method of extending this testing tool is to increase the number of drivers included in the package by focusing on the less commonly used emulations as they will most likely have less testing and may be more vulnerable to such types of attacks.

Support for finer resolution rules can allow testers to more quickly determine vulnerable components with a higher level of certainty. For instance, limiting the range of addresses to alter to only control addresses and values to write to known sets of control codes can allow for more intelligent testing. By increasing the granularity of these rules, the system could benefit greatly from an analysis engine that would refine rules sent to the device to hone in on more specific sub-rules given a crash.

5092

(7)

The project lacked a strong method of altering data sent through DMA because of the nature of this access. However, identifying the structure of the data and altering it prior to it being sent to the emulation (such as making alterations to network data at the netdev level instead of the driver level) would provide the ability to test how the virtualization layer is able to parse these structures, a task that is currently a non-trivial process to do with the current version of the tool.

8 REFERENCES

[1] Goldberg, I., D. Wagner, R. Thomas, and E. Brewer. A Secure Environment for Untrusted Helper Applications. 1996 USENIX Security Symposium, 1996.

[2] Gosling, J. “Java: An Overview,” Retrieved June,

2012 from http://www.cs.dartmouth.edu/~mckeeman/cs118/refe

rences/OriginalJavaWhitepaper.pdf

[3] VMWare Staff, “Virtualization overview”. White Paper: Retrieved June, 2012 from http://www.vmware.com/pdf/virtualization.pdf

[4] Ormandy, T. “An Empirical Study into the Security Exposure to Hosts of Hostile Virtualized Environments.” Retrieved June, 2012 from http://taviso.decsystem.org/virtsec.pdf

[5] Immunity, Inc. “Cloudburst.” Black Hat USA June 2009. Retrieved June, 2012 from

http://www.blackhat.com/presentations/bh-usa- 09/KORTCHINSKY/BHUSA09-Kortchinsky-Cloudburst-PAPER.pdf

[6] Elhage, N. “Virtunoid: Breaking out of KVM. “ August 8, 2011. Defcon 2011. Retrieved June, 2012 from http://nelhage.com/talks/kvm-defcon-2011.pdf [7] Gruskovnjak, Jordan. “VUPEN Vulnerability

Research Team (VRT) Blog”. Retrieved September, 2012 from http://www.vupen.com/blog/20120904. Advanced_Exploitation_of_Xen_Sysret_VM_Escape _CVE-2012-0217.php

[8] Kamalesh, B., S. Balbir. “Keeping the Linux Kernel Honest” In Proceedings of the Linux Symposium, pages 19-20, 2008.

[9] Stephen Kaisler, William H. Money, Stephen J. Cohen, "A Decision Framework for Cloud Computing," 2012. 45th Hawaii International Conference on System Sciences, 2012

[10] Carrette, G. “Crashme: Random Input Testing”.

Retrieved June, 2012 from http://people.delphiforums.com/gjc/crashme.html

[11] Opdenacker, M. “Linux PCI Drivers.” Retrieved June, 2012 from http://free-electrons.com/doc/pci-drivers.pdf

[12] Lineberry, A. “Malicious Code Injection via /dev/mem.” March 27, 2009. Black Hat Europe 2009. Retrieved June, 2012 from http://www.blackhat.com/presentations/bh-europe- 09/Lineberry/BlackHat-Europe-2009-Lineberry-code-injection-via-dev-mem.pdf

[13] M. Eddington, “Peach Fuzzing Platform”, 2009. http://peachfuzzer.com/

5093