Related Work

There are several classes of related work on virtualizing operating system code, including (a) OS support in virtualized processor architectures (or “codesigned virtual machines” [131]); (b) hardware abstractions in previous operating systems; (c) virtual machine monitors; and (d) operating systems that exploit language- based virtual machines or safe languages.

Four previous system architectures have used an organization with separate virtual and native instruction sets: the Transmeta Crusoe and its successors [55], the IBM DAISY project [61], the IBM S/38 and AS/400 families of systems [40], and the Low-Level Virtual Architecture (LLVA) [13]. Both Transmeta and DAISY emulated an existing “legacy” hardware ISA (x86 and PowerPC, respectively) as their virtual instruction set on a completely hidden VLIW processor. Both allowed existing operating systems written for the legacy instruction sets to be run with virtually no changes. These systems aim to implement legacy instruction sets

on novel hardware but do not make it easier to analyze or transform operating system code and, therefore, do not make it easier to enforce security policies on operating system or application code.

The IBM AS/400 (building on early ideas in the S/38) defined a high-level, hardware-independent in- terface called the Machine Interface (MI) that was the sole interface for all application software and for much of OS/400. The major difference between SVA and their MI is that their MI was effectively a part of the operating system (OS/400); significant components of OS/400 ran below the MI and were required to implement it. In their approach, a particular OS is organized to enable many or all of the benefits that SVA provides. SVA is OS-neutral and aims to provide similar benefits to existing operating systems without a major reorganization of the OS.

LLVA [13] forms the basis of SVA; it provides a virtual instruction set that is amenable to compiler analysis and transformation and supports C/C++ code. However, LLVA lacks the instructions needed for operating system kernel operations such as context switching, signal handler dispatch, and thread creation. It also provides no safety guarantees for application or kernel code. SVA extends LLVA with these features. Many modern operating systems include some design features to separate machine-independent and machine-dependent code, and at least a few do this by using an explicit architecture abstraction layer to define an interface to the hardware. Two such examples include the Windows Hardware Abstraction Layer (HAL) [119] and the Choices nanokernel [32]. These layers are integral parts of the OS and only achieve greater machine-independence and portability. In contrast, SVA’s instructions are an integral part of the (virtual) processor architecture and yet provides richer primitives than a traditional architecture for supporting an OS.

The Alpha processor’s PALCode layer [41] provides an abstraction layer similar to the one we designed for SVA. PALCode is special privileged code running below the OS that can execute special instructions and access special registers, e.g. the TLB. PALCode routines, like SVA instructions, act like additional instructions in the processor’s instruction set, have privileged access to hardware, and provide a higher level interface to the processor compared with traditional instruction sets. PALCode’s goals, however, differ significantly from SVA’s. PALCode is co-designed with the OS and moves OS specific functions into the abstraction layer. It does not hide the processor’s native ISA from the OS. Because it is OS-defined, it is unrecognizable by external compilers. In contrast, SVA hides the entire native ISA from software, provides OS-neutral abstractions, and has well-defined semantics. PALCode is logically part of the OS, whereas SVA is part of the processor architecture.

Virtual machine monitors such as VMWare [147], Denali [152] and Xen [60] virtualize hardware resources to enable sharing and isolation for multiple instances of operating systems. These systems are orthogonal to our work, which virtualizes the instruction set interface for a single instance of an OS. That said, stan- dardized interfaces for paravirtualized hypervisors [18] (in which the operating system kernel is modified for virtualization) abstract the hardware in a way similar to SVA, and Linux’s paravirtops interface [155] was the basis of SVA’s MMU configuration instructions (see Chapter 5).

Systems such as JavaOS [122], J-Kernel [74], JRes [53], KaffeOS [27] and the Java extension defined in JSR-121 [79] have incorporated OS-like features into or as a layer on top of a JVM. These systems aim to provide OS services to Java applications but are not designed to be general-purpose operating systems.

Chapter 3

Secure Virtual Architecture

3.1 Introduction

Many security policies are enforced via compiler instrumentation; examples include control-flow integrity [10], memory safety [108, 57, 15, 16], and information flow [85]. Furthermore, the protection provided by such compiler instrumentation can often be improved with the use of accurate static analysis. For example, SAFECode [57] and WIT [15] provide stronger security guarantees when they use increasingly accurate points-to analysis results. Likewise, Data-Flow Integrity (DFI) [33] provides stronger security when it can compute increasingly accurate results for reaching definitions analysis.

In this chapter1_{, we further describe the Secure Virtual Architecture (SVA) from Chapter 2. SVA is}

designed to support modern operating systems efficiently and with relatively little change to the OS. SVA leverages the virtual instruction set provided by LLVA to accurately and statically analyze OS kernel code and to instrument kernel code with run-time checks to enforce security policies. Since the OS kernel can be expressed without the use of assembly language code, SVA avoids the complications of analyzing and instrumenting assembly code. In addition, SVA utilizes the SVA-OS instructions from Chapter 2 to identify when the OS kernel is interacting with the hardware or manipulating application state; this allows SVA to enforce policies that require understanding when these operations take place.

Porting a kernel to SVA requires no major design changes; it is similar to, but significantly simpler than, porting to a new hardware architecture because SVA provides simple abstractions for the privileged operations performed by a kernel. Furthermore, the compilation and execution process is largely transparent: most application and kernel programmers would see no change when running SVA code (unless they inspect object code).

1_{This chapter is derived from a conference paper [50]. Andrew Lenharth helped with the design. The copyright owner has}

The rest of this chapter is organized as follows. Section 3.2 gives an overview of the SVA design. Sec- tion 3.3 describes the virtual instruction set and the kernel boot and execution process. Section 3.4 briefly describes the two implementations of SVA used in this work. Section 3.5 concludes.