Protected Execution and Secure Operating Systems

The ability to boot securely and to reliably attest the status of a platform to a user or a third party is of limited use if the software booted on the platform is not trusted in the first place. For some applications, trust in the software can be achieved by only allowing authorized code to run on the platform in the first place. This is a plausible solution for many smartcard solutions or specialized hardware security modules. On more-general platforms, like a personal computer or a mobile phone, however, this approach is hardly possible. For one, commonly used operating systems feature a complexity that makes it practically impossible not to have security vulnerabilities in the operating system itself. With a kernel size of several million lines of source code, and (estimated) the order of one security-relevant bug for each 1000 lines of code, such systems are insecure by design. Furthermore, it is harder to impose restrictions on programs run on such a platform. On both user and business platforms, it is very common to have various applications on the same platform that may try to negatively interfere with each other. Therefore, the next important step towards a trusted platform is to build trust into the correctness of the software, and prevent it from being corrupted at runtime.

Most current operating systems show a great weakness in this area, which is due to the basic design principles. The kernel, i.e., the part of the operating system that has unlimited access rights, features millions of lines of code, and one corrupted application can quickly compromise the entire system. To make things worse, automated attack tools that use already compromised systems to scan the network for vulnerable platforms have drastically increased the prob- ability of an attack on the running system; in some experiments, we saw the first attacks in less than a minute after connecting a computer to the Internet. Thus, currently the greatest challenge on the way towards trusted platforms is the design of operating systems in a way that allows the containment of an

128 K. Kursawe

attacker and prevents security problems in a single application to compromise the whole system. The first step towards this goal is mandatory access con- trol, as already implemented in some Linux versions [12]. This ensures that each program gets only the access rights it absolutely needs to perform its tasks. Thus, for example, a game has no access rights to the email program’s address book. To achieve real security, however, a more fundamental change in the structure of the operating system is necessary:

• Different processes on the platform must be strongly separated from each

other. Apart from a clearly defined and controlled API, two processes on the same machine should not even be able to send messages to each other, let alone tamper with each other’s execution.

• The trusted computing base, i.e., the code that one has to trust to have

no security-relevant bugs or back doors, must be as small as possible. Current security kernels have about 10,000 lines of source code, which makes a formal verification for correctness possible. This code should do not much more than resource management and message passing between the processes; all other functionality, ideally including the user interface and device drivers, should already be protected from each other.

While systems exist that implement these principles, most legacy oper- ating systems have not been designed with process separation in mind; as applications for such systems need to be able to run on a secure operating system, this requires a major redesign of the operating system while keeping the system compatible with legacy applications. The first attempt towards this goal was to create a secure operating system in parallel with the legacy one by using a secure processor mode [13]. While the insecure mode would run legacy applications without change, a new, independent operating system in the secure mode would execute security-critical applications, protected from the original, insecure system by the hardware and from each other by a secu- rity kernel. While this approach worked for some systems, it met significant problems in many others, especially in the PC world. The design would have required many applications to be split in two parts, the (large) legacy part and the security-critical part. The problems of splitting applications in this way, and defining the proper APIs between the two components, caused this approach to be largely abandoned. The alternative approach is to run the legacy operating system on top of the secure system. On top of the security kernel, a virtualization layer is built that emulates one or several indepen- dent platforms. The legacy operating system – or even several instances of it – run in a virtual machine, protected from the other instances that run on the same platform, but in different virtual machines. A number of systems are already available that implement variations of this design, though only recently have security issues become a major focus, and received major fund- ing [14,15,16,17]. For the time being, though, hardware support is insufficient; the virtualization layer requires either a modification of the legacy code, or

9 Trusted Platforms 129

imposes a high performance penalty on the system. Future processor archi- tectures, such as AMD’s Pacifica and Intel’s Silvervale, will have integrated support for virtualization techniques.

A different approach has been taken by Microsoft’s Singularity project and the JCOP JavaCard. Instead of building a virtual machine for generic code, the system only allows applications in a programming language that itself ensures isolation of different programs, as Java does due to the Java virtual machine. While this proved successful for small, largely legacy-free systems such as a smartcard, it remains to be seen if the approach can be scaled into a large platform.