OPERATING-SYSTEM LAYERING

Although the concept of layering is not unique to secure operating systems, it is particularly useful for them because it promotes a structured design that can help satisfy some assurance needs. A layered operating system is one whose internal structure looks like a stack of systems, each having an interface for use by the layers above (table 11-1).

Layers in a system are strictly hierarchical: the lower layers provide primitive functions, and the higher layers use the primitive functions to provide more complex functions. By employing the technique of data hiding, the software in each layer maintains its own global data and does not directly reference data outside its layer. Each layer knows only about its own data and about the set of functions available to it below; it knows nothing about higher layers except such information as can be deduced from interactions across the interface. The rule is “downward calls only.”

One way we can view the functions of the layers resembles the way we view layers in a kernel-based system implementing a virtual machine (refer to figure 10-3), where each of the three layers is a complete mini-operating system providing functions in all necessary areas: object management, process management, I/O, and so on. The mini-operating systems in the higher layers have more functions and are more complex than those in the lower layers. In such a structure, there is no reason why any layer need use functions belonging to a layer other than the one immediately below. The primary difference between the layering inside the operating system in table 11-1 and that in figure 10-3 is that inside the operating system there is no need to enforce vertical isolation between portions of the upper layers, as there is between processes or virtual machines. Performance of operating systems layered in this way tends to be poor because many of the frequently used simple functions that are implemented in the lower layers can be accessed only by a cascade of calls through multiple upper layers.

Table 11-1. Operating-system Layers. Each layer provides a set of functions for the

layers above it, using functions available to it in the layers below. Everything at or below layer 8 in this example is within the operating system (or security kernel). This architecture resembles that of PSOS (Neumann et al. 1980).

Another view constrains each layer to a particular subset of functions, as in the example in table 11-1. The hierarchical structure. (downward calls only) is maintained, but no single layer provides enough interfaces to be usable as an operating system in itself. A layer may directly call functions several layers down, bypassing intermediate layers. In table 11-1, for example, a user application in layer 10 may be, permitted to read and write a file by directly calling layer 4, without necessarily passing through intermediate layers.

The structure whereby a layer services a single functional area is more efficient, because the layer need not participate in functions of the lower layers; but it is also more error-prone, because software in a higher layer may be able to bypass intermediate layers and directly access objects that the intermediate layers should handle. In table 111, for example, layer 5 creates directories using the files provided by layer 4. If programs in layer 6 (or above) always call layer 5 before calling lower layers, the integrity of the directories can be maintained by layer 5. But if a program makes a direct call from layer 10 into layer 4, for example, that program might be able to write into a file containing a directory without the directory manager’s knowledge. To handle this type of problem gracefully, each layer must provide pass-through interfaces to functions in layers beneath it, allowing lower-layer functions to be used only where the data-hiding constraints of the layer are not violated. In our example, the directory layer would provide pass- through calls for the open-file function in the file layer, but only after checking the arguments of the open-file call to make sure that files containing directories are not opened.

Regardless of how the layers are interpreted, security decisions usually must be made by most layers. Normally, having a single “security layer” is impossible because each layer has its own set of objects that require secure management. It would be nice if, for example, the operating-system interface layer in table 11-1 could validate the legitimacy of all system calls, since that would relieve the lower layers of the duty to check their arguments. But the ability to

Layer Function 10 user applications

9 command language interpreter 8 operating system interface 7 device input/output 6 high-level processes 5 directories 4 files 3 segments 2 pages 1 low-level processes 0 hardware Outside Operating System Inside Operating System

do so, especially for the pass-through calls, might require that the interface layer know about details of the lower-layer data structures, thereby violating the data hiding principle of layering. If a whole operating system existed in each layer, with no pass-through functions, such knowledge would not be required because each layer would fully protect all its objects.

In most systems, the layering structure is enforced during development, rather than at run time, using design guidelines and development tools such as compilers that prevent out-of-layer references. When in execution, the entire operating system runs in one or two domains, and the hardware has little or no role in preventing erroneous software in the operating system from violating the layering. While the layering structure looks as though it could be handled by hardware that supports hierarchical domains, there is little security advantage in adopting this strategy because the entire operating system (or security kernel) is security-relevant. Once a system call passes through the operating system interface. and satisfies the initial checks of its arguments, any lower layer can cause a security violation if it misbehaves.

In summary, layering supports design verification by promoting a clean architecture and reducing the chance for design and implementation errors. Some verification techniques model the layering structure (see section 12.6.2). Using hardware to enforce layering during execution may add robustness to the system in the face of programming errors, but it adds little measurable security.