Each of these models has substantial limitations: some are inherent to extending com- plex systems, but others are properties of the extension mechanisms themselves. All assume either a lack of interest in the security extension from the perspective of the original OS vendor, or a commitment only to a narrow set of extensions.
3.3.1 Kernel source code access
Direct source code modification is premised on access to source code – implicit for open source systems, but historically problematic for widely-used proprietary systems. This has limited access by researchers, raised the cost of development, and required close vendor involvement in successful security extension work. Even where source access is possible for proprietary systems, intermittent dispersion of source code snapshots makes it difficult to track architectural changes, and precludes timely updates to third-party extensions. Where a vendor produces security extension products themselves, these problems may well exist within their organisation, and also discourage third parties from attempting the same.
3.3.2 Tracking vendor development
Operating systems are moving targets, even once a release has been shipped. A steady stream of major and minor updates, as well as critical security patches, risk security extensions falling out of sync with their targets; already true in 2000, this is even more the case today. This problem is especially troublesome for extension systems, such as system call interposition, that rely on a complete (and static) characterisation of the system’s application binary interfaces (ABI)4 in order to operate.
Between releases, active operating system development branches move even more quickly, with fundamental changes in internal architecture central to the creation of many new kernel services. These changes may have significant security side effects, changing assumptions about information flow, access control, or simply by changing the syntax and structure of code that has been modified or is depended on.
Other than in the area of specifically published APIs and structures (such as for file systems or device drivers), security extension authors can rely on little consistency between revisions. In addition, if a security extension relies on direct modification of operating system source code, there may be literal source code conflicts in changes to code modified by both the operating system vendor and security extension vendor, and a lag before the extended version of the OS product becomes available following an original OS vendor update.
Because of the significant change between releases, or even between service packs, any formal evaluations of the composed operating system and extension face substan- tial assurance challenges. Security vendors must make a complete argument for assur- ance not only concerning their own product, but also regarding the operating system vendor’s product, requiring high levels of additional investment for only incremental operating system improvements. The burden lies entirely with the extension developer to determine that the extension will operate correctly in the new environment.
3.3.3 Concurrency and lock order in threaded kernels
As discussed in Chapter 2, wrapping techniques, such as interposition and file sys- tem stacking, introduce fundamental problems in environments supporting kernel par- allelism: since the base system is unmodified, wrappers must ensure that appropriate synchronisation primitives are used to prevent time-of-check-to-time-of-use races within the kernel itself. In practice, this can require substantial duplication of work between the wrapper and the base component, as well as potential lock order violations and lock recursion (leading to deadlock).
For example, security extensions may require labels on files to provide protection
4Similar to an API, an ABI defines the binary interface between two components; typically this
will involve the names and types of symbols, as well as the binary layouts of data structures, etc. API changes often imply ABI changes, and like API changes, some ABI changes are backwards compatible (i.e., adding a new interface). Less innocuous ABI changes can lead to application misbehaviour and data corruption, requiring extreme care by OS vendors as they evolve their system interfaces.
for those files. To access the labels, the system call wrapper must perform a series of namespace lookups to traverse the file hierarchy to find the target of the operation. Once the check is performed, the wrapper must release all locks on the file and names- pace or risk violating the kernel’s lock order when the kernel attempts to perform the lookup operation. As locks are released, the namespace and protections on objects may change, resulting in a race condition between check and use. Similar races exist for all objects supporting fine-grained locking in the kernel: locks released on target processes in signal operations will permit the label on those processes to change before the kernel performs its own lookup, locking, and protection.
Since the MAC Framework design was first sketched out in 2000, this concern has only become more pressing: dual-core servers were unusual in 2000, while today, quad- core notebook computers are commonplace. Operating system vendors have invested significant effort in the intervening years to increase OS kernel concurrency in order to exploit this new hardware, making synchronisation with an interposed security frame- work even more difficult.
3.3.4 Policy composition
Trusted systems are often deployed with several different kernel access control poli- cies: UNIX discretionary access control (DAC), as well as one or more mandatory access control (MAC) policies. For example, PitBull and Trusted IRIX both ship with Biba-derived integrity policies for TCB protection, in addition to Multi-Level Security to protect the confidentiality of user data. Similarly, appliance vendors often com- bine vendor-specific hardening policies with base system access control features – for example, firewall and router vendors may rely on UNIX access control for base-line pro- tection, but supplement that with product-specific security features. Several examples of policy blends are considered in Chapter 4.
Safe composition of extensions is particularly important in environments where extensions may be used to respond to new threats not anticipated at design time. For example, new security limits might be imposed to reduce exposure to a newly discovered application vulnerability. However, the problem of access control policy composition is non-trivial, and a number of problems may arise in the development and use of composed access control systems.
Literal conflicts in source code
Security extensions tend to modify the same system components in the same places, resulting in a conflict: both sets of modifications cannot be simultaneously applied. For example, both Biba and MLS instrument the same file system accesses, even though they constrain flows of information in different directions.
Functional conflicts
Security extensions can also conflict in a purely functional sense, resulting in an un- usable or insecure system. Of particular concern are security extension that introduce new services or administrative interfaces, which in turn may not be properly contained by other security extensions limiting access to those interfaces. For example, naive simultaneous introduction of MLS and Biba might have each use policy-specific APIs to manage labels unprotected by the other policy. As a result, MLS would not limit the setting of Biba labels on files, and vice versa, introducing unprotected information flow channels in violation of both policies.
Fail-open composition
Security extensions may interact poorly when composed with other security extensions or applications, leading to fail-open semantics. For example, in early work to decompose system privilege in Linux, “compatibility” semantics were introduced allowing setuid root binaries to begin execution with less than full root privilege. In isolation, such a change seems reasonable – however, when composed with applications that assumed a root UID connotes full privilege, failure modes were problematic. For example, on security vulnerability resulted because sendmail did not check the return value of
setuid– prior to the kernel access control model change, this was safe (albeit possibly unwise). After the change, sendmaildid not properly detect that it had failed to drop privilege [122], allowing privilege escalation. Composition failures can have serious consequences, and are extremely difficult to reason about.
3.3.5 Financial cost of implementation
Many of technical limitations translate into increased development and maintenance cost. Of particular concern are:
• High level of complexity due to awkward composition of the extension and the base operating system, increasing the cost of implementation, testing, and maintenance of the extension.
• Minimal code reuse as different extension authors independently reproduce com- mon infrastructure, such as file system extended attributes.
• The moving target nature of operating system development increases maintenance costs and can lead to significant or complete re-engineering of the same extension over successive versions.
These costs discourage security extension development by third party vendors and researchers, resulting in poor adoption of new (or even old) access control approaches beyond those present in the base operating system. In a world of constantly evolving security requirements, addressing these concerns not only reduces the costs for, but also increases the effectiveness of, security researchers and security product developers.