Performance

FreeBSD is used in performance-sensitive environments: high-volume web servers, routers, firewalls, file servers, and more recently, low-power consumer devices where I/O and computational overhead translate into reduced battery life. As discussed in Chapter 3, this implies a number of performance goals for the framework, including minimising overhead when the framework is not in use in order to make enabling the framework in the default kernel more acceptable, minimising framework overhead when it is used, and requiring policies to only pay performance penalties for features they de- pend on. The previous chapter’s performance analysis is based on the MAC Framework

3_{It should be observed that the FreeBSD Project’s own ideas about API and binary compatibility}

evolved significantly over this period, as pressure mounted from users to improve application and device driver compatibility between versions.

as shipped in FreeBSD 8.1 in 2010; those results were accomplished only after several years of iterative development, profiling, and feedback from real-world use. In this section, I consider various performance optimisations introduced since 2003 in order to reach the current results.

Labelled networking

Information flow MAC policies, such as Biba and MLS, require labelling and control of all potential communication paths in the system. In most MLS extensions to UNIX systems, the practical implementation of this is labelling not only of IPC primitives directly engaged by applications (such as sockets), but also in-flight packets passing between IPC endpoints. Packet labels are typically derived from the socket or network interface where the packet originated. In FreeBSD, struct mbuf chains represent in- flight packets, which must therefore carry labels when policies such as Biba and MLS are used.

In the initial FreeBSD 5.0 implementation, a struct label was embedded in the network stacks’s struct pkthdr, which carries meta-data for packets, such as their length, originating interface pointer, VLAN information, as well as (to a limited extent) processing state associated with adding and removing protocol headers. Semantically, this provided exactly the desired behaviour: one label for each in-flight packet; however, even when the framework was compiled out of the kernel, automatic zeroing of the 20- or 40-byte structure led to a several-percent performance overhead in per-packet processing.

In FreeBSD 5.1, the MAC Framework approach for mbufs was revised to use the

new m tag packet meta-data facility; this allows a linked list of data structures to be

hung off of in-flight packets for less commonly used features, such as IPsec and MAC. In this design, overhead for non-MAC kernels is eliminated, but a more significant overhead potentially exists for policies that employ packet labelling, as MAC labels are stored in external memory that must be allocated, freed, and indirected to for every packet. Further, if other kernel features employingmbuftags are present, such as IPsec,

allocation overheads are multiplied, and the number of indirections to follow the list (along with associated cache misses) is similarly increased. To mitigate this expense for MAC policies that do not employ per-packet labelling, a new policy flag was added,

MPC LOADTIME FLAG LABELMBUFS; only when at least one policy had set the flag would

the MAC Framework allocate labels formbufs.

Labelling of other kernel objects

While network processing is particularly sensitive to packet labelling, the same princi- ples also apply to other kernel object types. Somewhat lagged from the evolution of

mbuflabelling, the MAC Framework’s handling of other kernel objects has also evolved.

• reducing memory overhead when the MAC Framework is compiled out,

• changing the implementation ofstruct labelwithout affecting the binary layout of kernel data structures,

• changing the implementation ofstruct labelwithout affecting MAC policy mod-

ules that consume it, and

• avoiding allocating labels unused by the actual loaded set of policies, especially for object types with large standing allocations (such as vnodes).

To this end, in FreeBSD 5.2, all embedded instances ofstruct label were replaced with label pointers, avoiding allocation of labels for non-MAC kernels, as well as al- lowing the implementation of the label structure to be modified without affecting the layout of core kernel data structures. This design choice has two downsides, however: an additional indirection is added to every label access, and label structures must be independently allocated and freed, requiring additional calls to the kernel memory allo- cator. This trade-off is considered worthwhile, but as micro-benchmarks in Chapter 3 showed, the overhead of additional allocations is non-trivial.

An important motivation to move to an opaque and separately allocated label struc- ture was the desire to be able to modify the size and layout of the per-policy data array in the label without changing the kernel binary interface (KBI) for loadable modules. From inception, the MAC Framework had explicitly passed label pointers as well as ob- ject pointers to MAC policies in order to avoid policies incorporating dependence on the binary layout of core kernel data structures containing labels. However, in FreeBSD 7.0, the framework KPI for policies was modified to add accessor functions, mac label get

and mac label set, hiding the internal implementation of label structures from policies as well.

In FreeBSD 8.0, a general strategy of avoiding label allocation for each object type unless required by a specific policy was adopted: rather than using a policy flag for each object type, the framework determines whether labelling is required for each object type by analysing the entry points a policy implements when it registers, eliminating the need for a specific MPC LOADTIME FLAG LABELMBUFS policy flag. If a policy registers a label initialisation function for an object type, then the MAC Framework will allocate a

struct label for each instance of the object the kernel creates4. For policies compiled into the kernel or loaded at boot-time, this maintains the invariant that all instances of the object type will have label storage; for policies loaded dynamically, this guaranteed may not hold as objects may have been allocated before the policy was loaded. Rather

4_{Mac OS X further refines this approach for certain types, such as}

vnodes by allowing labels to be allocated on specific objects only when required; this model is difficult to impose on all kernel objects, however, due to policy invariants that labels always be available, as unconditional memory allocation is not possible in all kernel contexts.

than dynamically modify all outstanding object instances, which would incur significant overhead, it is instead the responsibility of the policy to handle this case.

In the case of policies used with FreeBSD, these trade-offs seem to work well: in- formation flow policies that depend on ubiquitous labelling for correctness are already statically loaded. Dynamically loaded policies will need to be able to tolerate a lack of labels on, for example, packets instantiated before the policy loaded regardless. How- ever, this consensus has not been accepted in all environments: McAfee’s Sidewinder firewall directly embeds label information in the struct mbufheader in their local im- plementation, rather than accept additional per-packet overhead. The version of the MAC Framework in Mac OS X, forked before the introduction of automatic policy interrogation for label use, makes different trade-offs for several data types due to po- tential memory overhead in a largely unlabelled OS environment; this is discussed in greater detail in Section 4.3.

Where possible, the MAC Framework relies on existing object locking in the kernel, avoiding additional synchronisation to protect labels where label access aligns with object access; this reflects the practical (and intentional) reality that per-object locks are often held at the time that access control checks are performed. For most labelled policies, this works well – for example, Biba and MLS set or modify labels only on object creation or during explicit relabel operations. For LOMAC, however, writes to labels may occur when only object read locks are held, requiring additional locking.

Optimisations to FreeBSD locking over time have necessarily modified MAC Frame- work locking – fine-grained object locking in the kernel implies fine-grained locking in the framework, both for correctness and performance. For example, as locks in differ- ent layers of the network stack were differentiated to allow greater parallelism between processing across layers, changes were required to not just locking in the MAC Frame- work, but also labelling: caches of socket-layer labels were introduced in network pro- tocol layers, such as on struct inpcb describing the IP-layer state for TCP and UDP connections. This avoids the need to enter the socket layer from the network protocol to check labels, as is done for other cached values in the inpcb.

Synchronisation overhead entering the framework

As policy modules can be loaded and unloaded dynamically, the MAC Framework pro- vides synchronisation to ensure that modules are not unloaded while functions they contain are in execution. In FreeBSD 5.1, this synchronisation is provided by a refer- ence count, protected by a mutex, that is incremented whenever an entry point begins executing, and decremented whenever entry points complete. Attempts to modify the list of policies, either to add or remove a policy, wait for the reference count to drain and acquire the mutex.

In FreeBSD 5.2, this approach was refined by differentiating “dynamic” policies from “static” policies: a dynamic policy can be loaded or unloaded, and hence requires synchronisation around its invocation, whereas a static policy does not. A policy is

considered static if it is loaded during boot and its policy flags indicate that it cannot be unloaded; the policy mutex is not held when iterating over statically configured policies, but is acquired to test whether dynamic policies are present, in which case the reference count is bumped (and later dropped through another acquisition of the mutex). This approach is further refined in FreeBSD 5.3 by adding a kernel option

MAC STATIC, which allows the set of policies to be declared fixed, preventing run-time loading and unloading, but entirely avoiding the need for synchronisation in the frame- work – a useful feature for routers and firewalls.

In FreeBSD 8.0, further effort was made to reduce overhead from options MAC to near-zero so that the framework could be included in the GENERIC (default) kernel. An analysis of synchronisation overhead from simply entering the framework on entry points was performed and the reference count was replaced with optimised locking primitives. To avoid deadlock, one of two locks is used depending on the type of entry point: certain entry points occur in contexts permitting unbounded sleeping (such as on slow disk I/O), but others (such as in the network stack) forbid it. The introduction of read-mostly locks in FreeBSD 8 meant that not only were atomic operations no longer required to synchronise the framework, but also also avoided writes to cache lines shared across CPUs. Combined with improvements in CPU performance and a shift to cache-centric CPU designs, the resulting overhead is minimised.

As of FreeBSD 8.0, the MAC Framework is compiled into the default kernel across all architectures; while there have been reports of overhead in certain edge cases, most measurements to date have indicated that on contemporary (cache-rich) hardware, the overhead of the framework when unused is measurably insignificant. Optimisation strategies between FreeBSD 5.0 and 8.0 changed substantially: focusing on minimising overhead for first when the framework was compiled out, and later compiled in, by default. Optimisation also changed with hardware: as the focus on instruction over- head became less important, minimising cache footprint and locking overhead became critical, reflecting a transition to multicore hardware.