The Linux Kernel Network Model

The in-line measurement modules extend the Linux kernel’s networking functions and enhance packet-handling with in-line measurement functionality. As it was raised in the previous section, this kernel-level implementation of the modules provides the advantages of processing efficiency and operational transparency, which come at the cost of the additional implementation complexity due to the interaction with kernel-level data structures and processing routines. The remainder of this section focuses on the brief description of both packet generation and reception within the Linux OS kernel, and some of the most important methods123 available to manipulate packets across the different layers of the networking stack are outlined.

The top level of the protocol stack is formed by applications running in user-space that communicate over a networked environment by sending packets through sockets to a transport layer (TCP, UDP) and then on to the network layer (IP). After determining the route to a remote system, the kernel sends packets to a network device which is the bottom layer of the Linux protocol stack. The link layer output interface then ultimately forwards packets out over the physical medium. Upon reception on the physical medium of the remote system at the other end of communication, the link layer input interface will copy and forward the packet to the network layer, which will in turn route it and, if it is addressed to the local host, pass it up to the transport layer. The transport layer looks up the socket associated with the transport port specified in the packet header and puts it at the end of that socket’s receive queue124[Herr00]. Having many layers of network protocols each one using the services of another makes it necessary for the host OS to maintain state to hold information for each respective protocol

123 _{These methods were instrumental for the measurement modules to be able to insert and extract}

information within the datagrams, in the form of network layer headers.

124 _{This is a very broad description of the basic network stack operations in Linux. It intentionally}

avoids getting into the details of the several sanity and error checks that each layer performs to outgoing and incoming packets.

header, as the packet traverses the different levels of the protocol stack. When constructing network datagrams for transmission as well as when receiving link layer frames, maintaining the strict layering of protocols without continuously copying parameters between them is facilitated by the common packet data structure. The so-called socket buffer is a central

structure of the network implementation used to represent and manage either sending or received packets during their entire processing lifetime in the kernel.

When a network device receives packets from the network, it converts data into socket buffer structures which are then added to the backlog queue to be processed by the corresponding protocol handling routines. Similarly, when packets are generated either by applications or by network protocols, a socket buffer is built to contain the data and the various headers added by the protocol layers as the packet passes through them [Rusl99].

Using the concept of socket buffers, the payload data of each packet is only copied twice as the packet makes its way through the protocols; once between kernel and user space when the data is passed to/from the application process, and once between kernel space and the link medium when data is sent/received on the network. Figure 4-4 shows the structure of a socket buffer, as well as how different socket buffers are managed by Linux in a doubly-linked queue structure.

Each socket buffer has two conceptual parts to hold different information related to a packet. The Management Data part holds additional data, not necessarily stored in the actual packet, yet still required by the kernel to process each packet. This data include pointers to adjacent socket buffers and to the socket buffer’s current queue, a pointer to the owing socket that created the packet, a pointer to the network device on which the socket buffer currently operates, and a timestamp indicating when the packet arrived in the kernel. Additionally, pointers to the packet headers for the different layers (MAC, network, transport), as well as pointers to the total location that can be used for packet data, and the currently valid packet data are also included. The Packet Data part of the socket buffer stores the data actually transmitted over a network. Since the packet data space is allocated in advance, it is necessary to accommodate for maximum size scenarios (at the cost of wasting a few bytes) so that re- allocations are avoided during packet handling/creation. As each layer handles the packet, free storage in the Packet Data area changes and can be found in front (headroom) or behind (tailroom) the currently valid packet data.

Linux offers numerous functions to manage and manipulate socket buffers which can be grouped into two primary sets. Routines to manipulate doubly linked lists of socket buffers, and functions for controlling the attached memory [WePR05]. Figure 4-5 shows a sequence of functions frequently used to manage the Packet Data part of a socket buffer, and highlights the space alterations caused when each function is called. When a socket buffer is created (alloc_skb), all the available space of the Packet Data area is at the end (tailroom). A subsequent function (skb_reserve) allows specifying some room at the beginning (headroom) and further functions can be used to add data either at the headroom (skb_push) or at the tailroom (skb_put) of the Packet Data area, provided enough space has been reserved. Figure 4-5 also shows how the pointers that mark the valid data within the total

Packet Data location are manipulated by these routines. Additional functions to remove data form the headroom and the tailroom, to copy and clone125 _{a socket buffer, as well as to free a} socket buffer provided it is not cloned, are among others implemented in the kernel.