Customised Hardware and Software Packet Monitoring Architectures

There exists a very long and continuously increasing list of commercial and open-source software packet capturing/monitoring tools, yet their description here would be ineffectual since it would not provide a deeper insight into measurement techniques and architectures, and is indeed outside the scope of this thesis. The interested reader is referred to [SLAC] that provides a comprehensive and regularly updated list of available network monitoring tools. Such tools mainly build on top of common packet capture libraries and data-link access technologies (see e.g. section 2.3.3.2), and mainly focus on the analysis and visualisation of monitored packet data. Some of their features include the flexible specification of packet filtering rules, the decomposition of traffic data down to individual hosts, protocols and applications, and sometimes the troubleshooting or intruder detection and alarm generation based on certain traffic load conditions [MoQu04]. TCPDump48 _{is definitely worth-}

mentioning as an example, since it is the ancestor of many similar tools; it is based on libpcap

to read a configurable amount of bytes for each packet arriving at an interface and stores captured data on disk. TCPdump uses a high-level syntax to implement sophisticated packet filtering.

The remainder of this section provides a description of indicative research-driven customised hardware (and software) monitoring platforms, mainly developed to passively capture data on backbone and high-speed network links, and whose results were then used to characterise Internet traffic behaviour and/or to provide for network operations and engineering tasks. These architectures should mainly serve as examples of customised equipment that provide a complete bundle for data link access, packet capture methodologies, and captured data analysis techniques over high capacity network interfaces.

47 _{http://www.ist-scampi.org/events/workshop-2003/donnelly.pdf} 48 _{http://www.tcpdump.org/}

• OC*MON Systems

OC3MON [ApCT96] was among the first research-oriented, custom-built packet monitoring platforms, developed just after mid-1990s by MCI49 _{and CAIDA within the context of} NSF’s50 _{very high speed Backbone Network Service (vBNS}51_{) project. At the time, ATM} trunks at OC-3c had started being increasingly used for high-volume backbone links, and the already widely deployed statistics gathering tools based on lower-capacity multi-access media (e.g. FDDI and Ethernet) could not easily scale to higher speeds. Hence, the design of OC3MON aimed at providing a flexible and relatively low-cost data collection and analysis system to monitor these high-speed links, and to be easily extensible to operate on even higher rates.

The monitor was built on an IBM-compatible PC and was equipped with two ATM interface cards with an onboard processor that allowed custom firmware to be loaded to optimise the system’s operation. The receive port of each ATM card connected to the monitor port of an optical splitter, allowing OC3MON to capture traffic over an OC-3 link, in both directions. OC3MON employed customised DOS-based software consisting of device drivers and a TCP/IP stack, and higher-level software to perform real-time flow analysis. During the initial design of the monitor, the otherwise less-advanced features of DOS provided better control over the entire machine and its TCP/IP stack operation, lower interrupt latency than UNIX, and more efficient card-to-host buffer transfers by not fragmenting the large blocks of contiguous physical memory of the machine. However, a FreeBSD UNIX port of OC3MON was soon developed and released, in response to community feedback.

The monitor supports raw cell trace capture, active flow reporting, and expired flow analysis, as three different modes of data collection. When operating in the first mode, OC3MON produces a timestamped raw cell trace of either every cell or the first cell of every AAL5 frame, without further analysing the captured data. In the other two modes, the monitor collects statistics regarding active or expired flows using configurable flow definitions.

OC3MON has been used to report utilisation statistics and packet size distributions of backbone trunks, as well as to derive flow profile information from captured packets and produce parameterise-able flow statistics analysis.

A highly-cited Internet backbone measurements study based on traffic statistics collected by OC3MON systems revealed some important properties for the nature of the Internet [ClMT98]. Among others, it was stated that 60% of packets seen in MCI’s backbone are 44

49 _{http://www.mci.com/} 50 _{http://www.nsf.gov/} 51 _{http://www.vbns.net/}

bytes or less, however, they constitute 7% of the byte volume; over half of the bytes are carried in packets of size 1500 bytes or larger. TCP was reported to dominate the traffic mix, contributing to approximately 95% of bytes, 90% of packets, and 80% of the flows on the monitored links. Web traffic also dominated the monitored links, comprising up to 75% of the bytes, 70% of the packets, and 75% of the flows, when client and server traffic are considered together. Although this study has been conducted in 1998, it is widely believed (and sometimes proven from recent measurement studies) that the broad nature of the Internet still remains largely the same.

When MCI’s backbone transitioned from OC-3 to OC-12 rates, OC3MON naturally evolved to the OC12MON, which used new OC12 ATM cards and generally maintained almost the same specifications. The whole backbone OC monitoring project has been progressively renamed to the Coral measurement architecture, which became the ancestor of the more recent CoralReef suite (described below).

• Sprint Backbone IP Monitoring Project

The IP Monitoring (IPMON) project52 _{deployed a passive packet monitoring and analysis} infrastructure at certain Points Of Presence (POPs) of the Sprint Tier-1 IP backbone network. The system was designed to collect synchronised traces of data from multiple links to use in research projects studying, among others, the network delay performance, the behaviour of TCP, the nature of Denial of Service (DoS) attacks, and the development of network engineering tools.

The core component of the infrastructure is the IPMON systems, which collect IP and transport layer headers of packets transmitted through the monitored links. In total 31 IPMONs were scheduled to be installed at three different POPs of the Sprint network. Each IPMON is a probe architecture consisting of a Linux PC equipped with a large disk array and a DAG capturing card (section 2.3.3.3), which is attached on selected OC-3, OC-12, and OC- 48 links. For each captured packet, the first 44 bytes of IP data and a 64-bit timestamp generated upon arrival of the beginning of the packet are stored in a packet record, which is then transferred to the IPMON’s main memory. When 1 MB of packet records has been transferred to memory, an interrupt from the card triggers an application to copy the data to the hard disk. The uniqueness of the project lies in the ability of the infrastructure to collect traces at different points (IPMONs) in the network and correlate them through highly accurate timestamps [FrDL01]. Packet timestamps generated by the DAG cards are synchronised to within 5 µs using a stratum-1 GPS reference clock distributed to the IPMON systems.

A large tape library acts as the data repository for each IPMON system and archives the trace data. The IPMONs transfer the sets of collected traces to the data repository over a dedicated OC-3 link. It has been reported that a 24-hour-long trace from 11 IPMONs deployed at a single POP consumes approximately 1.2 TB of disk space, and therefore the trace data is compressed before being transferred to achieve to 2:1 to 3:1 data reduction ratio.

A 16-node Linux-based computer cluster is used for two categories of off-line data analysis.

Single trace analysis process data from a single link to measure traffic characteristics such as packet and flow size distributions, and dominant application traffic at certain links. Multi- trace analysis focuses on correlating traffic measurements among different links to perform one-way delay measurements and round-trip TCP behaviour analysis. Multi-stage analysis requires packet identification within traces from different IPMONs, at different links, which is a costly operation, and is hence performed by dividing each trace into several time segments and processing each one in parallel on different machines of the analysis cluster. Packet identification at multiple locations is based on comparing the unchanged fields of IP and transport headers (i.e. all but the TTL and IP checksum fields). However, link layer retransmissions and incorrect IP ID field generation by systems’ stacks can result in different packets appearing as being identical. Within the IPMON project these ‘false positives’ are a relatively rare phenomenon, representing 0.001% to 0.1% of the total traffic volume. Preliminary analysis of captured packet data within the Sprint backbone network focused on investigating the amount and (a-)symmetry of link utilisation, the application traffic mix, the packet size distribution, and the one-way delay experienced between different backbone links [FrDL01].

• CoralReef

CoralReef53 _{is a CAIDA project for developing and maintaining a comprehensive passive} monitoring software suite that consists of a set of high performance Internet traffic data collection and analysis modules/tools. The main objective of this suite is to support a superset of monitoring features and to provide APIs at many layers, so that it can serve as an engine for measurement and data processing applications to be easily built on top of it. The CoralReef architecture (Figure 2-18) is organised into two stacks of software components for raw traffic, and flow identification and analysis, respectively. The core of the raw traffic stack is a C library (libcoral) that provides an API for capturing traffic from multi-vendor specialised ATM and PoS monitoring cards and from pcap interfaces, at the same time hiding the details of the hardware and drivers from the higher layers of the stack.

Figure 2-18: CoralReef Software Components

Device drivers are provided for collecting data from a variety of supported specialised hardware cards (including DAG cards: section 2.3.3.3), as well as reading packet data from files of various formats to facilitate offline traffic analysis. Also supporting libpcap makes the raw traffic stack backward-compatible with existing IP analysis applications that can be built on it. CoralReef includes applications that perform a variety of passive monitoring tasks, such as capturing raw Payload Data Units (PDU)s to a file, traffic filtering, timestamp analysis and trace conversion to different formats.

The flows stack aggregates data across time to flows, based on principles and criteria similar to those adopted by flow monitoring architectures (see section 2.3.2). It includes modules for storage and manipulation of flow tables, and it provides interval and counter values containing expiration and cumulative volume information for each flow, respectively.

Detailed information on the CoralReef architecture, its building blocks and the main functions it supports, can be found at [KeMK01].

• AT&T Labs PacketScope

PacketScopes are custom-built packet monitoring components of a broader prototype infrastructure for measurement, storage and correlation of network data deployed at AT&T’s commercial IP network. The infrastructure [CaDF00] addresses network-wide traffic analysis by employing a combination of passive packet capturing, router-based flow statistics collection, and active probing of network paths between major router centres. All types of measurement data are fed into a high-performance, custom-built data repository, and are collectively post-processed to enable the characterisation of numerous aspects of Internet traffic. Data sets include packet headers captured by PacketScopes, Netflow flow statistics extracted from Internet Gateway Routers (IGRs), routing and forwarding tables from backbone and access routers, SNMP-based router statistics, server logs collected at AT&T’s web hosting service, and loss/delay/throughput active measurements.

PacketScope (initially deployed in 1997) is a high-performance system for collection of IP packet headers installed at representative locations of the AT&T ISP network, including a T3 (approx. 44 Mb/s) point-to-point peering link and two FDDI rings carrying traffic to/from modem banks and the web-hosting service, respectively. Each PacketScope consists of a UNIX® workstation, a striped disk array and a tape-robot, and has a dedicated T1 (approx. 1.5 Mb/s) access to the data repository for remote data collection. The workstation performs passive packet header capture using a modified version of the TCPdump tool, which enhances the packet transfer from the device driver to the packet filter utility to provide for reliable packet capture and graceful overload behaviour. Access to the operational traffic is provided in different ways, depending on the link each PacketScope operates. Monitors are attached directly to the multi-access FDDI rings, whereas in the T3 link case, an otherwise idle router gets a copy of the T3 line signal in each direction of the link and forwards all packets to the monitor through a dedicated 100 Mb/s Ethernet link [AnCD97].

Two special tools for monitoring multimedia traffic and for HTTP protocol tracing are developed within PacketScopes. The former parses a number of multimedia session control protocols to setup the appropriate packet filters to capture dynamically negotiated multimedia sessions, and the latter combines IP packets into TCP connections and then reconstructs individual HTTP transactions54_.

PacketScope traces have been used to among others evaluate the impact of streaming media and VoIP applications on the network, to characterise the large-scale behaviour of web traffic, to study the performance of web proxy caching, and to assess the effectiveness of usage-based dial service pricing schemes.