Efficient Methodical Internet Topology Discovery

(1)

Efficient Methodical

Internet Topology

Discovery

Alistair King

Supervisor: Dr Matthew Luckie

This report is submitted in partial fulfilment of the requirements

for the degree of Bachelor of Computing and Mathematical

Sciences with Honours at the University of Waikato.

(2)

c

(3)

Abstract

There are a wide-ranging and diverse set of tools and techniques which are used by researchers to gain an understanding of the topological layout of the Inter-net. These current macroscopic Internet topology mapping techniques however, are either inefficient, do not discover the complete topology accurately or simply have not been implemented in a useful manner. In order to carry out large-scale Internet mapping projects, a system which can map Internet topologies in an accurate and methodical manner without taxing the infrastructure is required. This report outlines the implementation and testing of an existing efficient al-gorithm in a useful context and also the research, design, implementation and testing of a novel algorithm, doubletree load balancer, which allows complete topologies to be discovered in a way that is far more efficient than the current state of the art. Results from testing show that doubletree load balancer has over 96% link coverage when compared to load balancer traceroute while using 30% fewer probes.

(4)

Acknowledgements

The author would like to thank the following people for their con-tributions to this project:

Dr Matthew Luckie - for supervising and providing invaluable con-tributions and oversight to the project.

WAND Group - for providing all sorts of miscellaneous help such as feedback and advice on presentations and support with LA_TEX.

CAIDA - for providing helpful advice and a location to work and conduct testing from.

Thanks also to friends and family who have provided support in a multitude of ways, without which, this project would not have been possible.

(5)

List of Acronyms

CAIDA _{Cooperative Association for Internet Data Analysis}

BGP _{Border Gateway Protocol}

DNS _{Domain Name Service}

ICMP _{Internet Control Measurement Protocol}

ISP _{Internet Service Provider}

IP _{Internet Protocol}

IPID _{Internet Protocol Identifier}

IPv4 _{Internet Protocol version 4}

SDSC _{San Diego Supercomputer Center}

SSH _{Secure Shell}

TOS _{Type Of Service}

TTL _{Time To Live}

UDP _{User Datagram Protocol}

(9)

Chapter 1 Introduction

There is fairly extensive research into the area of Internet topology discovery from which a number of tools and techniques have been developed which aim to improve on various aspects of the field. Many of the tools developed to date provide only a best-effort approach to topology discovery; that is, they make little attempt to ensure all of the topology has been discovered. This report has two main focuses. The first is the implementation of an existing traceroute improvement within a system which is currently in use in the longest running Internet topology mapping project. The second focus is the proposal, development and implementation of a new algorithm for efficiently gaining a complete and accurate map of Internet topology.

1.1 Motivation

It is useful for Internet researchers who are trying to develop models of Internet structure and growth to have accurate and complete maps of Internet topol-ogy. Creating these maps is a non-trivial task as the the majority of network operators are commercial organisations and do not make public the structural details of their networks in a detail required for good science. Due to this, tools have been created which utilise features of the Internet Protocol (IP) to infer what Internet topologies look like. The technical details of these tools will be

(10)

src dst (a) direct path from src to dest src A B C D dst (b) realistic path from

src to dest

Figure 1.1: Example of a common misconception of Internet paths. A, B, C and D are interfaces of routers on the path between a source and destination.

described in the next section.

Due to the scale of the Internet, inferring topologies is both time consuming and network intensive. In order to improve the efficiency of topology discovery, a technique, doubletree [9], has been developed which significantly reduces the amount of network traffic and time required. However, doubletree has never been implemented within a system which is, or even can be, used by the general research community.

Another problem with the conventional tools for Internet topology discov-ery is that they are prone to missing a significant amount of topology due to their assumption that there is only one path between two points in the Internet. There has been research into methods for discovering all of the possible paths [2]. However, the technique is even more time and resource intensive than the con-ventional methods. The goal of this work is to improve macroscopic Internet topology discovery to discover all possible paths without taking a prohibitively large amount of time and probes.

1.2 Internet Topology Basics

1.2.1 Internet Topology Structure

Most end users of the Internet do not know, and in fact do not need to know, the technical details behind how data gets from their computer to the host at the other end of the connection. In fact, to the end user, it could appear that their data travels directly to its destination as shown in Figure 1.1a. In

(11)

reality however, the data traverses many intermediate routers (hops), so the path between two points could look more like Figure 1.1b. The goal of topology discovery is to correctly infer the path between any two given points and thus the set of links which make up the path. By selecting a large number of destinations, and using a number of sources as vantage points, we can begin to assemble a map of the Internet.

1.2.2 Traceroute

Because there is no easy way to discover Internet topology, a technique is used which exploits a feature of the Internet Protocol. The technique is known as Traceroute [11] and works by setting the Time To Live (TTL) field in the IP header of a packet which it then sends towards the destination of the path to be inferred. The TTL field is supposed to be used to prevent a packet from never exiting a routing loop. In most operating systems, the TTL field in the IP header is set to 64 when the packet is sent. Each router that the packet visits on the path, decrements the field by one. Once the value reaches one, the packet expires; the router at which it expires sends an Internet Control Measurement Protocol (ICMP) Time Exceeded packet back to the source telling the source that the packet did not make it to the destination.

Traceroute uses the TTL field to elicit a response from a router at a defined distance into the network. The response contains the address of an interface at the router at which the packet expired. To build up a view of the path between a source and a given destination, traceroute starts by sending a probe packet with a TTL of one, records the address of the responding router and then sends a probe with a TTL of two and so on until a response is received from the address that it was probing toward. There are also other conditions which traceroute stops on depending on the implementation. For example, if a routing loop is detected or a series of hops are encountered which do not respond to probes. Because traceroute needs to send at least one probe per hop, in order to discover a complete path, many probes must be sent into the network. A

(12)

src A B D F dst (a) single path between source and destination (b) two paths between

source and destination C

B

E D

src A F dst

Figure 1.2: Simple example of Load Balancing. Load balancing is used to distribute traffic over redundant links.

path is comprised of a set of links which are inferred network links between the interfaces of two hops which have been discovered. An interface is an IP address of a router in the path. A router will have several IP addresses.

1.2.3 Load Balancing

Traceroute assumes that there is only one path between any two given points in the Internet. In reality it is possible for there to be several different paths between a source and destination. An example of what paths can actually look like can be seen in Figure 1.2. This is possible due to routers which perform a function known as ‘load balancing’. Load balancing routers have at least two links which they can forward outgoing packets over thus increasing capacity.

1.2.4 Load Balancer Traceroute

To be confident that all of the paths between a source and destination have been discovered given load balancing, a technique known as load balancer tracer-oute [3] is used. Load balancer tracertracer-oute works on the assumption that the majority of load balancing in the Internet is per flow. This allows load balancer traceroute to manipulate the paths that specific packets take through the net-work by manipulating the flow identifier of the packet. To discover a path with load balancer traceroute, a series of probes with varying flow identifier are sent to a hop in order to discover all the interfaces present. The reason that the

(13)

n 95% 99% n 95% 99% 2 6 8 12 63 82 3 11 15 13 70 90 4 16 21 14 76 98 5 21 28 15 83 106 6 27 36 16 90 115 7 33 43 17 96 123 8 38 51 18 103 132 9 44 58 19 110 140 10 51 66 20 117 149 11 57 74

Table 1.1: Number of probes which must be sent to rule out a hop having n interfaces, given a confidence level of 95% or 99%. [3]

flow identifier is varied is that if there is a load balancing router before the hop being probed, it will forward the packets out different links and thus different reply interfaces will be seen. Each hop is probed a certain number of times depending on the confidence level required and the number of interfaces discov-ered. Table 1.1 shows the number of probes that must be sent to have 95% or 99% confidence that all interfaces have been seen. Load balancer traceroute is discussed further in a later chapter.

1.3 The Problem

The problem with using conventional traceroute and load balancer traceroute for large scale Internet topology measurement projects is that they are designed to be used for tracing from a single source to a single destination. When tracer-oute is used to probe a large destination list, it becomes expensive in terms of both network traffic and time. Also, the traffic generated by traceroute is un-solicited and so can appear to system administrators as port scanning or other malicious activity. Generally speaking, macroscopic topology mapping projects have no confidence that they have seen all of the topology and the technique to enumerate the complete topology is prohibitively expensive in both time and probes.

(14)

1.4 Contributions

This project has made several contributions to the field of macroscopic Internet topology discovery. The first is the implementation and testing of doubletree, an existing efficient Internet topology discovery algorithm. Second a new algorithm, doubletree load balancer, which aims give confidence that a complete topology has been discovered whilst conserving time and network resources, has been proposed, implemented and tested.

This report is structured as follows: this first chapter introduces the moti-vation of the work and the basics of Internet topology discovery upon which the rest of the report builds on. The second chapter provides some background into the field of macroscopic Internet topology discovery, putting this project in con-text. The third chapter introduces and describes the doubletree load balancer algorithm which this project has developed. The fourth chapter documents the implementation work carried out with regard to doubletree, doubletree load bal-ancer, a distributed measurement control server and alias resolution. The fifth chapter reports on the results of testing the various components developed by this project. The sixth chapter concludes the report and describes future work that may be carried out.

(15)

Chapter 2 Background

Because of the exponential growth of the Internet in the 1990s, monitoring its topology became an increasingly difficult task. Out of this dilemma were born several projects and techniques which aimed to create topological maps of the Internet.

2.1 BGP

Border Gateway Protocol (BGP) is the de-facto external Internet routing pro-tocol which allows routers to advertise the locations that they can forward packets to and the sequence of networks through which the packets will be forwarded. A BGP routing table consists of a list of prefixes, which are IP ad-dresses ranges, that a router can forward packets to and the sequence of Internet Service Providers that the packets will be forwarded through.

2.2 Existing Topology Discovery Systems

There are several topology discovery systems that have been used over the course of the last decade. They all differ slightly in the method that they use but their purpose is largely the same. That is, to collect data about Internet topology which can be used by researchers to create models of the Internet. Several of

(16)

these projects are based on having a set of geographically distributed vantages points which use some variation of traceroute to collect data.

2.2.1 skitter

skitter [13] was one of the first large-scale projects to make use of a system of distributed monitors which constantly probe lists of addresses. skitter ran for ten years until February 2008 and had 18 monitors strategically positioned world-wide, many of which probed more than half a million addresses. skitter used traceroute to gather data about the routing structures of the Internet.

2.2.2 Ark

Ark [4] is the successor to the skitter infrastructure. Ark has two primary goals. The first is to provide a platform which reduces the effort which is required to develop and deploy large scale measurement projects. The second goal is to enable the research community to run measurement tasks on a secure, distributed platform.

Ark is a network of 27 monitors distributed throughout the world which all work in a coordinated fashion to conduct large scale measurements. Currently Ark is being used to perform large scale traceroute based topology measurements using team probing. Team probing means that the work is dynamically divided up between team members which allows the measurements to be made relatively quickly - between 48 to 56 hours for a team of 13 monitors to probe seven million address ranges [4].

2.2.3 Mercator

Mercator [12] is another program which carried out large-scale mapping of the Internet. Unlike skitter, Mercator does not start with a list of known addresses to probe. Mercator uses a technique known as informed random address probing which allows the program to start with no state and generate addresses to

(17)

A F E G src B C D dest BGP path src routed path

Figure 2.1: Source routing example. By using source routing, the default path for a packet can be overridden and the packet detoured through E, F and G.

probe based on the subnet of the monitor. Mercator then branches out semi-randomly, eventually probing almost all of the advertised, routable Internet Protocol version 4 (IPv4) address space.

Rather than using multiple physical monitors spread around the world as skitter does, Mercator uses source-routing to bypass the standard BGP defined route to the destination. This allows Mercator to see paths through the Internet that a monitor probing from one static location normally would not. Figure 2.1 illustrates how by using source routing a packet can be detoured from its de-fault route through the network. Only a small fraction of the routers in the Internet support source-routing (around 8%). However, every router that does support routing gives the same perspective as a Mercator instance running at that vantage point in the Internet.

Because Mercator is designed to be fully self-contained, it has the advantage of not needing to rely on access to any external resources, such as BGP routing tables. There is a disadvantage to this feature however. This is that due to the semi-‘blind’ nature of Mercator’s probing, far more probes are sent than are needed to form a map of the network. This, like skitter, means that large-scale, long-term probing of the Internet with Mercator could become intrusive and and an annoyance.

(18)

2.2.4 Rocketfuel

Rocketfuel [20] is an attempt to carry out Internet Service Provider (ISP) map-ping using an a method other than the brute-force, trace every address, method of the previous two systems described.

Rocketfuel makes use of several techniques to carry out its mapping. The first technique is known as directed probing. Directed probing makes use of BGP routing tables to identify paths which traverse the ISP being mapped. Once the set of paths which transit the ISP is identified, Rocketfuel uses a series of path reduction methods to identify traceroute probes which probably take the same path within the ISP. At this point traceroutes are made using the probes identified. Once the traceroutes have been carried out, Rocketfuel uses a form of alias reduction to merge the different interfaces of a router into one. Routers have more than one interface and each interface has its own unique address. As traceroute only identifies interfaces, alias reduction is used to identify individual routers in the topology. Alias reduction (or resolution) is the process by which a router’s aliases are identified and merged into a single representation of the router. Alias reduction methods are discussed in a later chapter.

Rocketfuel is very efficient and performs the least number of traces possi-ble to reliably map the links in an ISP. Because of this feature, it is rather unobtrusive and places very little load on the network infrastructure.

2.2.5 Summary

While these projects have provided invaluable data to the research community, their continued viability is questionable due to the inefficient methods that are used. As the projects are scaled up, so is the amount of unsolicited network traffic that they create.

A caveat of the brute-force approach that many of these systems use is that there is a set of links which will be probed many times due to the tree-like structure of the Internet. That is, the path from a vantage point into the so-called core of the Internet will be fairly consistent from trace to trace. This will

(19)

src A B C dst1 dst2 dst3 dst4 (a) monitor−rooted tree

src1 src4 src2 src3 dst A C B (b) destination−rooted tree

Figure 2.2: Internet path trees. (a) is an example of the tree which is seen when probing multiple destinations from a single vantage point. (b) is an example of the tree seen when probing a single destination from multiple vantage points.

cause unnecessary duplication of effort by the monitor. Also, a facet of Internet routing design which causes several of these techniques to produce inaccurate results is load balancing. Due to load balancing, a different path can be seen with each probe sent into the network. This can result in the generated map not being true to the actual topology of the network.

2.3 Improving Traceroute Efficiency

2.3.1 Doubletree

Doubletree [9, 6, 7] is an algorithm which extends the classic traceroute al-gorithm to reduce the number of probes needed when tracing a large set of addresses from a distributed set of monitors. Doubletree exploits the tree-like properties of paths in the Internet.

Internet paths form trees because of how the various commercial networks which comprise the Internet are connected. For a packet to reach any destination in the Internet, it must first traverse and exit the network of the ISP which is providing connectivity for the sender of the packet. The provider is likely to only have a couple of points which traffic can exit its network, it is at this point

(20)

that the trunk of the tree begins to branch. Each autonomous network that the packet travels through introduces further branching into the tree provided that it has multiple outgoing connections to other networks.

The doubletree algorithm treats the paths between the monitors and the des-tinations as two trees as illustrated in Figure 2.2. One tree is monitor rooted, leading to the destinations which that monitor can reach. The other is destina-tion rooted, leading to the monitors that can reach that destinadestina-tion. A monitor probes hop by hop along a path until it reaches an interface seen by other mon-itors probing that destination, assumes that it has reached a known branch and therefore the rest of the path to the destination is already known. Because there are two trees, there are two probing schemes in doubletree, forwards probing for the destination-rooted tree and backwards probing for the monitor-rooted tree. Doubletree makes use of these trees by remembering the interfaces from each tree that it has seen during probing. It does this by having two sets of data known as stop sets. There is the local stop set which is the a representation of the monitor-rooted tree, and the global stop set which is a representation of the destination-rooted tree. The local stop set contains all of the interfaces that a single monitor has seen while probing. The global stop set contains all the interfaces that the other monitors have seen probing a single destination. The global stop set is shared between all of the monitors participating in the probing; the local stop set never leaves the monitor.

Doubletree starts probing at some number of hops h from the monitor. h is determined by using prior knowledge about path lengths from a monitor so that the value of h minimizes the probability that the destination will be reached with the first probe. This allows doubletree to start probing at a distance sufficiently far into the network to avoid re-probing hops close to the monitor, yet not so far that it is re-probing hops that other monitors have discovered while probing this destination. From h, doubletree probes forwards to h+1, h+2, etc., until it reaches an interface another monitor has seen before. It then starts probing backward from h-1 until it reaches an interface it has seen before.

(21)

number of times, rather than with almost every trace as in classic traceroute, doubletree is able to probe not only much faster than other large-scale algo-rithms, but it can also probe without causing the operators of routers and hosts to think that they are being targeted by a malicious denial of service attack.

Until now, a problem with doubletree is that it has never been implemented into a system which is able to be used in the real world on a large scale. One of the contributions of this work is an integration of the doubletree algorithms into Internet topology mapping software (scamper) that is part of an existing project (Ark).

2.4 Improving Traceroute Accuracy

When using conventional traceroute to probe hosts on the Internet, there are a number of factors which can cause the outcome of a trace to vary from the actual, real-world path taken through the network. In order to improve the accuracy of traces, these situations need to be taken into consideration and techniques employed to counter their effects.

2.4.1 Alias Resolution

Routers have more than one interface connected to the Internet, as by definition, to forward packets, a router must be connected to more than one network. Each interface has an IP address assigned to it. This causes problems when discovering paths with traceroute, because traceroute only discovers interfaces. In other words, maps derived from traceroute data may contain more nodes than actually exist, as the mapping process assumes that each unique IP address represents a router in the path. Alias resolution is the process of folding the multiple IP addresses of a router into a single node in the map. Figure 2.3 shows how traceroute can infer a path which has more routers than actually exist, but once alias resolution is performed, interfaces can be assigned to the router topology.

(22)

C D E F G H src1 A B src2

(a) before alias resolution

E F G H src1 A B src2 C,D

(b) after alias resolution

Figure 2.3: Alias resolution example. C and D are both interfaces on a single router. What appears to be two distinct networks before alias resolution turns out to be one interconnected network.

src D F G B C E A x,y x1,x2,y1,y2 x1,y2 x2,y1 x2,y1,x1,y2

Figure 2.4: IPID usage. The packets identified by IPIDs x and y are fragmented by the router at A into x1, x2, y1 and y2. The load balancing router at B then forwards the fragments out links which have different speeds causing the fragmented packets to arrive at G out of order. The IPID allows the fragments to be uniquely identified and reassembled correctly into the original packets.

Alias Resolution Methods

There are several methods for performing alias resolution. One such method relies on routers using the IP address of the outgoing interface as the source address for ICMP port-unreachable messages [17]. This means that when two potential aliases are determined, packets can be sent to each of them, and if the source address of the replies are the same, then an alias has been found. This method is problematic as it assumes that there is only a single dominant route from all of the router’s interfaces to a given destination.

(23)

Another alias resolution method (as used in Ally) [20], takes advantage of how the Internet Protocol Identifier (IPID) counter is implemented in routers. The IPID field is used in IP packets to uniquely identify a packet for re-assembly as illustrated in Figure 2.4. Most router implementations simply increment the IPID field for each packet they create. Using this knowledge, when a pair of potential alias addresses are identified, packets can be sent to both and by comparing the IPIDs of the packets received back from each. If the IPIDs received back are sequential (or close to each other) and in the correct order, it can be assumed that provided there is some variation in the sequence of reply addresses, the two IP addresses are in fact aliases for the same router. This method is fairly reliable, provided that routers increment the IPID counter in a sequential manner. Some vendors, the Linux kernel for example, set the IPID of all packets to zero. This makes it impossible to determine whether the addresses are aliases or simply routers which do not follow the convention of incrementing the IPID.

One other method for resolving aliases is to perform some parsing of the Domain Name Service (DNS) names assigned to a router interface [19] to extract information about the router that they belong to. This method takes advantage of the tendency for ISPs to name their routers in a way which identifies where they are located geographically within the network. For example, sl-bb21-lon-14-0.sprintlink.net and sl-bb21-lon-8-0.sprintlink.net are aliases for the same backbone router [19]. This technique has limited applicability in the doubletree system as the software needs to be trained for each ISP’s naming system. Also, some ISPs do not use a systematic naming system, or simply do not even name their routers within the DNS.

Another method suggested by Spring Te al. [19] is to use the maps generated from the traceroutes and two inference rules which allow alias addresses to be folded together. The first rule is that two adjacent addresses in the map are likely to represent adjacent routers rather than the same router. This is because if these addresses were in fact aliases, there would be a routing loop since one alias is forwarding to another alias on the same router. The other

(24)

rule specifies that IP addresses immediately before a point where links merge are likely to aliases if the links are point-to-point, as there would not be one address connected to two different routers with a point-to-point link.

Project Applicability

The inference rule method could be useful in the doubletree system; however it would need to implemented as a post-tracing script due to it needing a complete map to work. The advantage to this method over the Ally IPID method is that it requires no additional probing of the network. This is in keeping with our effort to reduce the load on the networks being probed.

It would be fairly straightforward to implement the Ally IPID method in a doubletree system. Due to the nature of the processing performed to resolve aliases, it is not necessary for this process to be carried out at the same time as the actual probing and so could be implemented as an extra program which processes the output from the doubletree system. However, there may be some advantage to having aliases resolved in semi-real-time so that other monitors could take advantage of knowing other interfaces that a router has. This would allow probing to be stopped when a router visited previously is encountered rather than just interfaces previously seen. There has been some work done with regard to implementing Ally within the same framework as the other work that this report presents; this is also discussed later.

2.4.2 Paris Traceroute

As links in the Internet become more plentiful, network administrators are em-ploying load balancing techniques to enhance reliability and improve utilisation of the available infrastructure. There are three classes of load balancing. These are: per flow, per destination and per packet. Per flow load balancing works by attempting to forward packets which belong to a specific flow or connection out the same interface. A flow of packets is usually identified by a combina-tion of the values in the IP header, specifically the source and destinacombina-tion ports

(25)

and addresses and the protocol. It has been found that routers may use any combination of these fields to determine a flow identifier [1].

Per destination load balancing is similar to per flow load balancing except that the only header field which is taken into consideration when forwarding packets is the destination address.

Per packet load balancing is by far the least common method which works by simply sending packets over the first available link. The reason that per packet load balancing is so rare is because the different links likely have different bandwidths and so there is the very real risk that packets will arrive at the destination out of order.

In terms of mapping, per-destination balancing is the same as classic routing and so does not need to be dealt with. Of the other two, the only (and indeed most common) situation that can be addressed in terms of traceroute, is per-flow. In order to reliably probe past per-flow balancing routers, the traceroute packets must be manipulated in such a way that the router treats them all as being part of the same flow. This must be done in concert with the other re-quirement of a traceroute packet, that, its sequence in the trace can be identified once it is returned, so that the traceroute software can match the response with a probe that has been sent. As Augustin et al. [1] found, routers use a com-bination of the standard flow identifiers: source address, destination address, source port, destination port and protocol. They also found that routers can also use the IP Type Of Service (TOS), ICMP code and checksum fields to identify flows. This causes a problem for traceroute as traceroute traditionally varied the destination port field so that it could match responses with probes. In order to overcome this problem, the authors found that traceroute needs to use a field that is in the first eight octets of the header (so that ICMP in-cludes it in the return packet) but is not used for load balancing to uniquely identify probes. For User Datagram Protocol (UDP) probes, the checksum is manipulated. For ICMP probes, Paris traceroute varies the sequence field, as does standard traceroute, but it also varies the identifier field so as to keep the checksum constant.

(26)

Project Applicability

These improvements to the classic traceroute algorithm can be easily made so that the doubletree algorithm makes use of Paris traceroute. In fact, it is even more vital for doubletree to make use of Paris traceroute as doubletree makes assumptions about the path being constant between traces to allow it to stop probing at a previously encountered node. If paris is not used, then load balancing can cause a different path to be seen with later probing.

2.5 Improving Traceroute Completeness

As mentioned earlier, there can be several paths between two points in the Internet. Paris traceroute addresses this issue by ensuring that each trace only takes a single path to the destination. It has been found that there are a large number of load balanced links in the Internet [2] so this issue needs to be considered seriously. Using the Paris traceroute algorithm means that there is the potential for detail to be missed as the alternate paths through the network may never be encountered. To ensure that the maps collected are as accurate as possible, modifications and/or extensions need to be made to the doubletree algorithm to allow it to traverse all possible paths. There are two existing improvements in the area of discovering a more complete topology; these are load balancer traceroute and windowed doubletree.

2.5.1 Load Balancer Traceroute

The authors of the Paris traceroute algorithm proposed an algorithm which extends Paris traceroute to find the complete set of load balanced paths between a source and destination to a specified confidence level [3]. The algorithm works by taking advantage of how per flow load balancing works. By varying the flow identifier of the probe packets being sent in a methodical way, it is possible to discover all of the interfaces which are a given distance into the network. It still probes hop by hop as with conventional traceroute, but at each hop, the

(27)

D E B A src G F C TTL 3 TTL 4 16 probes >31 probes >61 probes TTL 1 TTL 2 8 probes

Figure 2.5: Cumulative probe totals at each TTL with Load Balancer Tracer-oute. The number of probes grows in proportion to the number of load balanced paths towards the destination.

algorithm probes for the next hop multiple times (according to the confidence level required). This discovers a set of interfaces to which the current hop is connected. Then, by keeping state about which path a particular flow takes, each of the new hops are probed and so on until the destination is reached.

The major drawback to load balancer traceroute is that in order to discover all of the topology, it must send a large number of probes into the network. That is, even if there is only one interface at a given TTL, that is, no load balanced links at a hop, it still sends eight probes to be sure that there are no others to 99% confidence. Figure 2.5 shows the cumulative number of probes that need to be sent to be confident that there are no other interfaces which have not been discovered in a simple scenario. Each time there is more than one interface discovered at any hop, the number of probes that must be sent to rule out further interfaces increases; the total number of probes sent grows to a large number quickly, even on a short path.

Due to the nature of how load balancer traceroute works, it is a non-trivial task to integrate it with the doubletree algorithm. Several issues must be ad-dressed such as choosing an appropriate place to start probing from. These problems and potential solutions are presented in the next chapter.

(28)

2.5.2 Windowed Doubletree

The authors of the original doubletree algorithm have also done some work in improving the algorithm so that it is able to map a more complete topology [8]. The windowed doubletree algorithm assumes that most of path diversity occurs in the middle of the Internet and so the algorithm randomly varies the starting hop along the path within a predetermined window. Rather than beginning probing at a constant hop h, each monitor randomly picks a value for h within a window which represents the area in which most of the missing links are located. The authors claim that this simple change to doubletree discovers up to 16% more links than standard doubletree in their simulations. It should be noted however that this technique has never been tested outside of simulation. Also, the simulation data that it was tested on was collected using non-paris traceroute techniques which infer incorrect topologies.

Although windowed doubletree makes a significant improvement to the cov-erage of doubletree for no extra cost, it still does not infer all possible links in a systematic and reliable way. It is still a best-effort approach which cannot provide a statistical confidence that all the topology has been seen.

2.6 Summary

While there are a plethora of existing techniques which can be used for Internet topology discovery, they each have their own short comings; whether that be only discovering a subset of the actual topology or discovering the complete topology in an inefficient manner. To address this issue, we present in the next chapter, a new technique, doubletree load balancer which aims to give confidence that the complete topology has been discovered whilst probing in an intelligent and economical manner.

(29)

Chapter 3 Doubletree Load Balancer

Traceroute

Because the load-balancer traceroute algorithm in its original state is inefficient both in terms of run time and network load, there is a need for an alternative to be developed which reliably discovers all of the topology whilst being more efficient. One such algorithm has been designed and tested and is outlined in this chapter.

3.1 Overview

From a high level perspective, the doubletree load balancer algorithm provides a way to determine the best TTL at which to start the standard load balancer algorithm so that it does not unnecessarily visit links that it has seen before. It also provides a way to stop at the earliest point from which the rest of the path has been seen before.

In order to reliably determine a suitable starting point for probing, the dou-bletree load balancer algorithm begins by probing backward from a predeter-mined TTL, maintaining a constant flow identifier. This differs from doubletree which begins by probing forwards. Another point of difference from doubletree is that doubletree load balancer adds to the local stop set when probing forwards

(30)

rather than backward.

To begin with, doubletree load balancer sends a probe to the firsthop TTL which is provided as a parameter to the algorithm. This firsthop TTL must be selected carefully to ensure that probing is not started either too early in the path or too late. If probing is started too early, then large numbers of probes will be wasted rediscovering topology. Conversely, if the firsthop is set too high then there will be probes wasted to probe back to a known point. Once a response has been received from the first probe, the interface which sent the response is checked against the local stop set. If the interface is not in the local stop set, then the firsthop value is decremented by one and another probe is sent. Decrementing the firsthop by one effectively finds the hop in the path prior to the one just discovered. When the reply to this new probe is received, it is again checked against the local stop set. If it is not contained in the local stop set then the firsthop is again decremented by one. This process is continued until either an interface which is in the local stop set is found, or probing reaches the beginning of the path. If the backwards probing happens to get all the way back to beginning of the path without finding an interface which is in the local stop set, then the load balancer algorithm is used in its original form.

If an interface which is in the local stop set is encountered, the doubletree load balancer algorithm then sends another probe to the TTL at which the known interface was seen. The flow identifier for this probe is varied from the first to try and discover other interfaces which exist at this TTL. Several of these probes with varying flow identifiers are sent until the required number of replies have been received to be confident that all of the interfaces have been discovered at this TTL. This process is much the same as for the standard load balancer algorithm; as more interfaces are discovered, more probes must be sent. If there are no alternate interfaces discovered once the required number of probes has been sent, then the load balancer algorithm is started from that TTL. If there are other interfaces seen, doubletree load balancer moves into a convergence phase where it attempts to follow the paths from each of the interfaces discovered, backward until they converge to a single point. This

(31)

4,5 3,4 2,3 2 1 TTL TTL TTL TTL TTL src A C B D E F initial TTL=3

Figure 3.1: Asymmetric load balancing causing false links (shown as dashed lines) to be inferred by doubletree. If probing is started at TTL 3, there are a number of links that doubletree would infer which do not actually exist.

convergence point is where the load balancer algorithm is started from. To find the convergence point, doubletree load balancer probes backward from each of the interfaces discovered in much the same fashion that it did to find the initial interface. The path from each interface is traced back systematically until all of the paths converge to a hop with a single interface. This allows confidence that any flow identifier used to probe with will cause probe packets to traverse this interface.

Once the load balancer algorithm has been started, it runs as described in an earlier chapter. The only differences being that as it probes forwards, it adds the interfaces that it discovers to both the local stop set and the global stop set. It also checks each interface discovered to see if it is in the global stop set, if it is, then probing is halted as it is assumed that the rest of the path has been seen with previous probing, just like doubletree.

3.2 Challenges

Load balancing introduces several challenges which all have to be addressed so that the doubletree load balancer algorithm can function reliably and produce valid data.

One issue that has to be addressed is that when doubletree starts probing forwards midway into the path, it has no knowledge of the paths which come before that point. Therefore if there was asymmetric load balancing earlier

(32)

A B D C F E G start TTL

discovered interfaces: C, E, G missed interfaces: D, F LSS: A, B, C

Figure 3.2: Alternate path causing topology to be missed due to bad choice of start point. By probing forward using flow identifiers that reach C, the alternate path comprised of D and F is missed.

in the path, probing forward could cause invalid topology to be inferred as it would not be known which flow identifiers traverse which parts of the path. This problem is illustrated in Figure 3.1. If doubletree load balancer was to start probing at TTL 3, it would see both C and D and then at TTL 4 it would see both D and E. From these replies, doubletree would incorrectly infer the topology shown by the dashed lines in Figure 3.1. This is why doubletree load balancer begins by probing backwards using a single flow identifier. Probing backwards with a constant flow identifier allows us to have confidence that the links that are being inferred are valid.

Another challenge is determining how to decide on the start TTL for the load balancer algorithm. If we were to simply use the first interface that it encountered which was in the local stop set as the start point, then it is possible that there are alternate, parallel paths which have not been seen with previous probing. These alternate paths would either never be discovered or would in fact cause invalid links to be inferred, similarly to the example illustrated in Figure 3.1. Figure 3.2 shows an example where a parallel path causes topology to be missed. By probing the TTL where the first local stop set interface is found multiple times and then converging all the discovered interfaces, we can be confident that there are no other unseen alternate paths.

One important modification that had to be made to the doubletree algorithm was when to add to the local stop set. Doubletree adds to the local stop set

(33)

only while it is probing backward. The reason that this is relevant to doubletree load balancer is that if doubletree load balancer was to add to the local stop set whilst doing its initial backward probing then it could not be assumed that given an interface in the local stop set, the path from the monitor to it is known. It is this assumption is what allows doubletree load balancer to start by probing backwards rather than forwards. Because doubletree load balancer cannot add to the local stop set whilst probing backward, it must add therefore add to it whilst probing forwards. The reason for doubletree not adding to the local stop set whilst probing forwards is uncertain, as the authors have stated that the local stop set “consists of all interfaces already seen by that monitor” [8]. In fact, the lead author of doubletree cannot remember why they chose to only add to the local stop set whilst conducting backward probing [5]. The reason that it is safe for doubletree load balancer to add to the local stop set whilst probing forwards (although there is no evidence to support why doubletree does not do this also) is that when there is no state held by a monitor, that is, it is the first trace being carried out, doubletree load balancer simply sets the load balancer algorithm going from TTL 1, recording interfaces into the stop sets. Therefore there is no risk that the local stop set will be populated with interfaces to which a path from the monitor has not been seen. It is important to note that even though the global stop set and the local stop set are both populated with the same addresses at the same time, they are still different and cannot be combined into one. The local stop set is a set of origin, interface pairs, where the origin is always the same (the address of the monitor). This set is consulted with every new interface discovered to determine whether tracing can be stopped (a stopping condition). This look-up in the local stop set is made for all destinations. The global stop set is a set of interface, destination pairs where the destination is the address being traced when the associated interface is discovered. Only a subset of the global stop set is consulted for a stopping condition, that is, the set of entries where the destination address matching the address being probed. The global stop set is also shared among monitors.

(34)

A B C E F D G start TTL per packet load balancer

possible links inferred: F−>G, F−>F

Figure 3.3: Per packet load balancer causing invalid topology to be inferred. Because the load balancing at B is not based on the flow identifier, packets will arrive at F at both TTL 4 and 5 in an unpredictable fashion causing invalid links to be inferred.

B

D C

E

Figure 3.4: A per-packet load balancer (B) in a diamond configuration such that the per-packet load balancer does not have any effect on the links inferred. All possibly inferred links between B and E are valid.

and doubletree. As their behaviour can not be influenced, they have the poten-tial to cause doubletree load balancer to infer invalid topology. As Figure 3.3 illustrates, if doubletree load balancer settles on TTL 5 (F) as its start point and begins probing forwards, because of the per packet load balancer at B and the asymmetric paths afterward, doubletree load balancer can again infer in-valid links. In this case, F is seen at both TTL 5 and 6. This is because a flow identifier which cause packets to be routed from B to D to F causes F to be seen at a distance of five hops into the path, whereas if the flow identifier causes packets to be routed from B to C to E and then to F, F will be at a distance of six hops into the path. In order to minimize the possibility of this occurring, once we have determined a start TTL, it then sends multiple probes with the same flow identifier to that TTL. If there is more than one interface seen in the replies, we infer that there is a potential per packet load balancer earlier in the path and probing is halted for that destination as there is the

(35)

possibility of inferring invalid links. In practice however, this stop condition is probably far too conservative. It has been observed that of the approximately 1.4% of routers which using this technique appear to be per-packet load bal-ancers, around 50% of them are simply aliases for a single router which would not cause invalid data [16]. Also, of the cases which actually are per-packet load balancers, Augustin et al. [2] determined that most per packet load balancers are found in a diamond as illustrated in Figure 3.4. Because both of the links out of the per-packet load balancer re-converge immediately after, there is no risk of inferring invalid topology as all possible links must be valid.

(36)

Chapter 4 Implementation

Along with developing a new algorithm for topology discovery, this project has also contributed by implementing several different algorithms within an existing large-scale parallelised Internet measurement application.

4.1 Background

Almost all of the implementation work which is described here was done within the scamper [14] program. Scamper is a large-scale parallelised Internet mea-surement application which is already in use in Ark, the longest running Internet topology mapping project.

Scamper is organised as seen in Figure 4.1. There are three main ways that scamper can be run; with a destination list supplied on the command line, a destination list supplied in a file, and as a daemon. Both the doubletree load balancer and doubletree implementations described here can be run using any of these methods, however, for them to be part of a distributed system, scamper must be run as a daemon.

When scamper is run as a daemon, it binds to the loopback interface so that it is only accessible from the local machine for security reasons. Processes can connect to the scamper process and issue commands to carry out various measurements. The results of the measurement are sent back in the binary

(37)

control

queue

scamper

file cmdline control _source trace dtree tracelb ping

response probe traceroute arts file cmdline control socket task

operating system

* * _*

warts_* * *

*

Figure 4.1: Scamper block diagram. Components marked with an asterisk (*) have been modified by this project.

warts format. The process that occurs once a command has been received over the control socket is much the same as when running using a destination list.

When running scamper from a destination list, which is the method used in testing, the list is read in and parsed into a queue of addresses to be probed. The scamper control module takes care of ensuring that the tasks in the queue are executed at the right time so as to probe at a specified packets-per-second rate.

When a task is ready to be executed, the code which is specific to the type of tracing required is called. This could be trace (standard traceroute), tracelb (load balancer traceroute), dtree (doubletree) etc. This code takes care of form-ing the probe packets accordform-ing to the command specifications and instructform-ing the probe code to send them. Once a reply to a probe is received, the low level packet is parsed and the relevant details are used to construct a scamper reply object which is again passed to the appropriate method code. This looks at the reply and decides what action to take, whether that be sending more probes or writing out the results if probing is complete.

The results can be written out in a number of formats. The default writes out the results in an ASCII format similar to the unix traceroute tool. The default for a socket connection is the warts format which is a detailed binary

(38)

a b c d e f forwards mode backwards mode

Figure 4.2: Doubletree probing modes. The forwards probing mode is used to probe a, b and c. The backwards mode is used to probe d, e and f.

representation of the data collected from the trace by scamper. The warts format contains the level of detail necessary to do good science with the data.

4.2 Doubletree Implementation

In order to implement doubletree within scamper, a new measurement command type, dtree, was created. This fits into scamper as shown in Figure 4.1.

When a doubletree traceroute is started, state for the trace is set up and populated with the appropriate data and then a probe is sent with the TTL specified by the firsthop parameter. Scamper uses the concept of modes which allow probe replies to be handled by code specific to the type of probing currently underway. Doubletree has two modes which correspond to the two types of probing; forwards and backwards. These modes are illustrated in Figure 4.2, where, assuming that doubletree starts from TTL 4, the hops designated a, b and c are probed in the forwards mode and hops designated d, e and f are probed in the backwards probing mode.

To start with, doubletree is in the forwards probing mode. Once the first doubletree probe is received back, the handleicmp dtree fwd function is called. This function deals with all replies whilst doubletree is probing forwards. After performing sanity checks to ensure that this reply is one that is relevant, that is, not a late reply which could cause invalid links to be inferred, several checks

(39)

are performed to decide what to do next. A check to determine whether the destination has been reached is carried out as this means that there is nothing to be gained by probing further forward. A check to see whether the interface that the reply came from is in the global stop set is also carried out. If the reply interface is in the global stop set, this implies that a known branch has been encountered and therefore the rest of the path to the destination is known so probing need not continue. If these checks fail, then the TTL is incremented and another probe is sent. When the reply to the new probe is received, the same process is carried out. If either of the checks succeed however, the TTL is reset to one less than the firsthop value, the mode is changed to backwards probing and a new probe is sent. Because this probe was sent while in the backwards mode, the reply is handled by the handleicmp dtree back function. This function is very similar to the handleicmp dtree fwd function except that it does checking to determine if the reply is from the first hop in the path (TTL 1) instead of checking if it is the destination. If the reply is from the first hop in the path, similarly to finding the destination, there is nothing more to be discovered so probing is halted. It also checks whether the replying interface in the local, rather than the global, stop set and adds discovered interfaces to the local stop set instead of the global stop set. If the interface is in the local stop set, it can be assumed that the rest of the path back to the monitor is known and so probing is stopped.

Both the stop sets are implemented using splaytrees. Due to how each of the sets are formed from trees which branch out from a single point, there tends to be some nodes in the set which are accessed far more often than others. Splaytrees are useful in this situation because they have the property that recently accessed elements are quick to access again because when a node is accessed an operation is performed to move it to the root whilst still keeping the tree roughly balanced. So that doubletree can be used within a distributed system of monitors, a command parameter was added so that the global stop set tree could be populated externally. This allows the doubletree control mechanism to update the global stop set of each of the monitors as the shared set grows.

(40)

4.3 Doubletree Load Balancer Implementation

Because the doubletree load balancer algorithm makes use of what is essentially the original load balancer algorithm, it made more sense for the implementa-tion of doubletree load balancer to be done within the existing load balancer traceroute module. This was achieved by adding a parameter to load balancer traceroute which enables the doubletree load balancer parts of the code.

Once the doubletree load balancer algorithm is enabled, a check is done to determine whether there is any existing state in the stop sets. If there is no existing state, then there is nothing to gain by using the doubletree load balancer algorithm as it will inevitably probe all the way back from the firsthop specified to the beginning of the path before starting probing forwards. Therefore, when there is no existing state in the stop sets, the standard load balancer algorithm is started from TTL 1. The only difference is that it adds to the stop sets whilst probing forwards so that there is some existing state for the next trace. The stop sets used by doubletree load balancer are implemented in the same way as with doubletree.

4.3.1 Doubletree Load Balancer Modes

If there is state, then the doubletree load balancer algorithm is used to determine the best starting point for probing. In order to do this, several new modes were added to the load balancer traceroute module. These modes and the modes that they can transition into are listed in Table 4.1.

Findfirst Mode

The findfirst mode handles probing backwards until an interface is discovered which is in the local stop set. Encountering an interface in the local stop set gives us confidence that we have seen at least one path from this interface back to the monitor. When a reply to a probe is received whilst in this mode, the TTL of the corresponding probe is checked to ensure that it is greater than one. If not, then there is nothing to be gained by continuing backwards probing as

(41)

Mode Name Description Transitions To FINDFIRST Probe backward from

firsthop until a known hop is found then re-probe to find anchors

FIRSTADDR, CONVERGE, FINDFIRST CONVERGE Probe backward from

anchors to find convergence point

DTPERPACKET, FIRSTADDR,

CONVERGE DTPERPACKET Re-probe a hop to ensure

no per packet load balancing

FIRSTADDR, terminate trace

FIRSTADDR Probe for the first address in the path

(Existing)

other existing load balancer modes

Table 4.1: Doubletree load balancer mode descriptions and modes which they can transition into.

the start of the path has been reached. If the TTL is greater than one, a check is made to determine whether this reply came from the destination. If so, as this is the first probe, this means that the initial firsthop TTL value was set too high. In this case, the same heuristic that doubletree uses is applied; the firsthop value is halved and a new probe is sent.

Once these sanity checks have been passed, the replying interface is looked up in the local stop set. If it is not in the local stop set, then the firsthop value is decremented and another probe with the same flow identifier as the first is sent, thus allowing the previous hop in the path to be discovered. Because the local stop set represents the tree rooted at the monitor, the local stop set nodes will on average have low TTL values so by probing backward until an interface which is in the local stop set is found allows us to start at a point at the edge of the topology which has already been discovered. This process of probing backward is repeated until an interface which is in the local stop set is encountered.

At this point, the flow identifier is incremented and the TTL at which the probe was sent to is re-probed. This re-probing is repeated until the required number of probes have been sent for the required level of confidence as listed in Table 1.1. For each unique interface that is discovered at this TTL during this

(42)

D E F G TTL: 4 flowid: 1 address: D TTL: 4 flowid: 3 address: F TTL: 4 flowid: 7 address: E TTL: 4 flowid: 9 address: G anchor array src _TTL1 2 3 4

Figure 4.3: Doubletree load balancer initial anchor state. None of the topology below TTL 4 is known at this point but is shown here to give the anchors some context. Eventual convergence point will be at TTL 2.

phase of probing (including the original one), there is a structure created and added to an array. These interfaces are known as anchors. An anchor consists of the TTL of the anchor, the flow identifier used to reach it, a flag indicating if it is in the local stop set and the address of the interface. Once the required number of probes have been sent and all the anchors at the initial TTL are discovered, doubletree load balancer switches to a convergence mode. The state that doubletree load balancer is in at this point can be seen in Figure 4.3. None of the topology below shown below TTL 4 is known by doubletree load balancer however it is shown here to allow some context for the anchor examples.

Convergence Mode

Once the anchors have been identified, the mode is switched to converge. The job of the convergence mode is to trace the paths from each of the anchors back to a single point of convergence.

In the example given in Figure 4.3, the convergence point will be TTL 2 as it is the first point while looking backwards from the anchors, that the paths converge to a single interface. When a reply is received in convergence mode, the algorithm first updates the relevant anchor with the new TTL and interface which this reply has discovered. This essentially moves the anchor to the new interface discovered. The information discovered about the previous anchor

(43)

D E F G B TTL: 3 flowid: 1 address: B TTL: 4 flowid: 3 address: F TTL: 3 flowid: 7 address: B TTL: 4 flowid: 9 address: G

anchor array (before)

TTL: 3 flowid: 1 address: B TTL: 4 flowid: 3 address: F TTL: 4 flowid: 9 address: G

anchor array (after merge)

TTL1 2 3 4

0

1

2

3

0

1

2

src

Figure 4.4: Merging two anchors. Anchors at D and E have both been brought back to B. Anchor 0 and anchor 1 are merged to give the new anchor 0 shown.

point is discarded as it is only useful in the context of anchor discovery and cannot be used for the actual trace.

The anchors are then sorted in descending order based on their TTLs. This allows the anchors to be brought back evenly as the anchor with the largest TTL is always brought back first. After this, code is executed which attempts to collapse any anchors which have already converged. Two anchors are deemed as having converged if they both have the same address. Because the two anchors that are being collapsed will have different flow identifiers a decision had to be made about which flow identifier (or both) to keep when merging them. It was decided that because both flow identifiers have proved to bring probes to the same point in the path, it was not necessary to keep both. Therefore, the procedure for merging two anchors is just simply to delete the anchor with the largest TTL as the goal is to bring back the anchor with the largest TTL. It is important to note that there may be the case where two anchors have the same address but differing TTL values. This is due to asymmetric load balancing earlier in the path. However, they can be safely merged as both flow identifiers are known to get probes to the same interface. Continuing with the example given in Figure 4.3, if we assume that the anchors have been brought back so that the anchors at D and E are now both at B as shown in Figure 4.4, the merging code will combine both of the anchors at B into a single anchor with

(44)

TTL 3.

The convergence process completes when one anchor point is left. If there is more than one anchor left, then the anchor with the highest TTL, that is, the first anchor in the array is selected for probing. This anchor is then brought back by decrementing the anchor’s TTL and then sending a probe with the flow identifier set to that which is stored in the anchor. Picking the anchor with the largest TTL ensures that the anchors are brought back evenly thus maximising the chance of finding a convergence point with the greatest possible TTL. Using the flow identifier that has been stored for the anchor gives us confidence that new probes will take the same path as the original probe which discovered this anchor.

Per Packet Mode

After the convergence point has been identified, the last mode that doubletree load balancer moves into before beginning tracing forwards is to detect per-packet load balancers earlier in the path. If there is a per-per-packet load balancer in the path that leads up to the convergence point, it could cause the links which are inferred whilst conducting forwards probing to be invalid as we cannot be sure which path packets are taking. The check for per-packet load balancers is relatively na¨ıve as it can incorrectly classify routers with multiple aliases as per-packet load balancers because a router with more than one return path to the monitor will send replies with more than one interface which, to this test, will appear to be a per-packet load balancer. The per-packet checking mode works by repeatedly sending probes to the TTL in question, all with the same flow identifier. If any of the responses received are different to the interface stored in the anchor structure, then it is assumed that there is per-packet load balancing earlier in the path and tracing is halted. Provided that all of the replies have identical interfaces, then it is safe to begin forwards probing.

(45)

Forwards Probing Modes

Forwards probing is where all the data about the topology is collected. To begin forwards probing, all the state that the load balancer traceroute algorithm uses is reset and the firsthop is set to the TTL of the last remaining anchor. After which, the mode is changed to firstaddr, which is the mode that load balancer traceroute usually starts in and then the regular load balancer traceroute algo-rithm takes over. The only modifications to the load balancer traceroute code after this point is to add the interfaces that are discovered to both of the stopsets and also to add logic which stops probing when an interface is encountered that is already in the global stop set.

4.4 Distributed Measurement Control Server

Both doubletree load balancer and doubletree can be run by running scamper as a stand-alone application. However this does not fully take advantage of the way that the doubletree algorithm works. By running several different monitors in parallel, each monitor can share their global stop sets and therefore reduce the total number of probes that need to be sent by stopping when the path to a destination has already been seen by another monitor. In order to have several doubletree (or doubletree load balancer for that matter) monitors running, there needs to be a system which allows the monitors to communicate with each other to share destination lists and stop set state.

In order to test the implementations described above, a control server was developed which could co-ordinate a set of monitors. To allow the implementa-tion to be as simple as possible, it was decided to have the control mechanism be a centralised server which controlled both task allocation and address set distribution. As mentioned earlier, scamper can run as a daemon which allows external programs to issue commands and receive the results over a network socket. The downside to this system however, is that the socket is only avail-able to processes running on the same host as the scamper process. In order

(46)

31567 31569 31572 31570 control server ports local 10001 local port monitor 1 SSH port fwd 10002 local port monitor 2 10003 local port monitor 3 SSH port fwd 10004 local port monitor 4 SSH port fwd SSH port fwd

Figure 4.5: Example control server configuration layout. The local ports 31567, 31569, 31570 and 31572 on the control server are transparently forwarded over SSH to the local ports 10001, 10002, 10003 and 10004 on the respective monitors.

to get around this, the control server makes use of Secure Shell (SSH) port forwarding [10] to forward a port on the control server to a local port on the monitor. This allows the control server to connect to what appears to be a local port and have SSH tunnel the traffic to a local port on the monitor as illustrated in Figure 4.5.

The control server takes a list of addresses to probe and a list of ports which are tunnelled to scamper monitors as parameters. When the server is started, it establishes connections to all of the monitors and prepares them for tracing. It then reads in the address list and sets up some state regarding each address. If all of the monitors were instructed to probe the same address at the same time, the effectiveness of doubletree would be reduced as there would rarely ever be information in the global stop set about the address being probed. Therefore the server divides the list up into portions and assigns each to a monitor. Also, to ensure that a monitor does not probe an address which is being probed by another monitor, the address is locked when tracing begins and unlocked when it completes. This way, if a monitor finishes tracing all of the addresses in its window before the next monitor, it does not catch up as it has to wait until the

Efficient Methodical Internet Topology Discovery