The Planet Proxy

Our planet proxy runs in user space on the PlanetLab nodes. Restricting our implemen- tation to user space eases deployment, but also prevents us from using fast kernel-level hooks to intercept network traffic [VYW+

02]. Thus, our planet proxy is subject to delays associated with the kernel’s scheduling policy of user-level tasks.

We implemented the planet proxy as a Linux daemon process in approximately 2,400 lines of C++ code.

Intercepting application traffic

When the proxy is started, it creates a virtual Ethernet device (a TAP device [KY99]) configured with a private subnet, such as 10.0.0.0/8. This private subnet is shared by all planet proxies and all application instances running on the planets. To run an application on SatelliteLab, the experimenter only needs to configure it to bind to the TAP device’s IP address. Thus, all traffic sent by the application is intercepted by its local planet proxy. When two application instances on the same planet exchange traffic, this traffic would normally be delivered directly by the kernel without being routed through the TAP de- vice. To avoid this, we use a simple technique borrowed from ModelNet [VYW+

02]: each application modifies certain bits in all destination addresses to ensure that they appear as remote addresses; once the planet proxy intercepts a packet, these bits are set back to their original value. This technique uses a dynamic library (libipaddr) and works with unmodified application binaries.

Communicating with the satellites

The proxy interacts with its satellites using a very simple UDP-based probe/response protocol. The proxy can send a d-byte UDP message PROBE(i,d,u), where i is an identifier

7.3 Implementation

and u is the size of the requested response packet. In response, the satellite sends a u- byte UDP message RESPONSE(i,u). To mask the occasional loss of packets, each probe (response) message includes information about the two probe (response) packets most recently exchanged between the proxy and the satellite. After intercepting a packet of application traffic, the proxy forwards the packet along the detour path using the above protocol on the edges and one extra message between the planets.

Let A and B be application instances running on planets PA and PB, respectively; let

SA and SB be the corresponding satellites, and let σ be the size of the data packet in

bytes. The complete packet forwarding process is as follows:

1. PA chooses an identifier i and stores the packet in an internal buffer indexed by i.

2. PA sends PROBE(i,12,σ) to SA, which responds with RESPONSE(i,σ).

3. PA retrieves the packet and forwards it to PB.

4. PB sends PROBE(i,σ,10) to SB, which responds with RESPONSE(i,10).

5. PB retrieves the packet and delivers it to B.

This forwarding process subjects application traffic to network conditions on the access links of the satellites. Since the access links are often the bottlenecks of the direct network path between the satellite nodes, the detour paths have similar access link delays, losses, and bandwidths as the direct paths.

If the probe packets are reordered or delayed, their corresponding data packets are also reordered or delayed. Similarly, if probe packets are lost, their corresponding data packets are not forwarded, and they are eventually dropped from the buffer by the proxy. Finally, the packet sizes of probes (and probe responses) match those of the data packets.

Compensating for packet loss

With the described protocol, SatelliteLab increases the packet loss rate on the access link compared to the direct communication between satellites. The reason is that two control packets – a probe and a response – are needed to emulate the transmission of a single data packet; the data packet is considered lost if either of the two control packets is lost.

In our experiments this problem occurred rarely, even on access links such as broadband as these control packets are small. Nevertheless, we developed an additional mechanism to mitigate the effects of this problem. SatelliteLab incorporates information about the control packets already sent in the padded data of subsequent packets. This enables quick detection and recovery of control packet loss. Thus, if a packet is lost in either direction, the recipient can obtain the missing information from one of the subsequent packets1

. Since this does not help if the last control packet in a sequence is lost, we send this packet twice, but only if the link is otherwise idle.

However, the planet only uses the information about the control packets already sent to compensate for losses that happen on the additional pathways introduced by SatelliteLab. When it emulates an upstream transmission, it compensates for lost probe packets; when it emulates a downstream transmission, it compensates for lost responses. Otherwise

1

Note that if the access link reorders packets, this can cause additional reordering. If this is an issue, the feature can be disabled.

SatelliteLab would overcompensate and push the loss rate below the access link’s actual loss rate.

Adjusting the MTU

When a packet is sent from an application instance on one planet to an instance on another, the proxies encapsulate the packet in a UDP datagram, which adds a 28-byte UDP/IP header and a four-byte SatelliteLab header. If the original packet was already MTU-sized, the resulting UDP datagram must be sent in two packets, which is undesirable because it increases the effective loss rate. Usually, it is possible to avoid this problem by reducing the MTU of the TAP device. However, in some testbeds (e.g., PlanetLab), the MTU cannot be changed. Therefore, we implemented a mechanism in the planet proxies that adds a MSS option to TCP SYN packets, which transparently reduces the packet size for TCP flows. Other transport protocols will still work, but their packets may be split on the planet-to-planet path and thus experience a higher loss rate.