Design considerations

Since we are not aware of any similar work in the data center networking context, we briefly consider the high-level approaches that one might take before discussing our approach to resequencing. In order for packets to be resequenced at servers, they must be marked in some way to identify their order. This can be done in one of two ways; with sequence numbers or with timestamps. We describe the implications of each approach in the data center context below.

Sequence-based resequencing:

Using sequence numbers would mean that a server creates a sequence number counter for each server that it is sends packets to. When it sends a packet to a given destina-tion, it would add the sequence number from the corresponding counter to the packet and then increment the counter. The receiver can then use the sequence number to determine which packet should come next and buffer any packets that arrive out-of-order. Since a missing packet may never arrive, the receiver must use a timeout to recover from loss. This is essentially the approach that TCP uses except that its sequence numbers identify the order of bytes within a flow. The key advantage of this approach over time-based resequencing is that a receiver can immediately determine

whether any packets are missing when it receives a packet. This means that packets arriving in-order can be delivered immediately and experience no added delay due to resequencing.

Per-flow state

One potential drawback of this approach is that it requires each server to maintain separate resequencing state for each server it is communicating with. At the sending side, it requires a separate sequence counter for each receiver and at the receiving side a separate queue to reorder the packets from each server sending to it. Given that data centers contain many servers which may be assigned to different tenants over time, this state would need to be managed dynamically. This may make it more difficult to implement in hardware (e.g. as a NIC feature) but can be managed in software. Since the resequencing we propose is meant to exist transparently below any network or transport layer protocol, we cannot know in advance when communication with another server begins or ends. As a consequence we would need to depend on a soft-state approach and use timeouts to remove resequencing state when servers have ceased communicating.

Sequence number agreement

This raises another practical issue which is that the sender and receiver must agree on what the initial sequence number is before they communicate. The use of timeouts means that a simple approach like assuming the initial sequence number is 0 won’t work since we cant be sure whether both servers have timed out and removed their state when they resume communicating. In fact there is no way to guarantee sequence number agreement without explicit two-way communication since any flag or special packet the sender might use could be lost. For TCP, this issue is handled as part of the two-way hand shake that occurs at the start of a flow. In the resequencing layer we cannot know a priori when communication with another server begins or ends which means we cannot perform such a handshake in advance. While this is a minor issue, we would have to accept that some traffic might initially be delayed or delivered out-of-order.

No Multicast

A final drawback to this approach is that it cannot easily be extended to handle

multicast. This is because sequence numbers only have meaning between two servers.

Separate state would be needed for each multicast sender in every multicast group.

Time-based resequencing:

The alternative to using sequence numbers is to mark packets with a timestamp.

This approach has been used in router interconnects where the inability to support multicast and the need to maintain separate resequencing state at ports is more problematic. In the data center, this method requires that servers mark each packet with a timestamp indicating when the packet was sent. An advantage of this approach is that each server only needs to maintain one resequencing buffer since all incoming packets can be reordered based on their timestamp. The main source of difficulty is that timestamps only indicate relative order. A receiver has no way of knowing whether packets with earlier timestamps may still arrive. The only solution is to establish some age threshold after which packets are considered “late”. To avoid out-of-order delivery, the age threshold must be large enough to accommodate the maximum delay between paths that is normally experienced.

Unnecessary delay

Thus a drawback of this approach is that the resequencer must delay every packet by the age threshold which effectively means that each packet experiences the maximum amount of network delay. To minimize this penalty, the age threshold can be made to adjust adaptively to changing network conditions [55]. However, in the data cen-ter context, servers in different subtrees have longer paths than servers in the same subtree. Since the buffer does not separate the traffic from different servers, the age threshold would need to be set to the delay over all paths leading to unnecessary delay for traffic between local servers.

Clock synchronization

The second and more practical concern is that for this approach to work, server clocks would need to be tightly synchronized. While it is not important to keep packets from two different servers in order, the packets from both servers must be buffered until they reach the age threshold and the only way the receiver can determine their age is by relying on their timestamps. This means that when they are serviced from the

same queue, the only way to keep the packets from both servers long enough is to increase the age threshold by the difference in their clocks. Since this adds directly to the overall delay, the performance can never be better than the degree to which synchronization can be achieved between servers.