TRILL—ADDING MULTI-PATHING TO LAYER 2 NETWORKS

The ever-increasing adoption of virtual environments in the data center necessitates a more resilient Layer 2 network. Efficient and reliable L2 infrastructure is needed to support the I/O demands of virtual applications, especially when applications are migrated across servers or even different data centers. Today’s STP-based networks limit the available network bandwidth and fail to maintain reliable complex networks architectures.

IETF is developing a new shortest path frame L2 routing protocol for multi-hop environments.

The new protocol is called Transparent Interconnection of Lots of Links, or TRILL, and it is expected to be completed in the second half of 2010. TRILL will enable multi-pathing for L2 networks and remove the restrictions placed on data center environments by STP (single path) networks. Data centers with converged networks will also benefit from the multi-hop capabilities of TRILL Routing Bridges (RBridges) as they also enable multi-hop FCoE solutions.

W

D

W

N

I

?

Resiliency in a network is always good. In the event of a failure in any one component, it’s important to have sufficient resources to allow the network to continue functioning around the problem areas. This is especially important in converged networks with storage and IP resources, where the interruption of service may result in the disastrous failure of one or more server platforms.

And to reap the greatest advantage from a network, an administrator would like to be able to use all of the available capacity of the network for moving traffic. If there are multiple paths of lowest equal cost through a network from Point A to Point B, in a best-case scenario all of those paths would be used. Unfortunately, the evolution of Ethernet has introduced restrictions that do not always allow for either the maximum resiliency or the highest capacity in a network.

Because Ethernet was originally a flat point-to-point topology across a single segment of shared media (see Figure 12), you didn’t need to be concerned about multiple paths through the network. Each node was logically connected directly to each of its peers with no intermediary devices along the way. That meant that the Ethernet protocol could ignore cases in which multiple paths from a source to a destination were available. As a result counters and headers for management metrics such as hop count and time-out values were unnecessary and were not included in the standard Ethernet frame.

Figure 12.

Basic Ethernet topology.

As Ethernet deployments became more common, new devices were introduced into the infrastructure to enable larger networks. Analog repeaters and digital bridges began to appear, and as they did, new complexities began to surface. For example, with these devices, you could design a network where there was more than one physical path from any one source to a destination (see Figure 13).

Figure 13.

More complexity in the Ethernet topology.

The problem was that a network device receiving a frame on a port didn’t know if it had seen that frame before. This introduced the possibility of a single frame circulating throughout the network indefinitely (see Figure 14).

Left unchecked, a network would soon be saturated with frames that couldn’t be removed because they couldn’t be identified or limited.

Figure 14.

Circulating indefinitely ….

To address this problem, a logical topological restriction in the form of a spanning tree was placed on Ethernet networks. The STP protocol meant that: although there may be many physical paths through the network at any given time, all traffic will flow along paths that have been defined by a spanning tree that includes all network devices and nodes (see Figure 15).

By restricting traffic to this tree, loops in the logical topology are prevented at the expense of blocking alternative network paths.

While STP solves the problem of traffic loops, it prevents network capacity from being fully used. Algorithms that calculate this spanning tree may take a lot of time to converge. During that time, the regular flow of traffic must be halted to prevent the type of network saturation described above. Even if multiple simultaneous spanning trees are used for separate VLANs to better distribute the traffic, traffic in any one VLAN will still suffer from the same disadvantage of not being able to use all of the available capacity in the network.

I

TRILL

To eliminate the restriction of a single path through the network, the IETF formed a working group to study this problem. The official documentation states the goal of the group this way: “The TRILL WG will design a solution for shortest-path frame routing in multi-hop IEEE 802.1-compliant Ethernet networks with arbitrary topologies, using an existing link-state routing protocol technology.”

In simpler terms, the group was charged with developing a solution that:

• Uses shortest path routing

• Works at Layer 2

• Supports multi-hopping environments

• Works with an arbitrary topology

• Uses an existing link-state routing protocol

• Remains compatible with IEEE 802.1 Ethernet networks that use STP The result was a protocol called TRILL. Although routing is ordinarily done at Layer 3 of the ISO protocol stack, by making Layer 2 a routing layer, protocols other than IP, such as FCoE, can take advantage of this increased functionality. Multi-hopping allows specifying multiple paths through the network. By working in an arbitrary topology, links that otherwise would have been blocked are usable for traffic. Finally, if the network can use an existing link-state protocol, solution providers can use protocols that have

already been developed, hardened, and optimized. This reduces the amount of work that must be done to deploy TRILL.

Figure 16.

TRILL provides L2 multi-pathing.

Just as important is what TRILL doesn’t do. Although TRILL can serve as an alternative to STP, it doesn’t require that STP be removed from an Ethernet infrastructure. Most networking administrators can’t just “rip and replace”

their current deployments simply for the sake of implementing TRILL. So hybrid solutions that use both STP and TRILL are not only possible but will most likely be the norm at least in the near term. TRILL will also not automatically eliminate the risk of a single point of failure, especially in a hybrid architecture. The goals of TRILL are restricted to those explicitly listed above. Some unrealistic expectations and misrepresentations have

Core

Aggregation

Access

Servers

T

TRILL P

Like other protocols, TRILL solutions will have three components; a data plane, a control plane, and devices that implement the protocol.

TRILL Encapsulation

TRILL encapsulation turns Ethernet frames into TRILL frames by adding a TRILL header to the frame. The new TRILL header (see Figure 17) is in exactly the same format as a legacy Ethernet header. This allows bridges (switches) that are not aware of TRILL to continue forwarding frames according to the rules they’ve always used. The source address used in the TRILL header is the address of the RBridge adding the header. The destination address is determined by consulting tables built by the link-state routing protocol. A new EtherType is assigned to TRILL. Note also the HC (hop count) field, a 6-bit field that allows for 64 hops. The HC field is used to prevent the formation of loops on the VLAN or the premature discarding of frames.

Figure 17.

Anatomy of the 64-bit TRILL header

Etype = TRILL (TBA)

•

•

•

Link-State Protocols

As noted earlier, TRILL will use link-state protocols to form the control plane of TRILL. The purpose of the control plane is to distribute the VLAN configuration to all the RBridges on the VLAN. Link-state protocols also continuously monitor the VLAN configuration and adjust the configuration data base in the event of changes. The control plane also provides the algorithms used to calculate the shortest path between any two RBridges on the VLAN. Considering the fact that TRILL will be used in converged environments where storage and TCP/IP networks are deployed, you can expect that link-state protocols from both worlds will be utilized by TRILL Routing Bridges

Routing bridges are a new type of L2 devices that implement the TRILL protocol, perform L2 forwarding, and require little or no configurations.

Using the configuration information distributed by the link-state protocol, RBridges discover each other and calculate the shortest path to all other RBridges on the VLAN. The combination of all calculated shortest paths make up the RBridge routing table. It is important to note that all RBridges maintain a copy of the configuration database, which helps reduce convergence time. When they discover each other, RBridges select a designated bridge (DRB), which in turn assigns a designation for the VLAN and selects an appointed forwarder (AF) for the VLAN. Although the DRB can select itself as an AF, there can only be a single AF per VLAN. The AF handles native frames on the VLAN.

M

TRILL D

TRILL works by adding a new header to the beginning of an Ethernet frame.

Using the original destination address as a key, a list of eligible next-hop RBridges is determined. This list contains the RBridges that could be the next step along all least-cost paths moving to the final destination. If more than one RBridge is in the list, a hash is used to distribute the traffic load and guarantee that all traffic in a single stream stays on the same path to avoid reordering overhead. The RBridge that results from this is placed in the TRILL header and the frame is sent on.

Figure 18.

TRILL adds a new header to the beginning of an Ethernet frame.

The new TRILL header (see Figure 18) is in exactly the same format as a legacy Ethernet header. This allows bridges (switches) that are not aware of TRILL to continue forwarding frames according to the rules they’ve always used. The source address used in the TRILL header is the address of the RBridge adding the header. The destination address is determined by consulting the tables built by the link-state routing protocol.

RB4

When a frame with a TRILL header is received by an RBridge, the RBridge removes the header and examines the original source and destination addresses. It then creates a new TRILL header using the method described above and forwards the frame. The last RBridge receiving a frame prior to the delivery of the frame to either the destination or the local segment that connects to the destination removes the TRILL header and forwards the frame (see Figure 19).

Figure 19.

TRILL header added and then removed at the last RBridge.

S

RB4 RB5

RB1

RB Encapsulation^TRILL RB B

(AF VLAN 1) RB2 (AF VLAN 2)

Host Target

RB3