The Spanning Tree Protocol (STP) is one of the simplest control planes available, and also probably one of the most widely deployed. Three specific points are important for the network designers to consider in using STP:
STP builds a single tree of the entire topology, rather than per destination/device. This implies that the topology must match the tree STP will build, or only those parts of the physical topology that match a spanning tree will be used by the protocol. Figure 5-1 provides an example.
Figure 5-1 Spanning Tree Inefficiency
While this topology is a loop, STP needs to build a single tree that can be used to forward traffic. Assuming Switch C is chosen as the root of the spanning tree, the link between Switches B and F will be blocked, so traffic from Server A to Host G is switched along the path [B,C,E,F]. This is clearly a much longer path than the optimal, which is [B,F].
All reachability (for both network nodes and end hosts) is learned in the data plane, rather than being carried through the network in the control plane. In the network in Figure 5-1, the spanning tree across the network is built node-to-node to include the links [B,C], [C,E], and [E,F]. This tree is enforced by shutting off all traffic along the [B,F] link, rather than actually building a local forwarding table based on reachability information learned from the control plane.
The implication for design is that mobility and loop-freeness are built on the back of data plane timers, rather than control plane
convergence speed. To increase the speed of convergence in an STP
network, the protocol is removed from edge links (such as [A,B]) by techniques such as the Cisco PortFast, timers are adjusted to produce faster detection of link and node failures, and faster link layer failure detection mechanisms are deployed, such as Ethernet OAM.
There are no loop breaking mechanisms in most data link layer protocol specifications, such as a time-to-live counter in the header; if a loop occurs, packets will loop until the control plane breaks the loop. If the looping packet interferes with the control plane’s capability to break the loop, it could be impossible to break the loop, causing a permanent failure condition (until a network operator intervenes and manually breaks the loop).
Some large scale data center networks have moved away from using STP by designing their physical topologies in the form of a spanning tree, and then using MAC address filtering or multiple spanning tree domains to prevent packet loops. One example of such designs is covered in Chapter 12,
“Building the Second Floor.”
TRILL
Transparent Interconnection of Lots of Links (TRILL) was originally conceived with a simple idea: what happens if we replace STP with a link state
protocol? A common misconception engineers often form from this simple basis is that TRILL routes Layer 2 frames much like any traditional routing protocol routes Layer 3 (IP) packets—but this isn’t really the idea behind TRILL, nor the way TRILL actually operates.
TRILL Operation
Figure 5-2 is used as a reference for explaining TRILL operation.
Figure 5-2 Example Network for TRILL Operation TRILL operation proceeds in three steps.
In the first step, a set of shortest path trees are built node to node (not edge to edge, as in routing), across all the switches in the network. IS-IS is used to find neighbors, advertise node-to-node links, and build the trees that provide connectivity across the switches. In this example, H would build a tree that includes reachability to the following:
Switch F along the path [H,F]
Switch C along the path [H,F,C]
Switch G along the path [H,G]
Switch E along the path [H,G,E]
Switch D along the path [H,G,E,D]
After this set of shortest path trees is built through the network, the second step, learning reachability, can begin. Unlike routing (and like STP),
reachability is learned through the data plane. If Server A forwards a packet toward Host K, Switch C now learns that Server A is reachable through interface C.1. Assuming Switch C doesn’t have any forwarding information for Host K, it will place this packet onto a multicast tree that reaches every other edge node in the TRILL domain. The packet isn’t transmitted “natively,”
but is encapsulated in a TRILL header (like Q-in-Q) that contains additional information, such as a TTL and information about the source switch in the TRILL domain. In this case, the header would contain Switch
C’s nickname, which is simply a unique identifier on the TRILL domain that can be used to forward packets directly to Switch C.
On receiving this packet from this multicast tree, each edge node will
examine the TRILL header and inner (original) packet and learn that Server A is reachable through Switch C. Again, assuming Switches D and H have no prior knowledge of Host K, they will both unwrap the TRILL encapsulation and flood this packet to an unknown destination onto their attached segments. Note that the “core switches,” or switches that are not on the edge, do not examine these packets, nor learn the MAC addresses carried in them. In this way, TRILL protects the core switches from learning the entire table of reachable Layer 2 addresses in the network.
In the third step, normal switching begins. Assume Host K responds; we can trace this response back through the process of being forwarded to Server A.
When Host K transmits a packet to Server A, Switch H receives this packet at interface H.1 and examines its local forwarding table for information about the destination address. Switch H finds it does have a forwarding entry;
Server A is reachable through Switch C. Switch H encapsulates the original packet into a TRILL header and forwards the packet to Switch F.
Switch F examines the TRILL header and finds the packet is destined to Switch C, so it simply forwards the packet on toward the correct destination.
When Switch C receives the packet, it looks at the TRILL header and the source address, and learns that Host K is reachable through Switch H. After this information is entered into its local forwarding table, Switch C removes the outer TRILL header and forwards the packet onto the correct local link based on the forwarding information built previously—forwarding information discovered when the first packet from Server A was transmitted.
TRILL in the Design Landscape
What are the practical implications of TRILL operation? The most obvious is that TRILL allows every link in the TRILL domain to be used, rather than just those links that fall along a single spanning tree for the entire physical
topology. This makes for more efficient link utilization. TRILL’s use of a link state protocol to discover topology also implies that all the positive and
negative aspects of a distributed link state control plane are brought into play in the TRILL domain.
Microloops are bound to be a part of any TRILL domain while it’s converging, just as they are in IS-IS and OSPF networks supporting IPv4 and IPv6.
Techniques used to provide fast reroute in link state based IP networks should also be applicable to TRILL domain, as well (although there are no standards in this space, nor has any vender expressed plans to implement fast reroute in TRILL based fabrics). Note the TRILL header does include a TTL to prevent either microloops or more permanent forwarding loops from looping individual packets until the control plane or manual intervention breaks the loop.
Although TRILL uses a link state protocol to discover the network topology, it does not use the link state protocol to discover reachability. This means that like STP, TRILL is reliant on data plane timers to discover when a device has moved from one place in the network to another; the current forwarding table information must time out in all nodes before devices can be assured their traffic is delivered to the correct end host in a TRILL network.
TRILL is also capable of providing control plane operations for large multitenant data centers supporting millions of virtual networks through various extensions and modifications to the base protocol.
TRILL and the Fabrics
TRILL and its close cousin, IEEE 802.1aq (Shortest Path Bridging), are both the foundation for a number of vendor fabric offerings, including
Cisco’s FabricPath. These offerings generally operate in a way that is similar enough to TRILL to be treated as a TRILL domain from a design perspective.
One major difference between these offerings and TRILL’s basic operation is that it’s possible to include reachability information in the IS-IS process that provides topology information for the TRILL domain. In the case of the
example given in Figure 5-2, Switch C, on discovering that Server A is locally attached, can send an IS-IS update including this bit of reachability
information, so the first packet directed at Server A doesn’t need to be multicast through the TRILL domain.
Note: Juniper’s Q Fabric is based on an MPLS overlay fabric (much like a Layer 2 VPN provided by a service provider), using BGP and route
reflectors. Q Fabric isn’t based on TRILL, though it can still be treated as a single fast flat Layer 2 domain with optimal routing for the purposes of network design.