GOV is a large government organization that provides networking solutions and data cen- ter infrastructure to several other government entities.
Challenges
To increase the availability of key applications, GOV’s IT department decided several years ago to implement a server cluster strategy. This strategy provided good application redundancy and scalability and significantly improved GOV’s capability to recover from server and operating system failures.
However, to benefit from new networking features, the implementations required cluster members to reside in the same network subnet. In addition, clusters relied on heartbeats that must run in a dedicated VLAN. To take advantage of current cluster technologies, GOV had to extend most VLANs within each data center.
ptg
Furthermore, GOV needed to improve its high-availability capabilities. In addition to han- dling server and operating system failures, clustering had to provide solutions for situa- tions such as partial data center power failures and site inaccessibility. Addressing these requirements meant extending clusters across multiple sites.
Like many other data centers, GOV’s data centers also began to encounter physical con- straints. Insufficient power, limited space, and inadequate cooling posed insolvable issues with server physical organization and operation, which led to GOV not even being able to install a new cluster member when application performance required it.
Solution
To address these issues, GOV determined that it required a solution that included a multi- site VLAN extension.
The initial solution was a Spanning Tree Protocol (STP) design that controlled four data centers in a global switching domain. GOV carefully followed best practices for L2 design, but the optical topology of the sites’ interconnection was unable to match stan- dard STP recommendations; dual hub-and-spoke topology and dense wavelength-division multiplexing (DWDM) protection are considered a must for STP. In addition, the size of the STP domain started to increase above any common implementation.
GOV operated its networks using this design for 1 year. During this time, several small failures occurred, which led to unpredicted results. In one instance, for example, a link failure did not report the loss of signal, leading STP to slow convergence. Every heartbeat over every data center timed out; consequently, all clusters experienced a split-brain situ- ation. Resynchronization took more than 1 hour to recover. During this time, all critical applications stopped operating. Other small failures had similar catastrophic effects. As a result, GOV contacted Cisco for recommendations about how to strengthen its network. Working in partnership with the GOV networking team, the server cluster vendor, and an external consulting team, Cisco recommended a VPLS solution as described in this book. The solution team also determined to provide Multiprotocol Label Switching (MPLS) fea- tures, such as Virtual Routing and Forwarding (VRF), to provide user-group security seg- mentation, and traffic engineering, to better manage link loads.
After thorough testing and a pilot phase, the solution was deployed in three GOV data centers. A fourth data center was added soon afterward, and the solution is running successfully.
Figure 6-1 illustrates the new GOV solution.
To build the L3-VPN network and the 10-Gbps MPLS core, GOV selected a Cisco Catalyst 6500 switch with a 67xx line card. This approach allows the easy deployment of VRF within the aggregation layer. L3-VPN extends to all data centers and toward user sites. To enable the MPLS Traffic-Engineering (TE) feature, the routing protocol had to be link- state-based, so the choice was reduced to either Open Shortest Path First (OSPF) or Intermediate System-to-Intermediate System (IS-IS) routing. In a network of this size,
ptg Toward Other Large Sites VFI VFI QinQ QinQ VRF VRF VRF VRF QinQ QinQ VFI VFI QinQ QinQ Cat 6K
- as Agg & L3-PE - AsP-Core
Cisco 7600
- as L2 N-PE on the Stick
L2 LAN Link VPLS Link MPLS Link VFI VFI VFI VFI VFI VFI VFI VFI VFI
VFI VFI VFI
VRF VRF VRF VRF VRF VRF VRF VRF VRF VRF VRF VRF QinQ QinQ VFI VFI VFI VFI
Figure 6-1 Global VPLS architecture.
IS-IS and OSPF offer quite similar capabilities, but IS-IS allows a clear demarcation with existing OSPF routing that simplifies deployment. GOV selected IS-IS as its MPLS core routing protocol.
Routing fast convergence is set with a target of a few hundred of milliseconds. Bidirectional Forwarding Detection (BFD) is used to detect long-distance link failure, which allows the system to react in approximately .5 seconds to any nonforwarding link. (GOV plans to include the MPLS Fast-ReRoute [FRR] function in future implementations, with the objective of achieving even more convergence on clear link failures.)
ptg
To implement the VLAN extension design using VPLS, the most advanced Network Provider Edge (N-PE) node was the Cisco 7600. Because Cisco 7600 Ethernet Service (ES) cards were not yet available at that time, GOV selected a SIP-600 card to provide 10 Gbps. (An ES card would be the right choice now.)
GOV selected Hierarchical-VPLS (H-VPLS) with Embedded Event Manager (EEM) scripts to provide STP isolation and long-distance link protection.
Four data centers required a VLAN extension to allow cluster extension. The solution included the Cisco 7600 N-PE on a stick. Figure 6-2 illustrates the concept of a “node on a stick.”
VPLS technology was quite new at the time of implementation. The “on a stick” design allowed GOV to avoid the insertion of new devices with the new Cisco IOS Software and new features along the existing L3 path. Because of this, VPLS failure would affect only L2 traffic, not IP traffic.
L2 traffic first passes in a bridge fashion through the aggregation Cisco Catalyst switch, and then is encapsulated in VPLS by the Cisco 7600 N-PE and pushed back to the Cisco Catalyst switch via a MPLS L3 port. Then traffic flows to the MPLS core.
The Cisco 7600 N-PE uses the 67xx LAN card toward the edge. Each ingress port is then encapsulated into a dual VLAN tag using the IEEE 802.1Q-in-Q VLAN tag (QinQ) fea- ture before being forwarded to VPLS. This QinQ encapsulation enables scalability to any number of VLANs. However, QinQ requires the careful management of overlapping
VRF VRF VFI VFI VFI VFI Local STP QinQ VPLS N-PE on the Stick Semaphores QinQ VRF VRF QinQ QinQ QinQ IP Traffic L2 Traffic MPLS Traffic
ptg
inter-VLAN MAC addressees. This issue is analyzed in depth in Chapter 11, “Additional Design Considerations.”
Enterprises should avoid extending network services such as firewalls or load balancers across data centers. In addition, good data center design uses different Hot Standby Router Protocol (HSRP) groups in each data center. These rules were implemented with GOV, where VLAN extension is strictly reserved for issues with multiple data center clus- ters and not used for other requirements.
In addition, LAN ports are protected from a data-plane storm using storm control for broadcast and multicast, which allows deployments to avoid the propagation of flood- ing across sites. This issue is also analyzed in depth in Chapter 11.
To enable N-PE backup, GOV deployed EEM scripting. The deployment did not include the Ethernet Virtual Circuit (EVC) feature because LAN port types do not allow it. VLAN load repartition is performed at the edge by using two 10-Gbps edge ports, with per-VLAN cost balancing.
To manage core load repartition over multiple paths, MPLS-TE was deployed, with each virtual forwarding instance (VFI) targeted to a different path.