OUT is an outsourcer that has deployed L2 DCI.
Challenges
Outsourcers such as OUT require the capability to easily perform add and move on request by individual servers. In addition, space, cooling, and power issues require the frequent reorganization of data centers. Flexibility is therefore a master word for out- sourcing services. With the arrival on the market of dynamic server virtualization (VMotion), outsourcers can now operate servers more dynamically and with increased flexibility. Flexibility to move servers requires VLAN extension, but outsourcers are well aware of the strong limitations of L2 bridging in terms of resiliency. VPLS is perceived as a solution that combines flexibility with high availability.
Solution
ptg
The OUT DCI network was designed to include ten sites interconnected through 10 Gbps over DWDM links.
Outsourcers have strong scalability requirements and must therefore deploy hardened and flexible solutions. Choosing the correct architecture is important. The best choice for N-PE redundancy will be available in the future and will use Inter-Chassis
Communication Protocol (ICCP) with multichassis Link Aggregation Control Protocol (mLACP). An EEM-based solution allows immediate deployment and is adaptable to any design topology.
OUT selected the Cisco 7600 with ES40 10-Gbps card to simultaneously perform H- VPLS, VRF, and core routing functions.
OUT chose H-VPLS with EEM for node redundancy. EEM offers a large panel of options detailed in Chapter 9, “EEM-Based Deployment Models.” OUT selected multiple options based on the site architecture:
■ EEM option 5 was selected for green-field data centers with Nexus virtual PortChannel (vPC) or with Virtual Switching System (VSS).
■ EEM option 4 was selected for existing data centers.
■ EEM option 3 was considered, but not yet deployed, for a very old-fashioned data center that does not support scripting into aggregation switches.
Wan VSS Aggr VSS Aggr Nexus 7000 Core Nexus 7000 Core D-WDM N*10GE WAN Core MPLS 7600+ES40" FrontEnd Services BackEnd Services Management Services VDC1 VDC2 VDC3 FrontEnd Services BackEnd Services Management Services VDC1 VDC2 VDC3
ptg
OUT considered connections to customer-owned data centers, mainly for migration pur- poses, but also for cluster extension. Because customer aggregation switches cannot be modified, it was decided to insert a physical small switch (User Provider Edge [U-PE]) per distant customer to perform the MAC-flush script on behalf; this is EEM option 4b, which is also described in this book.
One of the main issues comes from the multitenant aspect of the design. OUT offers its customers hosting for either applications or dedicated physical servers, and sometimes offers dedicated full DC zones. To create secure data centers, OUT selected the Nexus- based virtual data center (VDC) to ensure separation between front-end and back-end areas and to provide management-area isolation. A single VDC is shared between multiple customers, and connection to the 7600 N-PE uses one 10-Gbps port per VDC.
The initial plan was to create one VFI per VDC. This approach would have allowed easy scalability because N-PE would be transparent to the aggregation VLAN. However, shar- ing the same QinQ bridging domain between customers could result in the overlapping of MAC addresses.
While waiting for future MACinMAC encapsulation (802.1ah), which will be performed at ingress by an ES40 card, it has been decided to allocate one QinQ bridge domain per large customer, and one QinQ domain gathers smaller customers. To accomplish such a dynamic allocation, the Selective QinQ option will be intensively used. This feature allows OUT to identify a set of customer VLANs and encapsulate them in one core VLAN that will be transported using VPLS. Such dynamic encapsulation is a good com- promise between scalability and security.
OUT evaluated standard EEM options. However, to improve its convergence timing, OUT asked Cisco to tune scripts to allow active and standby pseudowires in the UP/UP condi- tion. (That is, the alternate path is up and ready but not forwarding traffic until backup.) This configuration is accomplished by tuning the EEM script that inserts and removes the bridge domain condition from the ingress port service instance. To avoid the need to modify the script each time a new customer is inserted, OUT created a set of preprovi- sioned VFIs.
Load repartition is a key element of the design. With load repartition, multiple parallel links are used to increase intersite throughput. With the Cisco 7600, a VFI is considered as a whole set, and its content is not balanced based on MAC addresses, whether the par- allel link uses equal-cost multipath (ECMP) routing or EtherChannel. Load repartition over a parallel links bundle is executed blindly on the last label of the stack, which is the pseudowire (PW) pointing to the remote VFI. Because granularity is per VFI, balancing is poor when the number of VFIs is low, and one link could easily be overloaded while oth- ers are almost empty. To avoid this situation, MPLS-TE was implemented to ensure con- trolled balancing for each VFI.
Figure 6-4 illustrates an efficient way to control balancing on a link bundle.
The approach illustrated in Figure 6-4 ensures that, in normal mode, traffic coming into an N-PE at 10 Gbps can find a 10-Gbps path toward the next hop, even in link-failure
ptg
conditions. Link-overload conditions could occur with complex traffic patterns or if a node fails, so standard DiffServ queuing is applied to protect key traffic. MPLS-TE is also offering measure of end-to-end traffic over the core, which provides a view of traffic pat- terns so that paths can be adjusted if needed.
Summary
VPLS with N-PE redundancy allows customers to flatten the bridging domain over multi- ple data centers by using the strongest technology currently available. This approach bene- fits both the server and application layers because it enables flexibility and availability for
■ High-availability extended server clusters
■ Virtual machines
■ Migration
This flexibility does inherently include some caveats, however:
■ The solution is complex and introduces constraints at the networking layer.
■ Extension of the broadcast domain might present some storm risk.
Balancing the risks and benefits between implementing VLAN extension and a lack of flexibility for servers led these organizations to select VPLS.
One Q-link per VDC multiple VFI per VDC.
Parallel TE tunnels one VDC balanced on each core link.
Balanced parallel TE tunnels for backup path on core path failure.
Multi links core
ptg