Position Paper The new data center edge Horizontal Stacking and Switch Clustering

Position Paper

The new data center edge — Horizontal Stacking and Switch Clustering

The enterprise data center is one of the most critical areas of the network. This is especially true when consolidating applications and services into a single or geo-redundant data center construct. The challenge to reduce overall costs is realized as servers, applications and network services are moved into central-ized data centers. This consolidation of critical resources places a greater burden on network availability and performance.

24 x 7 access to applications and data stores is a common requirement in most enterprises. Any disruption in network service within the data center has reper-cussions throughout the business. These effects are different in nature, scope and severity depending on the type of enter-prise, but the effects are all negative and likely impact the enterprise’s revenue. The availability of applications begins with the availability of the network.

The ability to nearly eliminate both planned and unplanned network downtime is an important factor when designing the next-generation data center. The need for continuous availability makes it nearly impossible to schedule any outages for network upgrades or routine maintenance. A data center architecture that provides the capability to schedule maintenance without inter-rupting the availability of the data center network gives the network administrator increased flexibility to maintain the network.

The other major factor for design consideration is Total Cost of Ownership (TCO). A data center solution must be cost-effective without sacrificing perfor-mance or resiliency while still paying attention to the need to be as eco-friendly (Green IT) as possible. TCO is often measured in two areas — capital tures (CAPEX) and operational expendi-tures (OPEX).

requirements for maintenance contracts and the energy costs associated with operating the equipment. The ongoing operation and management of the network are OPEX costs that aren’t as easily visible. If the network is simple to operate and manage, then the amount of time network administrators spend on these tasks is reduced, thus freeing them up to do other work and increasing their overall productivity.

The Server Access Layer evolution

The data center Server Access Layer continues to evolve. This is a direct result of the types and numbers of servers being deployed in the enterprise data centers. When there were fewer physical servers, it made sense to consolidate the Server Access Layer into a few strategic locations within the data center and run cabling out to the servers. This allowed for the best usage of ports on the network equipment and therefore the most economical solution. As the trend moved away from large servers to single purpose 1U rack-mounted devices, the amount of network connections increased dramatically. Many of these servers came equipped with multiple Network Interface Cards (NICs), and server administra-tors were taking advantage of multiple network connections. As an example, a common practice separates the produc-tion traffic from the data backup traffic across different network connections. This increased number of connections required many more cable runs from the servers to the network equipment. This initiated the trend to distribute the network equipment out to the server racks as opposed to running all those cables to a centralized network rack.

Today, as the Ethernet switches are distributed to the server racks, it has become common practice to dual home each switch back to the data center Distribution Layer or Core Layer. The availability requirements mandate a solution that supports the best utilization of bandwidth with sub-second failover yet without adding more complexity. This is the ultimate resiliency model of Nortel’s Switch Clustering technology discussed later. Data centers continue to expand due to consolidation efforts and increases in the sheer number of applications deployed within the enterprise. This increases the number of server racks and the number of Ethernet switches deployed within the Server Access Layer. A significantly increased number of ports are required in the Distribution/ Core to collect all the Server Access Switch connections. In an effort to curb this sometimes uncontrolled growth and to better utilize compute resources, application virtualization and Blade Center technology have become increas-ingly popular. This is definitely a step in the right direction; however, it is not the total answer.

At the same time, as this increase in the number of servers is happening, so is the need for high-speed, low-latency connectivity between servers. Many applications and services require extensive communications between the web, application and data base tiers in order to service end-user requests. These applications may be latency-sensitive and intolerable to any network delay. High-speed computing and grid computing environments also require ultra-fast server-to-server connectivity. A better solution for inter-server connectivity is required as millisecond latency between servers is no longer sufficient; it must be reduced to micro-seconds. This can be accomplished with Nortel’s hardware-based forwarding and stacking technology as described next.

Figure 1. Each server rack with two uplinks to the core (Gigabit or 10 Gigabit)

Stacking technology

Nortel introduced one of the industry’s first true resilient stacking technolo-gies in 1998 with the BayStack 450 Ethernet Switch. This resilient stacking architecture allowed a stack to be managed as a single entity and continue to operate as normal even in the event of a failure of one of the units. Since that time, Nortel’s Flexible Advanced Stack (FAST) architecture technology has matured considerably. This matu-rity is seen in functionality with many additional features such as:

• Ability to add or remove switches in the stack on the fly without any resets or reboots required

• Auto Unit Replacement performs configuration of the new unit and image upgrade/downgrade to match the existing stack

• Stack Monitor provides a warning if the size of the stack changes • Stack Loopback test allows the

network administrator to remotely test stack port and stack cable integrity

• Stack Counters provide information on traffic flowing across the stack ports

Along with feature enhancements, there has also been an appreciable increase in stacking bandwidth. A 2.5 Gigabit stack was sufficient to handle 10/100 ports (BayStack 450), but with the addition of 10/100/1000 and 10 Gigabit ports, there came the need to increase the stacking bandwidth, which now stands at 144 Gigabits per switch and up to 1.152 Terabits per stack of eight (Ethernet Routing Switch 5600). This increase in bandwidth becomes critical for server-to-server communica-tions within a single stack as well as supporting shared access to common uplinks.

The common deployment of stackable switches is vertically stacked within a single rack. These switches are normally installed in wiring closets as edge access ports for users. The stack is created by cabling one switch into another and then completing the stack with a return cable connecting the last switch in the stack to the first switch in the stack. This stacking technology not only provides high band-width between switches, but also provides a single management interface, so all are managed together as a single logical unit.

Horizontal Stacking

By leveraging Nortel’s resilient stacking architecture with its FAST technology, Horizontal Stacking provides a unique solution to solve today’s challenges in the data center Server Access Layer. The concept simply takes the “normal” vertical stacking of the Ethernet Routing Switches and cables the switches in a horizontal orientation. This now allows switches across multiple racks to be connected as a stack. No special soft-ware or configurations are required to take advantage of this unique physical solution. Nortel offers various lengths of stacking cables — 42cm, 1m, 3m and 5m — providing the needed flexibility to connect the switches across multiple server racks.

The physical configuration of the hori-zontal stack is extremely flexible. Figure 3 depicts one permutation in which there is a single switch installed in each rack. However, if needed, multiple switches can be installed in the same rack and horizontally stacked with other switches in adjacent racks. This can be thought of as combining both vertical and horizontal stacking simultaneously.

A Horizontal Stack can range in size from two switches up to eight switches. Switches can be added to the stacks without affecting existing stack operations. This allows for pay-as-you-grow functionality and no need for network downtime to add capacity. The Horizontal Stack can handle up to 384 10/100/1000 and 16 10 Gigabit connections using the Ethernet Routing Switch 5650TD. Any of the Ethernet Routing Switch 5000 series can be stacked together as required to meet the port count and interface requirements in the individual server racks.

Once the Horizontal Stack has been cabled up correctly, with every switch having two stacking connections, it is recommended to renumber the units in the stack. The base unit should be the leftmost or rightmost unit in the stack and identified as unit #1. Moving sequentially across the racks either left to right or right to left, number the units in the stack as #2, #3, #4, etc. Having the Horizontal Stack in sequential order makes it much easier for ongoing network management of the stack. This ensures that configura-tion changes are made to the appro-priate switch in the approappro-priate rack. In the event there is uncertainty about the unit numbers in the stack, the

Ethernet Routing Switches provide a feature to identify their unit number by illuminating port LEDs on the front of the switch that correlate to their unit number (i.e., unit number 6 will illumi-nate ports 1 through 6 LEDs).

By stacking the Ethernet switches, there is a reduction in the number of uplinks required to the Distribution/Core Layer. The number of uplinks and the speed of those uplinks (Gigabit or 10 Gigabit) are dependent on the amount of traffic expected to traverse the uplinks. There is no absolute formula for number of servers to uplink bandwidth as this is dependent on types of applications and amount of data flow expected.

In order to help characterize data flow and to assist in troubleshooting, Nortel supports IPFIX (IP Flow Information eXport) which is the IETF standard for flow management based on NetFlow v9. Offering these capabilities at the Server Access Layer allows the network admin-istrator to understand flows to and from users as well as between servers. IPFIX can easily be used for real-time traffic monitoring, troubleshooting, security and trending network traffic for capacity analysis. Nortel’s IP Flow Manager is the network management software that

provides the collection and analysis of the IP flow data.

IPFIX can help identify when there is a need to add uplink capacity. This proac-tive approach alleviates the uncertainty of uplink utilization and also mitigates the need for any outage to simply add uplink bandwidth when required. Nortel supports up to eight physical links within a single trunk group. Providing high-bandwidth and low-latency connectivity between servers highlights another major advantage in stacking the Ethernet switches across the racks. The Ethernet Routing Switch 5600 supports 144Gbps stacking bandwidth per switch and up to 1.152Tbps of bandwidth in a stack of eight units. Using these capabilities allows a high-bandwidth pipe between servers on the same stack. The Ethernet Routing Switch 5600 also supports the Nortel FAST technology which uses a shortest path algorithm across the stack for the most efficient use of that stack bandwidth.

The latency for server-to-server communication can be reduced from milliseconds to microseconds when utilizing Nortel’s stacking technology. The hardware-based forwarding and unique stacking algorithm support a 9µs average port-to-port latency within a stack. This order of magnitude improve-ment can be critical to time-sensitive applications and is near mandatory when working in high-speed computing and grid computing environments. Another performance benefit realized by stacking is better resource utilization of the Ethernet switches. The more traffic that stays local to the stack suggests there is less traffic that must traverse the uplinks. In many cases, traffic from a server must traverse the uplinks and Figure 3. Extend a stack up to eight racks wide with scalable capacity

Extend a stack up to 8 racks — total stack distance up to 35 meters 160 GbE

uplinks bandwidth

Figure 4. Resiliency all the way to the server interface Square or Full Mesh SMLT into Switch Cluster Core

Extend a stack up to 8 racks wide

Up to 320Gb uplink bandwidth IST

core just to have it come right back to the server in the next cabinet. This is a waste of bandwidth on the uplinks and degrades overall server performance. It is common with many applications that there is significant data flow between servers, and therefore advantageous to engineer the server connectivity to remain within a Horizontal Stack. Horizontal Stacking with Switch Clustering

Nortel’s Switch Clustering technology was introduced in 2001 and offers a simplified alternative to Spanning Tree. Simply aggregate two like Nortel Ethernet Routing Switches or stacks (Ethernet Routing Switches 8600, 8300, 1600, 5000) with an Inter-Switch Trunk (IST). This virtualizes the two Switch Cluster Cores and allows them to act as a single logical entity for all devices that are dual attached. No ports or links are blocked and there are no proprietary protocols required from the Switch Cluster Core to the edge devices. Therefore, Switch Clustering is

totally interoperable with all third-party devices that support any form of link aggregation.

The technology supports an active/ active model for uplinks and sub-second failover, and eliminates the need for Spanning Tree, which in turn signifi-cantly reduces the complexity of the network. Likewise, Switch Clustering supports the same active/active connec-tivity for servers. This now extends the network resiliency from the data center Core all the way to the server NIC. Any server that supports link aggregation can take full advantage of Nortel’s Switch Clustering technology. This active/active configuration provides added band-width to servers and added resiliency with sub-second failover, making it the perfect architecture to support virtual-ized applications with VmWare and Microsoft’s Hyper-V technology. The Nortel Ethernet Routing Switch 5600 supports Switch Clustering with an Advanced Software License. By combining the resilient stacking

architecture with the resiliency of Switch Clustering using Split Multi-Link Trunking (SMLT) or Single Multi-Link Trunking (SLT), the overall data center design provides the highest level of resiliency for all dual connected devices. The ideal design includes installing a switch from each Horizontal Stack in a server cabinet/rack. This would allow dual connections from the server to the Ethernet switches inside a rack — making cable installation, maintenance and troubleshooting much easier as all network connections are contained in the server rack.

Combining these two technologies provides several key advantages in the data center — the most important being the ability to dual connect the servers into network infrastructure in such a way as to allow for any planned or unplanned network outages without interrupting server access. Switch Clustering provides sub-second failover and therefore any outage of a switch or stack will not affect traffic to and from

Horizontal Stacking Horizontal Stacking with Switch Clustering

Potential single point of failure (stack)

the server. Outages due to hardware failure happen, but are very rare. What is more common is the need to perform network maintenance on a regular basis. Switch Clustering provides the answer to the needed resiliency in either case as there will be no outages to applica-tions, since the network connectivity will always be maintained.

Switch Clustering allows the servers to be dual connected to the network in an active/active model, which increases availability and increases available network bandwidth to the server. There is no proprietary software required on the servers to take advantage of Nortel’s Switch Clustering technology. Simply configure the servers for link aggregation. Terminology used by the various server vendors varies for this functionality, but all support some form whether called NIC teaming, static 802.3ad, Multi-Link Trunking, Sun Trunking, etc.). For network bandwidth-intensive applica-tions, Switch Clustering can support up to eight physical links and speeds of 10, 100, 1000 and 10 Gigabit. It is required that all physical links be of the same speed and duplex setting. For details on NIC teaming and Switch Clustering configurations, please refer to the Data Center Server Access Solution Guide (NN48500-577).

The connectivity between the Horizontal Stack Switch Cluster and the data center Switch Cluster Core is comprised of a Square or Full Mesh Split Multi-Link Trunk topology. These topologies ensure resiliency between the Server Access Layer and the data center Core Layer along with being scalable for the number of uplinks and amount of bandwidth required. Figure 5 shows the logical view of the Horizontal Stack Switch Cluster with server connectivity and Square SMLT to the Distribution/Core.

Price, performance and the importance of Green IT The Nortel Ethernet Routing Switch 5600 offers top-of-the-line performance, at a reasonable price. When comparing the switching capacity, stacking band-width and feature set, the Ethernet Routing Switch 5000 family stands out as a prime offering. Its unique ability to combine switching performance with features such as Switch Clustering puts the Ethernet Routing Switch 5000 family in a class by itself.

The addition of the ERS 5600 adds significant features and capabilities, especially when deployed in the Data Center. By supporting built-in N+1 power redundancy that is hot swap-pable, the ERS 5600 eliminates the need for any external redundant power supply devices. This saves valuable space in the rack and more importantly reduces the overall cost of the solu-tion while still providing full power redundancy. The advanced chipsets of the ERS 5600 also provide additional features such as many-to-many port mirroring, hardware that is capable of IPv6 routing and integrated IPS-Lite

functionality with its Denial of Service (DoS) Attack Prevention Package (DAPP).

The advantages don’t stop there. In an effort to help reduce OPEX costs specifically around network operations, the Ethernet Routing Switch 5000 delivers a common industry de facto CLI (command line interface) along with TACACS+, making it easy to learn and integrate into any network. Nortel now offers industry-leading warranty services for the portfolio of Stackable Ethernet Routing Switches — the ERS 2500, ERS 4500 and the ERS 5000 Series product lines. The complimentary next-business-day shipment of failed hardware is now extended to run for the full life of the product. Nortel also offers complimentary basic technical support (Level 1) for the supported lifecycle of the product. This includes the software version shipped with or updated through optional SRS Basic, or a separate Service Contract over and above our normal Warranty.

The Ethernet Routing Switch 5000 series is also making a critical impact with regard to Green IT. When compared to other industry leaders, the Ethernet Routing Switch 5000 family provides significantly lower power consumption (whether sitting at idle or loaded up to 100 percent). This lower power consumption translates into energy cost savings to power the device. Another benefit is lower BTU output, which in turn reduces the amount of cooling required in the data center, which also translates into energy cost savings. Last, but not least, is the carbon footprint and once again, the Ethernet Routing Switch 5000 family leads the way with a much lower value than comparable switches from other Distribution/ Core Switch Cluster Horizontal Stack Switch Cluster Server with NIC team SMLT SMLT/SLT

vendors1_{. For more information on}

Nortel Energy Efficiency, please visit www.nortel.com/startsaving and test out the Nortel Energy Efficiency Calculator tool to see how much savings are avail-able with Nortel solutions.

Conclusion

The Nortel data center architecture provides significant value to enter-prise customers. A key piece of this strategy is the Server Access Layer and the use of Horizontal Stacking and Switch Clustering. Horizontal Stacking provides:

• High bandwidth and low latency for server-to-server communication within the stack

• Reduces the number of ports required in the data center Core — no need for an uplink per switch

• Simplifies cabling between server and Ethernet switch — keeps all cabling within the server rack

• Easily add capacity (switches, uplinks) on the fly without any network downtime

And when combined with Switch Clustering provides:

• Dual-homed, active/active server connectivity — eliminates the network as a potential point of failure • Increased network bandwidth to

servers — multiple Gigabit or 10 Gigabit connections

• Simple network design — no need for spanning tree or Layer 3 load balancing of NICs

• Ability to perform network mainte-nance (i.e., switch software upgrades) with zero impact to server availability

1 _{The Tolly Group Report Number 208298, July 2008, reported the results of a power consumption evaluation of six}

Nortel converged data network products consisting of large and medium core and wiring closet Ethernet Routing Switches, enterprise branch office routers and IP phones compared to similar Cisco products on the basis of power consumption, heat dissipation and five-year operational cost. See also The Tolly Group report Number 208275, January 2008, and InfoTech Research Group Report, January 30th 2008.

Whether used separately or combined together, these features provide added benefits of performance, resiliency, Green IT and lower overall TCO as compared to any other vendor’s offering. Please refer to the Cambridge University Case Study posted on the Nortel website at:

BUSINESS MADE SIMPLE

Nortel 35 Davis Drive

Research Triangle Park, NC 27709 USA

In Canada:

Nortel

195 The West Mall

Toronto, Ontario M9C 5K1 Canada

In Caribbean and Latin America:

Nortel

1500 Concorde Terrace Sunrise, FL 33323 USA

In Europe:

Nortel

Maidenhead Office Park, Westacott Way Maidenhead Berkshire SL6 3QH, UK Email: [email protected] In Asia: Nortel United Square 101 Thomson Road Singapore 307591 Phone: (65) 6287 2877

Nortel is a recognized leader in delivering communications capabilities that make the promise of Business Made Simple a reality for our customers. Our next-generation technologies, for both service provider and enterprise networks, support multimedia and business-critical appli-cations. Nortel’s technologies are designed to help eliminate today’s barriers to efficiency, speed and performance by simplifying networks and connecting people to the information they need, when they need it. Nortel does business in more than 150 countries around the world. For more information, visit Nortel on the Web at www.nortel.com. For the latest Nortel news, visit www.nortel.com/news.

For more information, contact your Nortel representative, or call 1-800-4 NORTEL or 1-800-466-7835 from anywhere in North America.

Nortel, the Nortel logo, Nortel Business Made Simple and the Globemark are trademarks of Nortel Networks. All other trademarks are the property of their owners.

Copyright © 2009 Nortel Networks. All rights reserved. Information in this document is subject to change without notice. Nortel assumes no responsibility for any errors that may appear in this document.