Expert Reference Series
of White Papers
Planning for the Redeployment of Technical
Personnel in the Modern Data Center
Modern Data Center
Raymond B. Dooley, CCSI, Global Knowledge Course Director Introduction The modern data center is a highly technical combination of servers, memory devices, management workstations, security devices, software, network-related services, equipment racks, Ethernet LAN switches, multilayer switches, cabling, power systems, cooling systems, fabric switches, and a few people. From the standpoint of providing for the exchange of information to and from a client and one of the servers in the data center, the process is divided into two clearly defined halves. The Ethernet LAN side consists of all the switching, routing, and security technology for the client PC to establish and maintain a TCP/IP connection to the server. So, the LAN side consists of Ethernet technology and TCP/IP technology. The data center is composed of many complex components and protocols, including: • The three-layer model • Server attachment • Choosing a platform and general issues • One rack unit switching example • Layer 2 or Layer 3 access • High availability • Services Layer and left vs. right traffic flow Copyright ©2014 Global Knowledge Training LLC. All rights reserved. 2
Planning for the Redeployment of
Technical Personnel in the
Modern Data Center
Raymond B. Dooley, CCSI, Global Knowledge Course Director
Introduction
The modern data center is a highly technical combination of servers, memory devices, management workstations, security devices, software, network-related services, equipment racks, Ethernet LAN switches, multilayer switches, cabling, power systems, cooling systems, fabric switches, and a few people.
From the standpoint of providing for the exchange of information to and from a client and one of the servers in the data center, the process is divided into two clearly defined halves.
The Ethernet LAN side consists of all the switching, routing, and security technology for the client PC to establish and maintain a TCP/IP connection to the server. So, the LAN side consists of Ethernet technology and TCP/IP technology. The data center is composed of many complex components and protocols, including:
• The three-layer model • Server attachment
• Choosing a platform and general issues • One rack unit switching example • Layer 2 or Layer 3 access • High availability
The three-layer model consists of access layer, aggregation layer, and core layer and is used to describe
functionality as well as hardware and software components that support TCP/IP and LAN traffic into and out of the data center. These functions are summarized in the graphics. It is the best way to simplify and explain data center functions and design as well as help isolate management and troubleshooting issues.
Benefits of the Three-Layer Model
– Layer 2 domain sizing
– Service module support
– Supports a mix of
access layer models
– Supports NIC teaming
and high availability
clustering
Campus Core
Service Modules Data Center Core
Data Center Aggregation
Data Center Access
Layer 2 Clustering and NIC Teaming
Blade Chassis with Pass-Through Blade Chassis with Integrated Switch Mainframe with OSA Layer 3 Access Campus Core
Data Center Core
Data Center Aggregation Service Modules
Data Center Access
Layer 2 Clustering and NIC Teaming
Blade Chassis with Pass-Through Blade Chassis with Integrated Switch Mainframe with OSA Layer 3 Access
Data Center Architecture Overview
– Web, application, and database multitier environments
– Layer 2 and Layer 3 access topologies – Layer 2 adjacency requirements – Dual and single attached 1RU and
blade servers
– Mix of oversubscription requirements
– Multiple aggregation modules – Stateful services for security
and load balancing
Server Attachment Methods
Layer 2Layer 3
EtherChannels All Links Active:
Load Balancing Only One Link Active: Fault
Tolerant Mode
Server attachment at the access level is shown in the graphic. To support the four (99.99) and five (99.999) high availability objective, most servers are dual-homed to two separate access Ethernet switches for redundancy. This link can be in active/standby or active/active (load balancing) mode. There are also server network interface cards( NICs) that support Cisco EtherChannel for “bundling” multiple Ethernet links.
Considerations for Choosing an Access
Layer Platform
– Cabling design requirements
– Cooling requirements
– Power requirements
– Density
– 10-Gigabit Ethernet uplink support
– Resiliency features
– Intended use
Cabinet Design with 1RU Switching
– 1RU designs minimize
cabling from the cabinet.
– 1RU designs have several
issues:
• Cooling requirements limit the number of servers. • Additional Gigabit
EtherChannel or 10-Gigabit Ethernet uplinks are used. • Higher STP processing is
required.
• Management complexity is increased.
• Multiple 1RU switches may be needed for port density.
Single Rack 2 Switches Dual Rack 2 Switches
C abl ing R em ai ns in C abi net s
Since Layer 2 Ethernet switches are used in the data center, Spanning Tree Protocol (STP) is required to avoid Ethernet loops and broadcast storms. When VMware is used, it is necessary to span VLANs between access layer switches, and so converting to Layer 3 at the access layer is not possible to support all servers in the data center. The graphic shows a typical 1 rack unit (RU) server/switch design using either standalone or blade servers. This design can be improved with Cisco Fabric Extenders and Nexus switch solutions to reduce cabling requirements. STP issues can be alleviated by deploying Rapid Spanning Tree Protocol (RSTP) or Multiple Spanning Tree Protocol (MSTP), or using Cisco features such as VSS (Virtual Switching System) with the 6500 switch, Virtual Port Channel (VPC), and Virtual Device Contexts (VDC) with Nexus switches.
Copyright ©2014 Global Knowledge Training LLC. All rights reserved. 5
Access
Choosing Between Layer 2 and Layer 3
Access Designs
– NIC teaming and adjacencies – High-availability clustering and
adjacencies – VLAN extension
– Custom application requirements
– Ease in managing loops – Faster convergence – Link utilization on uplinks – Broadcast domain sizing
R api d P V S T+ or M S T OS P F, E IGR P Aggregation Access Layer 2 Layer 3 Aggregation Layer 3 Layer 2 Other considerations:
Staff skill set
Oversubscription requirements
Service modules support and placement
Layer 3 Layer 2
It would seem that implementing multi-layer switches at the access level would provide a simple solution to many problems, thereby eliminating the need for STP. However, applications such as VMware that require Layer 2 adjacency will not work. So, a modern data center design includes both Layer 2 and Layer 3 access layer switches.
High Availability in the Data Center
High Availability Recommendations Implemented No Redundancy Implemented Layer 2 Layer 3
• Common failures in the
path from server to
aggregation switch:
– Server network adapter – Network links
– Access switch
The first step in implementing high availability in the data center is to ensure that a link or device failure will not prohibit access to the server. A proper design is shown on the right side of the graphic.
Integrated Service Modules
+
Layer 4 through Layer 7 services can be integrated in Cisco Catalyst 6500 Series Switches.
– Includes server load balancing, firewall, and SSL services
Two deployment scenarios:
– Active/standby pairs (Cisco CSM, and Catalyst 6500 Series FWSM 2.x)
– Active/active pairs (Catalyst 6500 Series FWSM 3.1)
Blades provide flexibility and economies of scale.
Many additional services are required in a modern data center, such as: • Server load balancers
• Firewalls • Wireless support
• SSL (Secure Socket Layer) VPN termination • Content services
Scaling with Service Layer Switches
– Service layer switches can support service modules:
• Ideal for CSM and SSL modules • Opens slots in aggregation layer
for 10-Gigabit Ethernet ports
Data Center
Core Switch2 Service
Service Switch1
Access Aggregation
In this case, a separate set of 6500 switches can be attached at the aggregation level to provide these services or separate appliances such as the Cisco Advanced Security Appliance (ASA) may be used.
Active STP, HSRP, and Service Context
Alignment
– Align server access to primary components in the aggregation layer:
• STP root is configured with the
spanning-tree vlan vlan_id root primary command.
• Primary HSRP instance is configured with the standby
priority command.
• Active service context is configured by the placement of service modules.
– Provides more predictable design: • More efficient traffic flow • Simplified troubleshooting Core Root Primary HSRP Primary Active Context Root Secondary HSRP Secondary Standby Context
To explain all the required functions of the service modules and other components while maintaining a four or five-nines high availability objective, it is necessary to describe traffic flows into and out of the data center. Based on the graphic, traffic going to or from the data center must traverse the aggregation switch on the left side because that is where all the services (firewall, load balancer) are active. The same services are standby (in grey) on the right side of the aggregation switch. It is the purpose of the services and switches on the right side
to back-up the left side. This arrangement can be implemented on a VLAN-by-VLAN basis, so roughly one half of the traffic goes and comes on the left and one half goes and comes on the right. This is called the active/standby
and sometimes called the active/active model. The left vs. right idea is the same either way.
For this sophisticated solution to work correctly, several items must fit together properly. For VLANs assigned to the “left,” the following should be configured on the left aggregation switch:
• Spanning Tree Root for VLAN(s)
• HSRP, VRRP, GLBP (First Hop Redundancy Protocols) active (same VLANs) • Firewall active (same VLANs)
• Load balance active (same VLANs) • All other services active (same VLANs)
The right aggregation switch would be the STP root and active on all services for the other roughly one half of the VLANs.
The Storage Area Networking (SAN) side includes all the switching, routing, and security technology for the server to establish and maintain a Fiber Channel connection to a data storage device.
Block Storage Devices LAN
SAN Overview
Benefits:– Separates storage from the server (historically).
– High-performance interconnection can provide high I/O throughput.
• Storage is accessed at a block level.
– Storage can be shared among servers. – The SAN fabric is interconnection
hardware. Fibre
Channel SAN
Servers
Large Port Count Directors Reduce FSPF Routes Failure of One Device
Has No Impact on Others QoS and Congestion Control High Performance Crossbar
Major SAN Design Factors
– Port density and topology requirements
– Device oversubscription – Traffic management – Fault isolation
– Convergence and stability
Host Host Host Host Host Host
There are several protocols for delivering the SAN traffic payload (Small Computer System Interface [SCSI]), which has been the standard protocol for disk access since PCs were invented and still is. Note that in the graphic, IP is being used as the transport protocol when SANs are extended over distance.
Storage Protocols: FCIP vs. iSCSI
FCIP
• FC is tunneled over IP. • Connects SAN to SAN. • Supports demanding
QoS requirements. iSCSI
• SCSI commands are carried using IP. • Connects host to
storage.
• Supports demanding QoS requirements.
SCSI Applications (File Systems and Datsbases) SCSI Block
Commands SCSI StreamCommands Other SCSICommands SCSI Commands, Data, and Status
Parallel SCSI
Interfaces Fibre Channel
Parallel SCSI
Interfaces Fibre Channel Ethernet FCIP iSCSI
TCP IP Fibre Channel
The SAN deployment described leads to the design of two parallel networks in the data center.
LAN
Data Center SAN
Data Center LAN
Access Layer
Parallel Data Center Infrastructure
Traditional data centers often deploy multiple parallel infrastructures: • LAN for network connectivity • SAN for storage connectivity SAN B SAN A Ethernet Fibre Channel HPC
Technical advances in data center networking have been made that allow unification of LAN/SAN. A Converged Network Adapter (CNA) allows the server to use the same adapter to send both Ethernet frames and Fiber Chanel frames bundled inside Ethernet frames—Fiber Channel over Ethernet (FCoE).
The Cisco Unified Computing System solution includes an interface for the blade server rack that includes FCoE capability.
LAN
I/O Consolidation in the Data Center
I/O consolidation integrates multiple data center infrastructures on a single unified fabric.
Benefits:
• Reduced cabling • Fewer access layer
switches • Fewer network
adapters per server • Power and cooling
savings • Management
integration • “Wire once” cabling
model
SAN B SAN A
The business case for this idea is solid. Removing one layer of switching along with the cabling lowers the cost of equipment, power, and cooling. The resistance to this idea by SAN professionals has been consistent. They see a future where the SAN network no longer exists and everything they have built and operated is collapsed into the LAN side even though the skill sets of the personnel on either side are completely different and they are basically “left out,” so their resistance is understandable.
The SAN professional’s point-of-view could include the following:
• A system that has proven reliable and performed well is being changed. • SAN professionals do not know LAN and vice versa.
• Arbitrary decisions are being made that affect my life and career. • I do not think the new technology will work.
• Will I have job?
The following ideas would be beneficial to the transition process: • Go over the business case in detail with both sides.
• Explain that Fiber Channel technology is not being eliminated, it is being encapsulated and SAN expertise will continue to be needed.
• Explain that interexchange of SAN and LAN expertise benefits the careers of professionals on both sides. • Provide a clear career path and career counseling for anyone whose job is affected.
Whatever solution is selected for this transition, TRAINING will play a key role. The SAN engineers’ experience will fall into two categories:
1. The engineer is experienced with Fiber Channel network and has limited experience with the Ethernet/IP/switching and routing environment.
2. The engineer has some experience with the Ethernet/IP switching and routing environment from working with that side of the house in the past.
To complete an ideal transition for the SAN personnel, the following approach could be used to have the SAN folks achieve Cisco CCNP certification, which would make them full-fledged routing and switching engineers while retaining all of their SAN skills:
• Category 1: − ICND 1 − ICND 2 − ROUTE − SWITCH − TSHOOT − DCUFI − NEXUS
DCNX1K v2 - Configuring the Cisco Nexus 1000V
DCNX5K - Implementing the Cisco Nexus 5000 and 2000 v2.0
DCNX7K - Configuring Cisco Nexus 7000 Switches v2.1
• Category 2: − ROUTE − SWITCH − TSHOOT − DCUFI − NEXUS
DCNX1K v2 - Configuring the Cisco Nexus 1000V
DCNX5K - Implementing the Cisco Nexus 5000 and 2000 v2.0
DCNX7K - Configuring Cisco Nexus 7000 Switches v2.1
Cisco and Global Knowledge are close to offering the equivalent of a CCNA and CCNP certification specializing in the data center. It is basically the same courses with a Data Center focus.
The benefits of this training strategy are:
• The SAN staff is assimilated into the network group. • The network group now has more skills than before. • A truly integrated network can be supported.
