Here we consider enterprise data center design and discuss the three layers of the data center architecture. Modular versus one-rack unit access switch designs are compared, as are the options for scaling the data center for high availability.
Core and Aggregation Layer Infrastructure Design
The three layer data center design is as follows:
n Core layer: Composed of the high-speed packet-switching backplane
n Aggregation layer: Provides service module integration, Layer 2 domain definitions, spanning-tree processing, and default gateway redundancy
n Access layer: Provides physical connection for servers to the network
Data center core layer design: Core layer allows for high-speed packet switching between multiple aggregation modules. Inclusion of a data center core is based on a number of considerations such as 10 Gigabit Ethernet (GigE) port density, administrative domains and models used, and plans for future growth.
In this design, all links are Layer 3 at the core with the Layer 2/3 boundaries at or below the aggregation layer modules.
Open Shortest Path First (OSPF) Protocol routing recommendations include the following:
n Use NSSA from the core down.
n The auto-cost reference-bandwidth 10000 command should be used to set the bandwidth to 10GE and allow OSPF to differentiate the cost on higher speed links such as 10GE trunk links.
n Simplify troubleshooting by using the loopback interfaces for the router ID.
n Use the passive-interface default command.
© 2008 Cisco Systems Inc. All rights reserved. This publication is protected by copyright. Please see page 98 for more details.
CCDP ARCH Quick Reference
CCDP ARCH Quick Reference By Kevin Wallace, Michael Watkins ISBN: 9781587054990 Publisher: Cisco Press
Prepared for Kevin Kem, Safari ID: [email protected] Licensed by Kevin Kem
Print Publication Date: 2007/10/26 User number: 1023945 Copyright 2007, Safari Books Online, LLC.
This PDF is exclusively for your use in accordance with the Safari Terms of Service. No part of it may be reproduced or transmitted in any form by any means without the prior written permission for reprints and excerpts from the publisher. Redistribution or other use that violates the fair use priviledge under U.S. copyright laws (see 17 USC107) or that otherwise violates the Safari Terms of Service is strictly prohibited.
n Use OSPF authentication.
n Use the timers throttle spf command to tune OSPF timers.
Enhanced Interior Gateway Routing Protocol (EIGRP) routing recommendations include the following:
n Use the ip summary-address eigrp command to advertise a default summary route into the data center and to summarize the data center subnets.
n Apply the passive-interface default command.
Aggregation layer design: A pair of interconnected aggregation switches, referred to as modules, are used to scale the aggregation layer through the following:
n Spanning-tree scaling n Access layer density scaling
n Hot Standby Router Protocol (HSRP) scaling n Application services scaling
If Layer 2 is used, special consideration should be given to Spanning Tree Protocol (STP) design because the aggregation modules allow the spanning-tree domain to be distributed. Rapid STP (RSTP) is recommended over Multiple Spanning Tree (MST).
Integrated services module: The aggregation layer may also employ integrated service modules to provide such services as firewall, Secure Sockets Layer (SSL) offload, content switching, intrusion detection, and network analysis.
Service model designs: Redundancy for these integrated services may be deployed as either active/active pairs or active/standby pairs.
© 2008 Cisco Systems Inc. All rights reserved. This publication is protected by copyright. Please see page 98 for more details.
CCDP ARCH Quick Reference
CCDP ARCH Quick Reference By Kevin Wallace, Michael Watkins ISBN: 9781587054990 Publisher: Cisco Press
Prepared for Kevin Kem, Safari ID: [email protected] Licensed by Kevin Kem
Print Publication Date: 2007/10/26 User number: 1023945 Copyright 2007, Safari Books Online, LLC.
This PDF is exclusively for your use in accordance with the Safari Terms of Service. No part of it may be reproduced or transmitted in any form by any means without the prior written permission for reprints and excerpts from the publisher. Redistribution or other use that violates the fair use priviledge under U.S. copyright laws (see 17 USC107) or that otherwise violates the Safari Terms of Service is strictly prohibited.
Data Center Design for the Enterprise
Active/Active:
n Increases overall performance
n Allows uplink load balancing while having services applied Active/Standby:
n Predictable; simplifies troubleshooting
n Underutilizes access layer links, service modules, and switch fabric VRFs in the data center:
n Allows use of application services with multiple access topologies n Maps to path isolation MAN/WAN designs
n Supports security policy by user group n Enables partitioning of network resources
Design of the Access Layer
A number of models may be used in access layer design, including Layer 2 looped model, Layer 2 loop-free model, and Layer 3 model where Layer 2 services from the aggregation layer are not supported.
Layer 2 looped model: There are two primary Layer 2 model topologies, the looped triangle and the looped square.
© 2008 Cisco Systems Inc. All rights reserved. This publication is protected by copyright. Please see page 98 for more details.
CCDP ARCH Quick Reference
CCDP ARCH Quick Reference By Kevin Wallace, Michael Watkins ISBN: 9781587054990 Publisher: Cisco Press
Prepared for Kevin Kem, Safari ID: [email protected] Licensed by Kevin Kem
Print Publication Date: 2007/10/26 User number: 1023945 Copyright 2007, Safari Books Online, LLC.
This PDF is exclusively for your use in accordance with the Safari Terms of Service. No part of it may be reproduced or transmitted in any form by any means without the prior written permission for reprints and excerpts from the publisher. Redistribution or other use that violates the fair use priviledge under U.S. copyright laws (see 17 USC107) or that otherwise violates the Safari Terms of Service is strictly prohibited.
Benefits of the Layer 2 looped model:
n Offers Layer 2 adjacency
n Extends VLANs between aggregation switches n Supports sharing of service module across access layer n Provides redundancy using RSTP
© 2008 Cisco Systems Inc. All rights reserved. This publication is protected by copyright. Please see page 98 for more details.
Looped Square .1Q Trunk
Primary STP Root Primary HSRP Active Services
Secondary STP Root Secondary HSRP Standby Services
L3 L2
ACC 1 ACC 2
Looped Triangle
ACC 3 ACC 4
FIGURE 5-1 Layer 2 Looped Model
CCDP ARCH Quick Reference
CCDP ARCH Quick Reference By Kevin Wallace, Michael Watkins ISBN: 9781587054990 Publisher: Cisco Press
Prepared for Kevin Kem, Safari ID: [email protected] Licensed by Kevin Kem
Print Publication Date: 2007/10/26 User number: 1023945 Copyright 2007, Safari Books Online, LLC.
This PDF is exclusively for your use in accordance with the Safari Terms of Service. No part of it may be reproduced or transmitted in any form by any means without the prior written permission for reprints and excerpts from the publisher. Redistribution or other use that violates the fair use priviledge under U.S. copyright laws (see 17 USC107) or that otherwise violates the Safari Terms of Service is strictly prohibited.
Data Center Design for the Enterprise
Layer 2 loop-free models: Used when Layer 2 support is required but a looped topology is undesirable. Enables span-ning tree as a safeguard against loops and provides several benefits, including Layer 2 adjacency, stability, and active uplinks.
Layer 2 loop-free topologies: Loop-free U access and loop-free inverted U access.
Layer 2 FlexLinks: An alternative to the looped access topology. When using this design, STP is disabled on FlexLinks, and accidental loops between switches are possible.
Layer 3 in the access layer: A dedicated subnet is used to permit access switches connect to the aggregation switches using a Layer 3 uplink.
© 2008 Cisco Systems Inc. All rights reserved. This publication is protected by copyright. Please see page 98 for more details.
L3 L2
DC Access DC Aggregation DC Core DC Core
FIGURE 5-2 Layer 3 Access Model
CCDP ARCH Quick Reference
CCDP ARCH Quick Reference By Kevin Wallace, Michael Watkins ISBN: 9781587054990 Publisher: Cisco Press
Prepared for Kevin Kem, Safari ID: [email protected] Licensed by Kevin Kem
Print Publication Date: 2007/10/26 User number: 1023945 Copyright 2007, Safari Books Online, LLC.
This PDF is exclusively for your use in accordance with the Safari Terms of Service. No part of it may be reproduced or transmitted in any form by any means without the prior written permission for reprints and excerpts from the publisher. Redistribution or other use that violates the fair use priviledge under U.S. copyright laws (see 17 USC107) or that otherwise violates the Safari Terms of Service is strictly prohibited.
Layer 3 access model benefits:
n Reduces broadcasts and fault domains.
n Provides for server stability and application isolation.
n All uplinks are available up to Equal Cost Multipath (ECMP) maximum.
n Fast uplink convergence in the event of a failover or fallback.
Blade servers: These may be implemented in the data center access layer, often as a replacement for older server farms or where new applications that require clustering are deployed.
Blade server challenges and considerations:
n Administrative domains n Interoperability n Spanning-tree scaling n Pass-through cabling n Switch trunk topologies n Environmental Issues
Blade server connectivity: Blade servers can support either Layer 2 or Layer 3 topologies depending on the server broadcast domain or specific administrative requirements. One option for connecting blade servers is integrated InfiniBand switches.
Another feature of blade servers is Layer 2 trunk failover (link-state tracking), which provides Layer 2 redundancy in the network when used in conjunction with proper server network interface card (NIC) adapter tuning.
© 2008 Cisco Systems Inc. All rights reserved. This publication is protected by copyright. Please see page 98 for more details.
CCDP ARCH Quick Reference
CCDP ARCH Quick Reference By Kevin Wallace, Michael Watkins ISBN: 9781587054990 Publisher: Cisco Press
Prepared for Kevin Kem, Safari ID: [email protected] Licensed by Kevin Kem
Print Publication Date: 2007/10/26 User number: 1023945 Copyright 2007, Safari Books Online, LLC.
This PDF is exclusively for your use in accordance with the Safari Terms of Service. No part of it may be reproduced or transmitted in any form by any means without the prior written permission for reprints and excerpts from the publisher. Redistribution or other use that violates the fair use priviledge under U.S. copyright laws (see 17 USC107) or that otherwise violates the Safari Terms of Service is strictly prohibited.
Data Center Design for the Enterprise
Scaling Data Center Architecture
In designing the data center architecture, both density and scalability implications between modular and one-rack unit (1 RU) access layer switching models must be considered, as must the following:
n Cabling n Cooling n Power n Density
n 10 Gigabit Ethernet uplink support n Resiliency features
n Intended use
Bandwidth and Uplink Density Consideration
The port-channel load-balance command improves load distribution for EtherChannel ports because it presents more unique values to the hashing algorithm. EtherChannel utilization can be further optimized with the Min-Link feature, which allows for the specification of a minimum number of available ports for a PortChannel to be considered a valid path.
Service Layer Switches
Service layer switches provide greater scalability by supporting service modules, but may call for quality of service (QoS) or separate links for fault-tolerant paths. This may also require Layer 3 peering with route health injection (RHI), and only necessary Layer 2 VLANs should be extended to service switches.
Cisco Application Control Engine (ACE) modules may also be used to scale uplink port density or aggregate layer switch slots.
© 2008 Cisco Systems Inc. All rights reserved. This publication is protected by copyright. Please see page 98 for more details.
CCDP ARCH Quick Reference
CCDP ARCH Quick Reference By Kevin Wallace, Michael Watkins ISBN: 9781587054990 Publisher: Cisco Press
Prepared for Kevin Kem, Safari ID: [email protected] Licensed by Kevin Kem
Print Publication Date: 2007/10/26 User number: 1023945 Copyright 2007, Safari Books Online, LLC.
This PDF is exclusively for your use in accordance with the Safari Terms of Service. No part of it may be reproduced or transmitted in any form by any means without the prior written permission for reprints and excerpts from the publisher. Redistribution or other use that violates the fair use priviledge under U.S. copyright laws (see 17 USC107) or that otherwise violates the Safari Terms of Service is strictly prohibited.
Spanning-Tree Design for High Availability
The recommended spanning-tree protocols for use in a data center are 802.1w, implemented by Cisco as Rapid PVST+(RSTP) and 802.1s, known as Multiple Spanning Tree (MST).
STP logical interfaces: To determine STP logical interfaces, sum[(each trunk on switch) * (active VLANs on each trunk)] + (number of nontrunking interfaces on the switch).
Virtual ports: These are a per-line card value that reflects the total number of spanning-tree processing instances used on a line card. To calculate STP virtual ports, sum[(each trunk port on line card) * (active VLANs per port)].
1 RU designs: With this, the chances of a larger spanning-tree diameter, and possibly more STP issues, increase. It is best to use aggregation modules to scale STP and 10GE density.
Guidelines for scaling STP designs:
n Manually prune trunks.
n Use MST if Rapid Spanning Tree Protocol (RSTP) cannot scale sufficiently.
n Limit Hot Standby Router Protocol (HSRP) instances to 500.
n Divide the STP domain by adding aggregation modules.
Providing high availability in the data center:
Three key areas are seen in common failures in the path from server to aggregation switch: network links, access switch, server network adapter. To address these, dual attached servers using network adapter teaming software connected to dual attached access switches may be deployed.
Having a server with a single network interface card (NIC) might lead to as many as three single points of failure; the NIC, the cable, and the switch to which it is connected. NIC teaming can eliminate these single points of failure.
© 2008 Cisco Systems Inc. All rights reserved. This publication is protected by copyright. Please see page 98 for more details.
CCDP ARCH Quick Reference
CCDP ARCH Quick Reference By Kevin Wallace, Michael Watkins ISBN: 9781587054990 Publisher: Cisco Press
Prepared for Kevin Kem, Safari ID: [email protected] Licensed by Kevin Kem
Print Publication Date: 2007/10/26 User number: 1023945 Copyright 2007, Safari Books Online, LLC.
This PDF is exclusively for your use in accordance with the Safari Terms of Service. No part of it may be reproduced or transmitted in any form by any means without the prior written permission for reprints and excerpts from the publisher. Redistribution or other use that violates the fair use priviledge under U.S. copyright laws (see 17 USC107) or that otherwise violates the Safari Terms of Service is strictly prohibited.
Data Center Design for the Enterprise
NIC teaming configurations include the following:
n Adapter fault tolerance (AFT) n Switch fault tolerance (SFT) n Adaptive load balancing (ALB)
Server attachment methods: EtherChannel provides scalable bandwidth for network servers that can bundle multiple links to allow higher throughputs between servers and clients, and to provide redundancy.
Failover times: Layer 2, Layer 3, and Layer 4 components all contribute to overall failover time. Components at each layer have different recovery times and should be evaluated and optimized.
Nonstop forwarding (NSF) and stateful switchover (SSO): Intrachassis SSO at Layers 2 to 4 can be provided by NSF with SSO. This is an excellent method for redundancy. SSO synchronizes the state of trunks, interfaces, EtherChannels, port security, and Switched Port Analyzer / Remote Switched Port Analyzer (SPAN/RSPAN). STP, UniDirectional Link Detection (UDLD), and VLAN Trunking Protocol (VTP), or NSF with EIGRP, OSPF, Intermediate System-to-Intermediate System (IS-IS), or Border Gateway Protocol (BGP) allows for recovery with no route flapping.
© 2008 Cisco Systems Inc. All rights reserved. This publication is protected by copyright. Please see page 98 for more details.
CCDP ARCH Quick Reference
CCDP ARCH Quick Reference By Kevin Wallace, Michael Watkins ISBN: 9781587054990 Publisher: Cisco Press
Prepared for Kevin Kem, Safari ID: [email protected] Licensed by Kevin Kem
Print Publication Date: 2007/10/26 User number: 1023945 Copyright 2007, Safari Books Online, LLC.
This PDF is exclusively for your use in accordance with the Safari Terms of Service. No part of it may be reproduced or transmitted in any form by any means without the prior written permission for reprints and excerpts from the publisher. Redistribution or other use that violates the fair use priviledge under U.S. copyright laws (see 17 USC107) or that otherwise violates the Safari Terms of Service is strictly prohibited.