Molina-HLD-V1.0(Latest)

(1)

Cisco

Cisco Advanced Services

MOLINA HEALTHCARE

Data Center Networking

High Level Design Document

V1.0 (Draft)

Corporate Headquarters

Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706

(2)

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense.

The following information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it is not installed in accordance with Cisco’s installation instructions, it may cause interference with radio and television reception. This equipment has been tested and found to comply with the limits for a Class B digital device in accordance with the specifications in part 15 of the FCC rules. These specifications are designed to provide reasonable protection against such interference in a residential installation. However, there is no guarantee that interference will not occur in a particular installation.

You can determine whether your equipment is causing interference by turning it off. If the interference stops, it was probably caused by the Cisco equipment or one of its peripheral devices. If the equipment causes interference to radio or television reception, try to correct the interference by using one or more of the following measures:

Turn the television or radio antenna until the interference stops. Move the equipment to one side or the other of the television or radio. Move the equipment farther away from the television or radio.

Plug the equipment into an outlet that is on a different circuit from the television or radio. (That is, make certain the equipment and the television or radio are on circuits controlled by different circuit breakers or fuses.)

Modifications to this product not authorized by Cisco Systems, Inc. could void the FCC approval and negate your authority to operate the product. The following third-party software may be included with your product and will be subject to the software license agreement:

CiscoWorks software and documentation are based in part on HP OpenView under license from the Hewlett-Packard Company. HP OpenView is a trademark of the Hewlett-Packard Company. Copyright  1992, 1993 Hewlett-Packard Company.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright  1981, Regents of the University of California.

Network Time Protocol (NTP). Copyright  1992, David L. Mills. The University of Delaware makes no representations about the suitability of this software for any purpose.

Point-to-Point Protocol. Copyright  1989, Carnegie-Mellon University. All rights reserved. The name of the University may not be used to endorse or promote products derived from this software without specific prior written permission.

The Cisco implementation of TN3270 is an adaptation of the TN3270, curses, and termcap programs developed by the University of California, Berkeley (UCB) as part of the UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright  1981-1988, Regents of the University of California.

MOLINA HEALTHCARE

Data Center Networking

High Level Design Document

V1.0 (Draft)

(3)

DMZ Services 28 2.1 INET Services 28 2.1.1 Security 28 2.1.2 Load Balancing 30 2.2 WAN Services 30 2.2.1 Security 30 2.2.2 Load Balancing 30 2.2.3 Optimization 31 2.3 DCN Aggregation Services 31 2.3.1 Production Network 31 2.3.1.1 Security 31 2.3.1.2 Load balancing 31 2.3.2 Development Network 32 2.3.2.1 Security 32 2.3.2.2 Load balancing 32 2.4 SAN Services 32 2.4.1 Data Replication 32 2.4.1.1 Local replication 32 2.4.1.2 Remote replication 33

3. Layer 1, 2 ,3 Design & HA Technologies 35

3.1 L1 Design 35

3.2 L2 Design 35

3.3 L3 Design 35

3.4 HA Technologies 36

3.4.1 Logical Redundancy 36

3.4.1.1 HSRP (Hot Standby Router Protocol) 36 3.4.2 UDLD (Uni-Directional Link Detection) 37

3.4.3 NSF/SSO (Non Stop Forwarding/ Stateful Switchover)37 3.4.4 GOLD (Generic Online Diagnostics)38

3.4.5 uRPF (unicast reverse path forwarding) 39 3.4.6 Trunking39

3.4.7 VTP (VLAN Trunking Protocol) 40 3.4.8 VLAN Hopping 40

3.4.9 Unused ports 40 3.4.10 ISSU 41

3.5 Control Plane and Management Plane Policing 41

3.5.1 Developing a CoPP Policy 43 3.5.2 COPP on NX-OS 44

3.5.3 CoPP Risk Assessment 45

4. Security Technologies 46

4.1 Firewall Technologies 46

4.1.1 Transparent Mode 46 Overview 46

(5)

4.1.1.2 Transparent Firewall in a Network 47

Figure 4.1 TRANSPARENT FIREWALL NETWORK48 4.1.1.3 Transparent Firewall Guidelines 48

4.1.1.4 Unsupported Features in Transparent Firewall 49 4.2 Routed Mode 50

Overview 50

4.2.1 Routed Firewall in a Network 50 4.3 ASA Virtual Context 50

Overview 50

4.3.1 Understanding Multiple Contexts 51 4.3.2 System execution space 51

4.3.3 Admin context 51

4.3.4 User or customer contexts 51

4.4 Packet Flow, Shared Interfaces and Classification in Multimode 52 4.5 Failover Functionality Overview on the ASA 52

4.5.1 Stateful failover 52

4.5.2 Failover and State Links 53

4.5.3 Intrusion Detection & Prevention 53

5. Load Balancing & Technologies 56

5.1 Server Load Balancing 56 5.1.1 Routed Mode 56

5.1.2 One-Armed Mode56

5.2 Global Server Load Balancing 57 5.3 WAN Optimization 57

6.1 Feature Recommendations 58

6.1.1 VSANs 59

6.1.2 Devices Aliases 60

6.1.3 Zoning and Zonesets 60 6.1.4 Security 61

6.1.5 Role Based Access Control (RBAC) 62 6.1.6 Logging 62

6.1.7 Monitoring 63 6.1.8 Call Home 63 6.1.9 Port-Channel 63

6.1.10 N Port Identifier Virtualization 63 6.1.11 N Port Virtualizer 64

6.1.12 Licences 64

6.2 Future SAN 64

6.2.1 Consolidated IO (Fibre Channel over Ethernet FCoE) 64

7. Scalability & Virtualization 66

7.1 DCN Scalability 66 7.1.1 CORE 66 7.1.2 AGGREGATION 66 7.1.3 ACCESS 66 7.1.4 Services 67 7.1.5 SAN Scalability 67

7.2 Network Virtualization Technologies 67

7.2.1 Virtual Port Channel ( vPC ) 68

7.2.2 Virtual Device Context ( VDC ) 68

7.3 Server Virtualization 69

(6)

7.3.2 VMWare with Standalone Server 70 7.3.3 VMWare with Blade Server 70

7.4 Access Layer Architecture for Blade Server 71

8. Management 73 8.1 Network Management 74 8.1.1 SNMP 74 8.1.2 SSH/Telnet 74 8.1.3 Logging 74 8.1.4 NTP 74 8.1.5 RBAC/AAA/TACACS+ 75 8.2 Management Technologies 75

8.2.1 Cut-Through Proxy (Management Firewall) 75 8.2.2 DCNM 75

8.2.3 ANM 76 8.2.4 CSM 76

8.2.5 Fabric Manager 76

9. Rack Design 77

9.1 Data Center Sizing – No of Servers & Server NICs 77

9.2 RACK & POD Design 79

9.2.1 Rack Space Division 79 9.2.2 POD Assignments 79 9.2.3 POD Design 80

10 Document Acceptance 83

(7)

Figures

Figure 1 BLOCK LEVEL DESIGN

13

Cisco incorporates Fastmac and TrueView software and the RingRunner chip in some Token Ring products. Fastmac software is licensed to Cisco by Madge Networks Limited, and the RingRunner chip is licensed to Cisco by Madge NV. Fastmac, RingRunner, and TrueView are trademarks and in some jurisdictions registered trademarks of Madge Networks Limited. Copyright  1995, Madge Networks Limited. All rights reserved.

Xremote is a trademark of Network Computing Devices, Inc. Copyright  1989, Network Computing Devices, Inc., Mountain View, California. NCD makes no representations about the suitability of this software for any purpose.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PRACTICAL PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

AccessPath, AtmDirector, Browse with Me, CCDE, CCIP, CCSI, CD-PAC, CiscoLink, the Cisco NetWorks logo, the Cisco Powered Network logo, Cisco Systems Networking Academy, Fast Step, Follow Me Browsing, FormShare, FrameShare, GigaStack, IGX, Internet Quotient, IP/VC, iQ Breakthrough, iQ Expertise, iQ FastTrack, the iQ logo, iQ Net Readiness Scorecard, MGX, the Networkers logo, Packet, RateMUX, ScriptBuilder, ScriptShare, SlideCast, SMARTnet, TransPath, Unity, Voice LAN, Wavelength Router, and WebViewer are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, Discover All That’s Possible, and Empowering the Internet Generation, are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert Logo, Cisco IOS, the Cisco IOS logo, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Enterprise/Solver, EtherChannel, EtherSwitch, FastHub, FastSwitch, IOS, IP/TV, LightStream, MICA, Network Registrar, PIX, Post-Routing, Pre-Routing, Registrar, StrataView Plus, Stratm, SwitchProbe, TeleRouter, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries.

All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0105R)

INTELLECTUAL PROPERTY RIGHTS:

THIS DOCUMENT CONTAINS VALUABLE TRADE SECRETS AND CONFIDENTIAL INFORMATION OF CISCO SYSTEMS, INC. AND IT’S SUPPLIERS, AND SHALL NOT BE DISCLOSED TO ANY PERSON, ORGANIZATION, OR ENTITY UNLESS SUCH DISCLOSURE IS SUBJECT TO THE PROVISIONS OF A WRITTEN NON-DISCLOSURE AND PROPRIETARY RIGHTS AGREEMENT OR INTELLECTUAL PROPERTY LICENSE AGREEMENT APPROVED BY CISCO SYSTEMS, INC. THE DISTRIBUTION OF THIS DOCUMENT DOES NOT GRANT ANY LICENSE IN OR RIGHTS, IN WHOLE OR IN PART, TO THE CONTENT, THE PRODUCT(S), TECHNOLOGY OF INTELLECTUAL PROPERTY DESCRIBED HEREIN.

(8)

Figure 2 WAN EDGE DESIGN 14

Figure 3 INET EDGE DESIGN

15

Figure 4 DCN DESIGN

16

Figure 1.5 DESIGN FOR 1G SERVERS 17

Figure 1.6 DESIGN FOR 10G SERVERS 18

Figure 1.7 SAN CORE EDGE DESIGN 19

Figure 1.8 MNET DESIGN 20

Figure 1.9 TAPE SAN DESIGN 25

Figure 2.1 INTERNET EDGE FIREWALL 28

Figure 2.2 DMZ FIREWALL 29

Figure 2.3 PARTNER FIREWALL 29

Figure 2.4 WAN EDGE FIREWALLS 30

Figure 2.5 VIRTUAL FIREWALL 32

Figure 2.6 DMX + TIME FINDER 34

Figure 2.7 DMX+RECOVERPOINT 34

Figure 3.1 CONTROL PLANE POLICING 42

Figure 3.2 LOGICAL PLANES OF ROUTER 43

Figure 4.1 TRANSPARENT FIREWALL NETWORK 49

Figure 4.2 ROUTED FIREWALL NETWORK 51

Figure 4.3 IDS PLACEMENT 55

Figure 4.4 WEB TO DATABASE SERVER TRAFFIC FLOW 55

Figure 6.1 FCoE TOPOLOGY 61

Figure 7.1 VMWARE WITH STANDALONE SERVER ARCHITECTURE 66

Figure 7.2 VMWARE WITH BLADE SERVER ARCHITECTURE 67

Figure 7.3 TYPICAL SERVER CONNECTIVITY 68

Figure 7.4 BLADE SERVER ARCHITECTURE WITH FLIPPED-U DESIGN AND VSS 68

Figure 7.5 BLADE SERVER ARCHITECTURE IN PASS-THROUGH MODE 69

Figure 9.1 POD DESIGN TYPE-1 1GIG RACK GROUP 80

(9)

Tables

Table 1 Project Team Contact 12

Table 2 Current Project Site List 12

Table 3 Project Contact Information 12

(10)

Document Information

Author: Talha Hashmi

Change Authority: Cisco Advanced Services

Change Forecast: High

Template Version: 4.1

Review and Distribution

Organization Name Title

Molina Healthcare Amir Desai CIO, IT NOC & Operations Molina Healthcare Sri Bharadwaj Director, IT Infrastructure Molina Healthcare Shawn Shahzad Director, IT Transaction Services Molina Healthcare Larry Santucci Director, IT Infrastructure Molina Healthcare Joel Pastrana Manager, IT Infrastructure Molina Healthcare Sudhakar Gummadi

Molina Healthcare Rajeev Siddappa Manager, IT Networking & Telecom Cisco Advanced Services Dale Singh Project Manager

Cisco Advanced Services Talha Hashmi Lead Network Consulting Engineer Cisco Advanced Services Steve Hall L4 - L7 Network Consulting Engineer Cisco Advanced Services Eric Stiles Security. Network Consulting Engineer Cisco Advanced Services Damon Li SAN. Network Consulting Engineer Cisco Advanced Services Umar Saeed Delivery Manager

Modification History

Re

(11)

0.1 25-May-09 Talha Hashmi Draft Initial draft for [internal] Cisco review 0.2 10-July-09 Talha Hashmi Draft Released.

(12)

Introduction

Preface

This document known as the High Level Design (HLD) addresses the architecture & technology recommendations for building the new Data Center in Albuquerque, New Mexico for Molina Healthcare. The information in this document is in view of the technical requirements gathered in the CRD and the final BOM. Specific implementation details of each technology will be covered in the Low Level Design Document.

Audience

This document is intended for use by the Cisco AS and Molina Healthcare engineering teams. Technologies and recommendations decided here will dictate the implementation details on the Low Level Design document.

Scope

The scope of this document will cover the New Mexico Data Center design architecture, technology integration in reference to the requirements gathered on CRD and the hardware available represented on Bill of Material. The HLD will include the features, and functions that will satisfy the stated technical objectives of the project.

(13)

Assumptions

This document is focused on the design specific to the New Mexico Data Center (NM DC-2 solution). Any Cisco hardware and/or software information in this document is based on current performance estimates and feature capabilities.

References

(14)

Project Overview

Customer Description

Molina Healthcare, Inc., is among the most experienced managed healthcare companies serving patients who have traditionally faced barriers to quality healthcare- including individuals covered under Medicaid, the Healthy Families Program, the State Children's Health Insurance Program (SCHIP) and other government-sponsored health insurance programs. Molina has health plans in California, Michigan, New Mexico, Ohio, Texas, Utah, Washington, Missouri and Nevada, as well as 19 primary care clinics located in Northern and Southern California. The company's corporate headquarters are in Long Beach, California. Molina's success is based on the fact that it has focused primarily on the Medicaid and low-income population, and is committed to case management, member outreach and low-literacy programs.

More than 25 years ago, the late C. David Molina, MD, founded the company to address the special needs of Medicaid patients. Today, Molina carries out his mission of emphasizing individualized care that places the physician in the pivotal role of managing healthcare.

Project Overview

The primary business requirements for building the new Data Centre aims to consolidate and migrate the existing Molina Healthcare network infrastructure from

(15)

high risk earthquake zone, while also increasing network capacity, High Availability and Resiliency.

Project Scope

The scope of this project covers the planning, design, testing and implementation of the NM DC as described in the SOW. Cisco Advanced Services has been engaged with Molina HC engineering team in collecting the requirements, which were compiled and delivered as CRD. The 2nd deliverable under Phase 1 of this project is the High Level Design Document which will be followed by a Low Level Design Document in the second Phase.

Project Timeline – Phase 1

CRD Workshop: April 20th_{, 2009}

CRD Delivery: May 11th, 2009

CRD Acceptance: May 15th_{, 2009}

BOM (Draft): May 9th, 2009

BOM (Final): May 15th_{, 2009}

HLD (Draft): June 10th_{, 2009}

(16)

Project Team

The following Molina and Cisco resources are members of the project team.

Table 1 Project Team Contact Information

Name Title Organiza

tion

Email

Dale Singh Project manager Cisco dalising @cisco.com

Talha Hashmi Lead Network Consulting Eng Cisco [email protected]

Steve Hall L4 – L7 Consulting Engineer Cisco [email protected]

Damon Li SAN N.Consulting Engineer Cisco [email protected]

Eric Stiles Security N. Consulting Eng Cisco [email protected]

Project Sites

The following sites are currently in scope for the DCN project.

Table 2 Current Project Site List

Address City State Postal Code

One Golden Shore Long Beach CA 90802 (1GS) 5610 Turning Dr SE Albuquerque NM 87106 (ALB) 1500 Hughes Way Long Beach CA 90810 (HWS)

(17)

1. High Level Data Center Network Design

1.1 DCN Functional Blocks

Based on the requirements gathered in the CRD from Molina HC engineering team, attached below is a block level representation of the NM-DC-2 design. The diagram represents over all data center design segmented into different functional blocks explained below in detail.

(18)

ISP PARTNER ACCESS FW FW WAAS OUTSIDE INSIDE DMZ OB-MGT HUGHS WAY DR DC SAN ISP 1 ISP 2 MPLS VZ MPLS 2 GSS WAN ACCESS 1. BRANCHES

2. CORP USER 1. B2B Access

CORE L3 AGG L3-L2 ACC L2 SAN LB IPS WAN EDGE PNET ACC INET EDGE DC N MGT FW DMZ LB IPS

Figure 1.2 BLOCK LEVEL DESIGN

1. WAN EDGE

2. INET EDGE ( Internet Edge ) 3. PNET ( Partner Network ) 4. DCN ( Data Center Network ) 5. MNET ( Management Network )

Each block level design is further translated into High Level Design and technologies that match the required functions.

(19)

WAN EDGE WAN 1.1 WAN 1.2 WAN 2 WAAS TO MNET TO INET WAN FW WAN AGG L3 VZ

Figure 1.2 WAN EDGE DESIGN

1.1.1 WAN EDGE

WAN EDGE block will enable remote branch access termination through WAN MPLS circuits and further provide aggregation of WAN traffic. This block will host Security(Firewall, IPS, Proxy Services) and Wide Area Application Services for optimizing and securing the WAN traffic. The traffic passing through this block will be destined for either, 1. DC Core (DCN) for access to data & Voice services, 2. Internet (INET Edge) for access to the internet, 3. Management Network (MNET) for Network Operations Center 4. ( INET Edge for DMZ ) servers. The technologies used at the edge are 7206 VXR which will aggregate to CAT 6509. The WAN traffic has dedicated ASA for securing access to the DCN and WAAS appliance to optimizing the WAN traffic.



Recommenda tion

In alignment with Molina’s vision to have off shore NOC in addition to the remote NOC in Hughes Way. From security design prospective it will be feasible to have off shore NOC access provisioned through WAN Edge rather than through INET EDGE.



Not e

1. Service provisioning in this block is explained in detail under chapter 2 of this document.

(20)

DMZ CORE

INET EDGE TO PNET

GSS 1 Gig 1 Gig OUTSIDE 10 Gig 10 Gig INET FW VPN FW DMZ EDGE DMZ SRV

TO WAN EDGE TO DCN CORE

ISP -1 ISP -2

IDS _IDS

Figure 1.3 INET EDGE DESIGN

1.1.2 INET EDGE

INTERNET EDGE block will interface 1. Two ISP connections, 2. WAN Traffic through WAN EDGE, 3. Provide secure access to the Partner network and 4. The DCN Core. This block will also host the DMZ servers. Services hosted in this block are 1. Global Site Selector, 2. Firewall, 3. Server Load Balancing, 4. Intrusion Prevention.

Routers terminating the ISP will be 7206VXR which will aggregate to the CAT 6509 SW DMZ Core. The DMZ core will provision will security services for Partner and Internet access through dedicated ASA and to the DMZ servers through CAT 3750 SW.



Not e

1. Service provisioning in INET block is explained in detail under chapter 2 of this document.

2. DMZ Access SW will be repurposed from the existing DC environment. ( CAT 3750 are not DC class switches )

1.1.3 PNET

PARTNER access will be provisioned through INET EDGE block. Only P2P Layer 3 connectivity will be provisioned to the partner devices which will be managed by the partner.



Cauti

on Since the PNET devices are not owned by Molina and only L3 connections are provisioned. Proper capacity planning is essential to provision a scalable design.

(21)

If the predicted growth is high for such partner connections, it’s recommended to move the PNET design similar to the WAN EDGE and provide dedicated services. 10 Gig 10 G ig 10 G ig 10 G ig LAYER 3 DC CORE AGGREGATION SERVICE CHASSIS 10 Gig 10 G G 10 1 Gig 1Gig 4Gig 4 G ig 4 G ig 10 Gig 10 G_ig LAYER 3 ACCESS FC OE C O R E A G G R E G A T IO N 1 Gig Servers CORE

IDS

_IDS

P D

_{P D}

10 Gig Servers 10 G ig 10 G ig 10 G_ig LAYER 2 MOR TOR TOR SAN EDGE SAN CORE A C C E S S TO INET EDGE Figure. 4 DCN DESIGN

(22)

1.1.4 DCN

DCN Data Center Network is a 3 tired architecture which is based on Cisco best practice and also aligns with Molina’s requirements. The DCN block will interface the INET EDGE with 10 Gig connectivity which will provide access to WAN, PNET and INET. The DCN architecture is N+1 design which will have 10 Gig connectivity in each layer i.e. Core, Aggregation and Access.

The Core and the Aggregation layer will use the Nexus 7010 switch which can be virtualized to provide traffic segmentation for Production and Dev-Test environments as well as scalability for future growth. The selection of Nexus technology is based on Data Center class features recommended and discussed in detail under technology section. Additionally Cat 6509 will be used as a service chassis in the aggregation layer to provision services like SLB (Server Load Balancing) and Security. The service chassis design allows for modular approach to grow and scale services as required in the future.



Not

e For Functional details of each DCN layer please refer to section 1.2 DCN Design Principals. For feature recommendations please refer to chapter 3. The DCN access layer is designed to reduce structured cabling by consolidating the server connections within the rack using ToR technologies. The DCN access layer design is of two types:

Type 1: This design is to accommodate the existing servers that have 1 Gig NICs.

A C C E S S R A C K ₂₀G OS TBD 1:1.2 OS TO AGGREGATION NEXUS 2148 MDS 9124 NEXUS 5000 1 2 3 4 5 1 G ig

(23)

This design uses Nexus 2148 as ToR (Top of Rack) to provision 48 ports of 1 Gig connectivity in the Active/Active or Active/Standby NIC configuration. The Nx 5010 is used as MoR (Middle of Row) to aggregate the traffic from Nx 2148. Each Nx 2148 can provide upto 40 Gig of Bandwidth to the rack. The rack also consolidates the HBA connectivity through ToR MDS 9124 switches which provide 32 Gig of BW. The design represented above is termed as POD. Each POD constitutes 5 access racks.



Not e

Exception for SAN ToR: Based on the server requirements, some servers may connect directly to SAN Core. For SAN ToR (Edge SW) Design Please refer to the SAN section 1.1.5

The Nx 5010 in the MoR is used for Nx 2148 ToR access aggregation & provides 10 Gig uplink connectivity to the DCN aggregation layer. Based on the desired oversubscription ratio per POD and the availability of the 10 Gig ports, the middle rack can also accommodate limited no of 10 Gig servers. The management design is also segmented to support 5 racks with Nx 2148 as ToR Edge and 5010 as MoR as Spine.



Cauti on

The Nexus 2148 does not allow Port Channeling to the Host based on the Nx-OS available today. Therefore if there are any Server Blade Center Chassis requiring PC to the access port, the POD can provision limited connectivity by leveraging the first 8 ports of the Nx5010.

Nx 5010 provisions L2 connectivity only which is in line with Molina’s requirements.

Type 2: The design is to accommodate the New 10 Gig servers with FCoE capability.

A C C E S S R A C K TBDOS 1 0 G ig

(24)

Figure 1.6 DESIGNS FOR 10G SERVERS

The design uses Nexus 5010 as ToR to provision 10 Gig Ethernet with FCoE connectivity to the servers. Each rack has two dedicated Nx5010 with Ethernet uplinks to the Nexus 7k in the DCN aggregation layer and Fibre channel uplinks to the MDS 9513. The POD architecture not necessarily applies to this type as compared to type one. But from the management prospective the architecture remains the same i.e. one spine per 5 racks.



Not e

For Functional details of each DCN layer please refer to section 1.2 DCN Design Principals. For feature recommendations please refer to chapter 3

1.1.5 SAN

Core 1 Edge 1 Edge 2 Fabric A Fabric B Edge 40

...

Core 2 Edge 3 Edge 39 Core 1

Edge 1 Edge 2

...

Edge 40

Core 2

Edge 3 Edge 39

Figure 1.7 SAN CORE EDGE DESIGN

The Storage Area Network (SAN) is a two tier core edge topology. It is based on Molina’s requirements and Cisco’s best practices.

The SAN Core-Edge topology consists of two redundant fabrics. Each fabric has two core directors and multiple edge switches. In the core-edge architecture the core directors support all the storage or target ports in each fabric as well as ISL connectivity to the edge switches. The core director’s act as the central insertion point for FCIP SAN extension for replication between sites and SANTap for network based traffic splitting. This topology provides consolidation of storage ports at the core.

(25)

The hosts connect to edge switches. The edge switches are connected to the core via ISL trunks. Since storage is consolidated at the core switches in this topology, this design can supports advanced SAN features like IVR, SME, DMM, SANTap, FCIP and virtualization on the core switches.

This topology also provides deterministic host to storage over subscription ratio. This is a future proof architecture for FCoE. When Molina is ready to deploy FCoE, the edge switches can be swap out for Cisco Nexus 5000 FCoE switches without additional cables and re-wiring.

MNET MNET - FW TO WAN EDGE MGT CORE MGT SPINE TOR EDGE 1G 10/100/1000 R1 R2 R3 R4 R5

Figure 1.8 MNET DESIGN

1.1.6 MNET

MNET Management Network is designed to provide true out of band management to the network devices and servers in the DC. The Management network will host the management technologies. The access to the management will be secured through dedicated firewall between management network and other network segments. The management network was designed based on inputs from Molina engineering team which uses Nexus 2148 for 1gig connectivity and CAT 2960 for 10/100/1000 (for server connectivity ) as TOR on each rack. The aggregation of which will be done at the Nuxus 5010 (Spine SW). The aggregation design was segmented to 5 racks as preferred by Molina (except for Network racks which will be a group of 8 racks). Further all Spine switches will aggregate to the Management Core.



Cauti

on There is limited redundancy in the management network as the Nx2148 which acts as a remote line card can only home to one 5010 as per the technology available today. Therefore the network dose posses a single point of failure in two cases. 1. If the spine switch fails, the connectivity to the entire TOR connecting to that spine will be lost. 2. If the TOR (2148 / 2960) fails, only connectivity to that rack will be lost. However there is N+1 redundancy on management core. The single point of failure can be eliminated by dual homing the ToR into 2 spines which is supported in a scheduled in July. If that route is selected there will be additional cabling required to support this

(26)

redundancy.

1.1.6.1 Out of Band Management

Out of Band Management refers to having dedicated physical facilities connecting the NMS and the managed equipment (network devices, managed servers, etc). This ensures that the NMS traffic is not interfering with application traffic, and vice versa. Indeed, the primary value is that the NMS traffic can be immune to any problems on the application links, and can be used to repair application link problems, either due to congestion, broken devices/facilities, or configuration error. From the NMS product perspective, the out of band provisioning is handled simply by the IP addressing



Not e

To keep the standard design, all access & network racks will be provisioned with management switch as TOR. The TOR SW may contain more physical ports then required specially in the Network racks as the No of devices will be less as compared to access racks. However this decision was reached in consideration to Molina’s requirement to minimize inter rack cabling.

1.2 DCN Design Principles

This section describes the overall functioning design components and the design rules. The design proposed for Molina HC Data center is based on Cisco’s multi-layer hierarchical architecture and best practices.

1.2.1 Multi-layer Tiers

The hierarchical three-tiered network design is the preferred architecture for most Local Area Networks (LANs). The hierarchical design principle is applicable to enterprise data center network. In a data center network, the three-tiered architecture is comprised of the Access layer, the Aggregation layer, and the Core layer. Each layer of the three-tiered network design has different responsibilities that permit some degree of flexibility for aggregating servers, configuring network redundancy, and supplying services to the network. The multi-layer architecture provides a secure, scalable, and resilient infrastructure. Molina Data Center Network design is based on hierarchical three tier architecture.

1.2.1.1 DC Core

This layer is purely L3 and provides high speed routing between the different aggregation layer switches. In addition DC Core layer also provides connectivity to INET EDGE. Core of the network will be based on Nexus 7000 platform. The Core will

(27)

be virtualized using the VDC technology to provide path isolation between Production and Test Dev environments.

1.2.1.2 Service & Distribution

Distribution/Aggregation layer comprises of layer 3 switches providing aggregation point and layer 2 connectivity to access layer switches and also providing layer 3 connectivity to the DC core. The services chassis are attached to the aggregation switches. Service chassis are designed to host load balancing & security service. Distribution layer will be built on Nexus 7000 platform with a pair of CAT 6500 as the services chassis. The distribution switches will be virtualized using the VDC

technology to provide path isolation between Production and Test Dev environments.

1.2.1.3 Access Layer

Access Layer will provide the capability for 10 Gig & 1 Gig connectivity and FCoE with Nexus 5k and 2k technologies. Segmented POD architecture is proposed for

provisioning deterministic oversubscription and bandwidth allocation. vPC (virtual Port Channel) technology can be leveraged to reduce over subscriptions.



Not

e For Virtualization technologies please refer to chapter 7.

1.3 SAN Design principles:

This section details overall SAN design components and design rules. The design proposed for Molina SAN is based on Cisco’s SAN core edge architecture and best practices.

1.3.1 SAN Core

There are four MDS 9513 as core directors, two MDS 9513 directors in each fabric as high density and high performance core switching. Servers and storage that require high bandwidth and high performance connect directly to the MDS 9513 core switches via home run links. Blade servers with 9124e also connect directly to the core as well. Edge switches connect to the core switches with multiple ISLs to maximize performance from the host to the storage device. The current configuration recommends 4 x 4G ISLs for edge to core links. Total of 16G is available from the edge to the core a per fabric. Each rack has a total of 32G uplink capacity on both fabrics. The core directors are enabled with NPIV for multiple logins per N port to support NPV edge switches. Edge switches runs in NPV mode. NPV reduces domain ID usages and eliminates most of the configuration on the edge switches. After the initial switch setup, NPV enabled switches will be treated as an HBA. The edge switches will not run any fabric services such as fabric login and name servers. Zoning and other operational activities will be performed at the core directors, resulting in minimum management overhead.

(28)

The core directors have 18+4 cards installed for SANTap and FCIP. It is not determined which direction Molina will take for data replication across sites. The 18+4 cards will support SANTap or FCIP at the time of deployment.



Not

e Refer to replication section below for more information.

1.3.2 SAN Edge

Edge switches connect to the core switches with multiple ISLs to maximize performance from the host to the storage device. The current configuration recommends 4 x 4G ISLs for edge to core links. Total of 16G is available from the edge to the core per fabric. Each rack has a total of 32G uplink capacity on both fabrics. Dual 9124 switches are located in each rack function as edge switches to provide cost-effective connectivity to servers. Most production servers should connect to the edge switches.

Fabric Manager will be deployed for SAN management. Fabric Manager and Device Manager Offer fabric and device management of the SAN fabric.

1.3.3 SAN Connectivity

1.3.3.1 Inter-Switch Link (ISL) Connectivity

The 24 port linecard offers dedicated full 4Gb/s per port. There are 8 ISLs between core directors in each fabric. There are 4ISLs between the core director and the edge switches with 16 4GB/sec to the uplink director. NPIV is enabled on all core directors and NPV is enabled on all edge switches.

High Availability ISL design requires the following:

• More than one ISL between switches. This ensures switch to switch connectivity is maintained in the event one ISL fails due to a cable, line card, and port group issue.

• ISLs between switch should be on different line cards. This ensures that switch to switch connectivity is maintained in the event of a line card failure

• Multiple ISLs on the same line card should be placed into separate port groups of a line card. This allow for an even distribution of bandwidth on a line card. In addition, a port group failure will not affect all ISLs.

• ISL should be bundled into a port-channel to allow for bandwidth management. Channels allow for non-disruptive changes to the number of ISLs. Port-channels provide load balancing.

(29)

• ISL ports should be placed into VSAN 1. This is a best practice, as VSAN 1 can never be deleted. However, to meet this requirement, VSAN 1 should never be placed into a suspended state.

• ISLs require dedicated bandwidth. Between switches, this is typically 4Gb. • ISLs to bladecenter should follow the requirements. However, as a starting

point, two by 4Gb ISLs between the bladecenter and its core switch has proven to be sufficient. With both an A and B fabric to connect to a bladecenter, this will provide 16 Gb of bandwidth. This is based on two, 4GB links per fabric.



Cauti on

Cisco does not provide embedded fiber channel switches for blade switches. There are four HP c-class blade switches in Molina that have Brocade switches. Molina will have to procure the Cisco 9124e embedded blade switches from the server vendor.

• Fabric Manager Server’s Performance Manager should be monitored to determine whether ISLs are under of over sized. ISLs should be increased if the trending and prediction queries in FMS show it is required.

• The description field on the ISL port should contain the source switch name and port and the remote switch name and port information.

ISL  [Source Switch name-physical port – destination switch name-physical port]

E.g. switch1-fc1/1 to switch2-fc1

1.3.3.2 Host Connectivity

The SAN directors for Molina are configured with a combination of 24 port and 18+4 line cards to provide and FC ports, FCIP and SANTap support as required. Each rack will have two 9124 switches with uplinks to the core directors. The 9124 ports are 4 Gb/sec line rate with a maximum of 5:1 oversubscription for the uplink. High performance servers and tape devices should connect directly to the core directors for full 4 Gb/sec bandwidth.

It is recommended for the ease of troubleshooting and management to leave all the unused ports in a shutdown state and to turn on ports as and when they are required. It is also recommended to have detailed descriptions for each port as they are enabled.



Recommenda

tion The descriptions for the server/host HBA connected ports on the switch should include the server name, HBA vendor and model and HBA instance on the server.

Host  [hostname-HBAvendor-hba instance] E.g. server1-hba0

(30)

The Storage ports and the tape libraries will be connected to the 24 port line cards. The ports on this card can operate at 1/2/4 Gb/sec. At all speeds, the ports operate at full rate. Media servers should connect to the same director as tape devices to minimize traffic traversing the ISL between the directors.

It is recommended for the ease of troubleshooting and management to leave all the unused ports in a shutdown state and to turn on ports as and when they are required. It is also recommended that detailed descriptions for each port be defined as they are enabled.



Recommenda tion

The descriptions for the storage target port connected ports to the SAN switch should include the Array model, Array serial number and the port identifier.

Storage  [Storage array serial number and port id] E.g. 8300-3c Storage ports should be added to line cards in a round-robin fashion. Storage ports would have the most potential to run their line rate.

1.3.3.4 Tape Backup Architecture

Tape SAN Design

Core 1 Edge 1 Edge 2 Fabric A Fabric B Edge 40

...

Core 2 Edge 3 Edge 39 Core 1

Edge 1 Edge 2

...

Edge 40

Core 2 Edge 3 Edge 39 Media server Tape libraries Media server Tape libraries

Figure 1.9 TAPE SAN DESIGN

As part of the unified SAN architecture, tape backup traffic will be integrated into production SAN MDS switches. Separate backup VSANs will be created to segregate tape backup traffic. The core edge SAN design accommodates VTLs and Tape devices connecting directly to the core switches. The current design allows up to 60 VTLs and/or tape devices for line rate throughput of 4Gb/s per fabric. Tape devices are not dual-homed, but can connect to either fabric. The backup servers can connect to multiple directors to maximize port utilization on both fabrics. Tape devices and media server pairs should be connected to the same SAN director to reduced ISL traffic and highest throughput.

(31)

1.3.3.5 Data encryption

Data encryption is performed on the tape devices with Quantum i500 tape library. Quantum Encryption Key Management (Q-EKM) system is used for key management. Molina currently does not have requirements for data at rest. Symantec PureDisk provides 256-bit AES encryption for data at rest and data in transit. Cisco does offer Storage Media Encryption (SME) on the SAN network. The current backup environment is not expected to change with MDS SAN design.

1.2 High Availability & Resiliency

1.4.1 High Availability

The objective of designing a highly available network is to reduce recovery time in case of any failure and also to maximize resource usage. High available network should have predictable normal and failure behaviour. To design a high available network redundancy should be used where appropriate and utilize a multi-tier architecture design. A crucial component of the network is to fine tune network parameter and utilize advance features ensuring availability.

Each element in the Molina data center network is built with high-availability in mind. A highly available supervisors with fast failover protocol such as SSO/NSF is used in Cisco Catalyst 6500 plus all interconnect are configured in port channel form such that a single link failure does not cause an outage or a reconvergence (may that be Layer2 or Layer3) event. The V-shaped interconnect design principles are used such that multiple link/devices does not cause an outage, which is in accordance to Molina HC network availability requirement.

The design includes the proposal to use HSRP on the aggregation switches for those VLANs that are trunked to the aggregation or access switches. The aggregation switches provide layer 3 terminations (interface VLAN) for the access switch VLANs, HSRP configured on each switch pair will provide first hop redundancy for hosts connected to these VLANs.

The security infrastructure has been designed with a focus on high available and single points of failure avoidance. We have achieved this resilience by designing the infrastructure around segmentation of functionality and physical hardware separation. As you continue to review this document you will notice we have divided the infrastructure into multiple zones based on function and within each zone high availability is achieved by providing active/standby pairs for the security enforcement points. This provides sub-second failover and maintains TCP sessions to provide seamless session redirection to the standby device.

The server load balancing solution utilizes stateful connection tracking so existing user traffic is unaffected in the event of a device fail over. The fault tolerant mechanism of the server load balancers can also interact with the switching infrastructure (by tracking interfaces and HSRP groups) to make optimal decisions on which device is active.

(32)

Redundant SAN fabrics are utilized with this design, providing dual path connectivity to the host and storage devices. The SAN core is composed of two MDS 9513 directors on each fabric. The high availability features of the MDS 9513 include non-disruptive code upgrades, hot insertion and removal of blades, dual Supervisor modules with stateful failover, dual power-supplies. New edge switches can be added to the topology non-disruptively and without major changes to the architecture.



Not e

Specific Technology recommendations & best practices are covered under chapter 3,4, 5 & 6

1.4.2 Resiliency

The biggest risk to a data center network is Spanning-tree loop. A spanning-tree loop in any part of the network can cause outage for the entire network. L2 access layer switches are required to support stateful devices such as load balancers and server NIC teaming. Use of the recommend technologies such as UDLD, CoPP etc can further reduce these risks.

The infrastructure and security design provide redundancy at multiple levels for a robust, secure, and resilient environment. The security infrastructure achieves this by providing hardware failover and maintains session state during the failover. In addition to failover the segmentation design provides each zone protection against cross zone failure.

The SAN network is a fully redundant architecture consists of two fabrics. Core-to-core directors and core-to-edge switches connectivity are bundled in a port channel. Link disruptions within the port channel will not affect data flow. Each host is connected separately to the redundant fabrics. Storage traffic continues to flow even in the event that one of the links is unavailable.



Not e

Specific Technology recommendations & best practices are covered under chapter 3,4, 5 & 6

(33)

2. Data Center Services

DMZ Services

The DMZ zone requires 4 separate functions to Molina’s infrastructure. They are: • Internet Services – Internet Outbound traffic management

• Partner Services – Partner communications • DMZ Services – Internet Inbound traffic management • The DMZ edge environment

• WAN Services – Remote office communication and Remote office Internet traffic management

2.1 INET Services

2.1.1 Security

Internet Services function is designed to manage communications from the data center to unknown destination on the Internet. The focus of this security enforcement point is to validate outbound connections against internal security policy. Given various governance requirements applied to Molina’s infrastructure it is required to manage all outbound communications. This enforcement point will not permit any inbound originating traffic.

AccessPath, AtmDirector, Browse with Me, CCDE, CCIP, CCSI, CD-PAC, CiscoLink, the Cisco NetWorks logo, the Cisco Powered Network logo, Cisco Systems Networking Academy, Fast Step, Follow Me Browsing, FormShare, FrameShare, GigaStack, IGX, Internet Quotient, IP/VC, iQ Breakthrough, iQ Expertise, iQ FastTrack, the iQ logo, iQ Net Readiness Scorecard, MGX, the Networkers logo, Packet, RateMUX, ScriptBuilder, ScriptShare, SlideCast, SMARTnet, TransPath, Unity, Voice LAN, Wavelength Router, and WebViewer are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, Discover All That’s Possible, and Empowering the Internet Generation, are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert Logo, Cisco IOS, the Cisco IOS logo, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Enterprise/Solver, EtherChannel, EtherSwitch, FastHub, FastSwitch, IOS, IP/TV, LightStream, MICA, Network Registrar, PIX, Post-Routing, Pre-Routing, Registrar, StrataView Plus, Stratm, SwitchProbe, TeleRouter, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries.

(34)

Inside Network

Outside Network

DMZ CORE Internet Firewalls Internet Edge Routers Internet Data Center Primary Path Secondary Path

Figure 2.1: INTERNET EDGE FIREWALL

The DMZ services zone focuses on providing Molina services to external customers. These services include web, e-portal, ftp, and others. They represent business critical functions and increased security concerns. The design of this infrastructure is to provide a dedicated pair of hardware, management inbound and outbound

communications, and isolate services to decrease exposure to security breaches. The secure infrastructure will be managed by a firewall and intrusion detection devices.

Internet Edge Routers

Internet

DMZ Firewalls

(35)

Figure 2.2: DMZ FIREWALL

Partner services provide a direct link to Molina business partners. These partners participate and assist Molina in day to day operation and represent a business critical connection/communications. The policy requirements for the partner zone differ greatly from all other zones and in this design we have segmented their traffic to a dedicated firewall to manage the security enforcement policy. The firewall

infrastructure will segregate partners by interface and provide intrusion detection systems. Partner 1 Partner 2 Partner 3 Partner 4 Partner Firewalls DMZ Core

Figure 2.3: PARTNER FIREWALL

2.1.2 Load Balancing

ACE server load balancers will be utilized in both the DMZ and inside network

aggregation. These are service modules that utilize the fast backplane and advanced routing of the catalyst 6500 switching platform. The ACE server load balancers have the ability to virtualize into many virtual modules, allowing separation of hardware resources and configuration. This allows the same device to perform in different functions .

2.2 WAN Services

2.2.1 Security

WAN Services provides remote offices and agencies direct access to data center resources and an internet security enforcement point for proxy services. The WAN security infrastructure will include firewalls and intrusion detection devices. The

infrastructure will provide a direct internet link for the remote offices and segment this traffic from traversing the data center routing core.

Molina-HLD-V1.0(Latest)

Cisco Advanced Services

MOLINA HEALTHCARE

Data Center Networking

High Level Design Document

V1.0 (Draft)

MOLINA HEALTHCARE

Data Center Networking

High Level Design Document

V1.0 (Draft)

Contents

Figures

Tables

Document Information

Review and Distribution

Modification History

Introduction

Preface

Audience

Scope

Assumptions

Related Documents

References

Project Overview

Customer Description

Project Overview

Project Scope

Project Timeline – Phase 1

Project Team

Project Sites

1. High Level Data Center Network Design

1.1 DCN Functional Blocks

1.1.1 WAN EDGE





1.1.2 INET EDGE



1.1.3 PNET



IDS

IDS

P D

P D

P D

P D

1.1.4 DCN









1.1.5 SAN

...

...

1.1.6 MNET



1.1.6.1 Out of Band Management



1.2 DCN Design Principles

1.2.1 Multi-layer Tiers

1.2.1.1 DC Core

1.2.1.2 Service & Distribution

1.2.1.3 Access Layer



1.3 SAN Design principles:

1.3.1 SAN Core



1.3.2 SAN Edge

1.3.3 SAN Connectivity

1.3.3.1 Inter-Switch Link (ISL) Connectivity



1.3.3.2 Host Connectivity





1.3.3.4 Tape Backup Architecture

...

...

1.3.3.5 Data encryption

1.2 High Availability & Resiliency

1.4.1 High Availability



_IDS

_{P D}