High-Availability Enterprise Network Design

(1)

1

505

0911_04F9_c3

High

-

Availability

Enterprise Network

Design

haviland

(2)

2 505

Staying On Target

HA Focus

vs

Distractions!

“Flat networks

are easier”

beware!

Five

nines is

job one!

Inherited

complexity

hard to purge

The latest

cool stuff

older is more

stable

“Variety”

of vendors,

protocols,

designs, etc.

“Feature

rich”

let’s use all

the knobs!

Change is hard,

sometimes $$$

(3)

3 505

HA Features of the Catalyst 6500

Consider for Backbones & Server Farms

✔

Fabric Redundancy

switch fabric module

in CatOS 6.1

✔

Supervisor Redundancy

HA feature in CatOS 5.4.1

stateful recovery

image versioning on the fly

✔

MSFC Redundancy

config-sync feature

IOS 12.1.3 CatOS 6.1

(4)

4 505

Thinking Outside the Box

For HA/HP design

“outside the box”

☛

the logical design

is critical

☛

network features

& protocols

☛

geophysical

diversity is powerful

Inside:

“HA”,

RAID,

UPS,

MTBF,

etc.

(5)

5 505

Dramatis

Personae

Our Cast of Symbols

✔

Links

GE, DPT, SONET, etc.

✔

_{L2 switching}

L2 forwarding in hardware

✔

L3 switching

L3/L2 forwarding in hardware

✔

_Routing

L3 forwarding (SW or HW)

✔

Control plane = IOS

routing protocols & features

✔

_{QoS where required}

✔

_{Application intelligence}

Catalyst 4000

Cisco 7500

Cisco 12000

Catalyst 6500

(6)

6 505

Client

Blocks

Distribution L3 Access L2

HA Gigabit Campus Architecture

survivable modules + survivable backbone

Backbone

Server

Block

Server Farm Distribution L3 Access L2 E or FE Port GE or GEC Ethernet or ATM Layer 2 or Layer 3

☛

Define

the mission

critical parts

first!

(7)

7 505

High Availability Design

Why a

Modular ABC Approach

✔

_{Many new products, features,}

technologies

✔

HA and HP application operation is

the goal

✔

_{Start with modular, structured}

approach

(the “logical” design)

(8)

8 505

Price per 10/100

Catalyst 2912G Catalyst 2948G Catalyst 2980G

24

24 10/100 Ports

10/100 Ports

Gigabit Ports

24-500+

24-350+

3-38+

8-64+

Catalyst 5XXX

32-96

6-12

Catalyst 4XXX

$100 $200 $250 $300 $350

Switching Capacity

20 Mpps20 Mpps Up to 72 MppsUp to 72 Mpps Up to 150 MppsUp to 150 Mpps

Backplane

24 Gbps24 Gbps 1.2-3.6 + 10Gbps1.2-3.6 + 10Gbps 250+ Gbps250+ Gbps

New

Modules

Catalyst 6XXX

Design the Solution

Then Pick the Products

New

Modules

(9)

9 505

HA Design Reality Check!

Assume Things Fail

-

Then What?

✔

Networks are complex

✔

Things break, people make mistakes

✔

What happens if a failure occurs?

✔

_{Simple, structured, deterministic design}

required for fast recovery

✔

The “tradeoffs”

(10)

10 505

Layer 2

Access Access Distribution Distribution Building Building Core L3 Core L3 Server Server Distribution Distribution Server Farm Server Farm

Layer 3

3 2 1 5 6

Branches

WAN WAN backup 4

Network Recovery

How Long? What Happens?

(11)

11 505

Failure

Scenario

Failure

Scenario

1,2 server

3,4 uplink

5,6 core

dual-path L3

EtherChannel

L3 routing

L2 general

DPT

1,2 server

3,4 uplink

5,6 core

dual-path L3

EtherChannel

L3 routing

L2 general

DPT

Recovery

Mode

Recovery

Mode

Recovery

Time

Recovery

Time

Server NIC

HSRP (& UplinkFast)

HSRP track

alternate path used

channel recovery

EIGRP or OSPF

L2 spanning tree

IPS

Server NIC

HSRP (& UplinkFast)

HSRP track

alternate path used

channel recovery

EIGRP or OSPF

L2 spanning tree

IPS

< 2 seconds

tune to 3 seconds

< 2 seconds

< 1 second

depends on tuning

tune (up to 50 seconds)

50 milliseconds

< 2 seconds

tune to 3 seconds

< 2 seconds

< 1 second

depends on tuning

tune (up to 50 seconds)

50 milliseconds

Network Recovery Times

If You Follow the Rules

(12)

12 505

Design for High Availability

How to Build Boring Networks!

✔

_{The Concepts}

✔

_{The Rules}

✔

_{Design Building Block}

✔

_{Design Backbone}

(13)

13 505

HA Network Design Concepts

thinking outside the box

1)

Simplicity & Determinism

2)

Collapse the Sandwich

3)

Spanning Tree Failure Domain

4)

Map L3 to L2 to L1

5)

Scaling and Hierarchy

6)

ABCs of Module + Backbone

Design

(14)

14 505

1) Simplicity and Determinism

reducing the degrees of freedom

✔

Every Choice Affects Availability!

✔

Determinism or Flexibility?

✔

Would you support 27 desktop environments?

✔

Would you support 13 network vendors?

✔

Would you use 57 varieties of Cisco IOS?

Flexible

Complex

Varied

Simple

Structured

Deterministic

“HA Continuum”

(15)

15 505

Traditional

Model

Fiber

SONET

Big Fat Pipe

• Lower equipment cost

• Lower operational cost

• Simplified architecture

• Scalable capacity

Optical

Internetworking

Fiber

IP

FR/ATM

IP

2)

Collapse the Sandwich

route IP over glass

Service

Traffic

Eng

Fiber

Mgmt

(16)

16 505

3

3 )

₎

Minimize the Failure Domain

_{Minimize the Failure Domain}

public enemy number one

Where should root go?

What happens when

something breaks?

How long to converge?

Many blocking links

Large failure domain!

Broadcast flooding

Multicast flooding

Loops within loops

ST from heck

Times 100 VLANs?

avoid highly meshed, non-deterministic large scale L2 = VLAN topology

Building 1

Building 2

(17)

17 505

4)

Map L3 to L2 to L1

_{Map L3 to L2 to L1}

✔

_{Easier administration & troubleshooting}

Clients in subnet 10.0.55.0

VLAN 55

wiring closet “55” on floor 55

access switch “55”

interface VLAN 55

all match and life is good

go fishing with your kids

10/100 BaseT GE or GEC

(18)

18 505

5) Scaling and Hierarchy

Strong hierarchies

like telephone

system and

Internet segment

addressing and

therefore scale

U

C

N

U

C

N

U

C

N

C complexity U unmanageable N number of devices

Flat L2 Ethernet is

easy but does not

scale

ATM LANE is

logically flat, scales

as N squared

(19)

19 505

6)

Building Block &

Backbone Design ABCs

WAN

Ecommerce

Solution

PSTN

Distribution

Core

LAN Access

Distribution

Server Farm

Internet

A design bb

B design BB

C connect bb to BB

Divide and conquer

Cookie cutter

configuration

Deterministic

L3 demarcation

WAN Access

(20)

20 505

7) Four Square Network Redundancy

or the Four Corners Problem

One Chassis

Two Chassis

One

Supervisor

Two

Supervisors

Simplest

No Redundancy

Most Complex

Belt and Suspenders

GeoPhysical

Effective

When space

is limited

“HA”

L3

(21)

21 505

Dos and Don’ts for HA Design

1)

Eliminate STP Loops

2)

L3 Dual-Path Design

3)

EtherChannel Across Cards

4)

Workgroup Servers

5)

Use HSRP Track

6)

Passive Interfaces

7)

Issues with Single-Path Design

8)

Oversubscription Guidelines

9)

HA for single attached servers

10)

Protocol Tradeoffs

(22)

22 505

Rule 1) Eliminate STP Loops

in the backbone and mission critical points

No blocking links to

waste bandwidth

Avoids slow STP

convergence

Very deterministic

Routed links not VLAN

trunks

L2 Gigabit switch in

backbone

subnet X = VLAN X

Too many cooks spoil the broth

L3 control is better

X.2 X.3

X.1

Root

VLAN X

(23)

23 505

Rule 2) Dual Equal

-

Cost Path L3

✔

Load balance - don’t waste bandwidth

unlike L1 and L2 redundancy

✔

Fast recovery to remaining path

detect L1 down & purge - about 1s

✔

Works with any routed fat pipes

Path A

Path B

Destination

network X

Equal cost

routes to X

Path A

Path B

(24)

24 505

Rule 3)

EtherChannel

Across Cards

Increased availability

✔

Sub second recovery

✔

Spans cards on 6500

✔

Up to 8 ports in channel

Small complexity increase

✔

Single L2 STP link

✔

Single L3 subnet

(25)

25 505

Rule 4a) Connect Workgroup Server

With no L2 recovery path, what happens if link

breaks ….

Workgroup server X.100

attached to distribution layer

L2 path to client X.1

Client X.1

VLAN X in purple

includes clients

and workgroup

servers attached

at different places.

A

B

C

Links to core

Link CB

breaks ….

(26)

26 505

Rule 4b) Connect Workgroup Server

• Subnet X now discontiguous

• Incoming traffic gets dropped

Workgroup server X.100

attached to distribution layer

L2 path to client X.1

Client X.1

Routers A & B continue to

advertise reachability of

subnet X ...

A

B

C

X.1 not

reachable

X.100 not

reachable

(27)

27 505

Rule 4c) Connect Workgroup Server

• Introduce L2/STP redundancy

• Adds a loop (band-aid fix)

Workgroup server X.100

attached to distribution layer

L2 path to client X.1

Client X.1

•VLAN trunk AB forms L2 loop

•recovery path for STP

•prevents black hole

A

B

(28)

28 505

Rule 4d) Connect Workgroup Server

Real Lessons:

☛

_{Enterprise Server Farms}

are better

☛

_{L3 demarcation is better}

☛

_{Example of why extended}

(29)

29 505

Rule 5a) Use HSRP Track

• Review - Hot Standby Router Protocol

• Fast recovery can be tuned to 3s or less

X is M.100 HSRP Primary Priority 200 Y ( becomes M.100) HSRP Backup Priority 100

Z

Router X acts as gateway

router for subnet M, IP address M.100. If link Z fails router Y will take over as M.100 gateway with same MAC address

Subnet M

(30)

30 505

Rule 5b) Use HSRP Track

• Track extends HSRP to monitor links to backbone

• Ensures shortest path - best outbound gateway

Track interface A - lower priority 75 Track interface B - lower priority 75 HSRP triggers if both A and B lost

10/100 BaseT GE or GEC X is M.100 HSRP Primary Priority 200 Y ( becomes M.100) HSRP Backup Priority 100

Z

Subnet M hosts M.1 M.2 M.3

A

B

(31)

31 505

Rule 6a) Use Passive Interfaces

• L3 switches X & Y in distribution layer

• 4 VLANs per wiring closet

• 10 wiring closets

X Y

ABCD EFGH IJKL MNOP

… Ten total

Wiring closet switch Distribution

switch

(32)

32 505

Rule 6b) Use Passive Interfaces

• **What X and Y see is 4*10=40 routed links**

• Increased protocol overhead & CPU

X Y A C B D E F G Etc. A.1 C.1 B.1 D.1 E.1 F.1 G.1 Etc. A.2 C.2 B.2 D.2 E.2 F.2 G.2 Etc.

(33)

33 505

Rule 6c) Use Passive Interfaces

☛

Turns off routing updates & overhead

☛

Leave two routed links for redundant paths

☛

CDP, VTP, HSRP etc. still function on all links

X Y A C B D E F G Etc. A.1 C.1 (passive) B.1 (passive) D.1 (passive) E.1 F.1 (passive) G.1 (passive) Etc. A.2 C.2 (passive) B.2 (passive) D.2 (passive) E.2 F.2 (passive) G.2 (passive) Etc.

(34)

34 505

Rule 7a) Issues With Single Path

Designs

✔

L3 engine MSFC on

core-X reloads

✔

Lights are on but

nobody home - HSRP

does not recover

✔

Remove passive

interface to wiring

closet subnets A, B

✔

Provide longer routed

recovery path

Single path to core GE Subnet A Subnet B X HSRP primary Core L3 Access L2 Y New, longer outbound routes

Outbound case ...

(35)

35 505

Rule 7b) Issues with Single

-

Path

Design

✔

Recovery must take

place in both

directions

✔

Routing protocol

recovers longer route

from X to subnets A, B

✔

Therefore dual-path L3

is better & faster than

single-path

Single path to core GE Subnet A Subnet B X HSRP primary Core L3 Access L2 Y New, longer routes to A, B

Inbound case ...

(36)

36 505

Rule 8a)

Oversubscription

Guidelines

✔

Oversubscription part of

all networks - not bad

✔

Non-blocking switches

do not mean a

non-blocking network

✔

You determine the

amount of “blocking”

GE GE Non-blocking design GE GE Blocking design 2:1 GE

(37)

37 505

Rule 8b)

Oversubscription

Guidelines

✔

Oversubscription rules

of thumb work well

✔

20:1 at wiring closet

✔

Less in distribution and

server farm

✔

QoS required IFF

congestion occurs

✔

Protect real time flows

at congested points

n:1 20:1 Core L3 use non-blocking switches Dual-link GEC 200 100BaseT GE 8 uplinks Distribution L3

(38)

38 505

Rule 9) Dual Supervisors

HA for Single Attached Servers

✔

Single point of failure

✔

Dual supervisors - fast stateful recovery

✔

No increase in complexity

Single attached server

mission critical application

HA dual supervisors

Catalyst 6XXX

(39)

39 505

Rule 10)

Protocol Tradeoffs

Automatic or Manual Configuration

✔

_{Configuration up front rather}

than CPU overhead later, for

example:

➙

set VTP mode transparent

➙

set/clear VLANs for each trunk

➙

set trunks on or off

➙

set channel on or off

✔

_{Choose flexibility or}

(40)

40 505

Rule 11)

UniDirectional

Link

Detection

✔

UDLD detects mismatch when physical layer

checks out OK

✔

Prevents various failure conditions including

crossed wiring

Tx Fiber

Rx Fiber

The lights

are on,

BUT …..

(41)

41 505

Building Block Means Survivable

Self

-

contained Backbone

✔

Autonomous Survivability

Unit - HSRP

✔

L3 Broadcast Multicast

demarcation

✔

Cookie cutter configuration

✔

L3 Demarcation of failure

domain

✔

Simple, repeatable,

deterministic

✔

Redundancy adds 15% cost

at mission critical points like

server farm

L2

L3

ASU

delimits

failure

domain

(42)

42 505

Building Block Templates

Use “As Is” or Combine

1)

Standard Model

simple, structured

2)

VLAN Model

more flexible

3)

Large Scale Server Farm

Model

accommodate dual NIC

4)

Small Scale Server Farm

Model

(43)

43 505

1) Standard Building Block

no loops

-

no STP complexity

HSRP Primary Subnets/VLANs 10, 12, 14, 16 HSRP Primary Subnets/VLANs 11, 13, 15, 17 Access L2 root switch VLAN 10/11 Subnet 10 Subnet 11 GE/GEC VLAN Trunks 10/100 BaseT

GE or GEC Dual Path with Tracking

Subnet 12 Subnet 13 Subnet 14 Subnet 15 Subnet 16 Subnet 17

Highly Deterministic

L1 maps L2 maps L3

No blocking links

Shortest path always

Not “flexible”

(44)

44 505

2) VLAN Building Block

make L2 design match L3 design

All

VLANs

terminate at L3 boundary

STP root VLANs 10 12 14 16 HSRP primary subnets 10 12 14 16 STP root VLANs 11 13 15 17 HSRP primary subnets 11 13 15 17

L2

L3

All VLANs All Subnets GE/GEC VLAN Trunks

Dual Path with Tracking

All VLANs All Subnets All VLANs All Subnets All VLANs All Subnets L2 Path 10/100 BaseT GE or GEC

More flexible

FO forwarding odd BE blocking even etc. FE BO FO BE FE BO FO BE FE BO FO BE FE BO FO BE

L2

L3

Uplink-Fast

(45)

45 505

3) Large

-

Scale Server Farm

Building Block

Dual-NIC Server

Example Fault Tolerant Mode (FTM) Same IP Address - seamless recovery

GE/GEC

VLAN Trunks

Dual Path with Tracking L2 Path Access L2 UplinkFast 10/100 BaseT GE or GEC

based on VLAN building block

aggregates traffic - high BW

L2

L3

L2

L3

STP root VLANs EVEN HSRP primary subnets EVEN STP root VLANs ODD HSRP primary subnets ODD

(46)

46 505

4) Small

-

Scale Server Farm

Building Block

Dual-NIC Server

Example Fault Tolerant Mode (FTM) Same IP Address - seamless recovery

Dual Path with Tracking L2

Path

Simplified building block with

no STP loops

Use if port density permits

Use if no oversubscription

(non-blocking) is a

requirement

L2

L3

L2

L3

HSRP primary subnets EVEN HSRP primary subnets ODD

(47)

47 505

Redundant Backbone Models

all good

-

increasing scale

1)

Collapsed L3 Backbone

2)

Full Mesh

3)

Partial Mesh

4)

Dual-Path L2 Switched

(48)

48 505

Core L3

Access L2

1) Collapsed L3 Backbone

large building or small campus

Clients

Collapsed

Backbone

GE/GEC

Scale depends on

physical plant and

policy more than

performance

Server Farm

(49)

49 505

Client

Blocks

2) Full Mesh Backbone

small campus - n squared limitation

Server

Block

Distribution L3 Access L2 Note importance of passive wiring closet interfaces in meshed designs! 2 blocks - 6 peerings 3 blocks - 15 peerings 4 blocks - 28 peerings 5 blocks - 45 peerings E or FE Port GE or GEC

(50)

50 505

Distribution/Core L3 Access L2

Client

Blocks

3) Partial Mesh Backbone

medium campus

-

traffic flow to server farm

Server

Block

E or FE Port GE or GEC Predominant traffic pattern

(51)

51 505

4) Dual

-

Path L2 Switched Backbone

no STP loops or VLAN trunks in core

South

Client

Blocks

Dual L2 Backbone

Distribution L3 Core L2 Access L2 “red” core subnet=VLAN=ELAN “blue” core subnet=VLAN=ELAN West North E or FE Port GE or GEC