Fabrics SDN and Big Data. What this means to your white space

(1)

Fabrics SDN and Big Data. What this means to your white

space

Carrie Higbie RCDD/NTS, CDCP, CDCS

Global Director Data Center Solutions and Services

[email protected]

TechTarget – Ask the Experts and columnist SearchNetworking, SearchVoIP, SearchDataCenters

BOD and Fmr. President BladeSystems Alliance Member Ethernet Alliance, IEEE

Columnist Performance Networking, SearchCIO, SearchMobile, ZeroDowntime, ComputerWorld,

IT World, Network World, Financial Review

Twitter: @carriehigbie

(2)

(3)

• TIA

• ISO/IEC

• Ethernet Alliance

• IEEE

• ANZ standards committees

• BICSI

• US Green Building Council

• Various World Green Initiatives

Technology Leadership

• Various World Green Initiatives

• Uptime Institute

• ASHRAE

• AFCOM

• 7x24Exchange

(4)

Cloud Security Issues

• Data dispersal and international

privacy laws

–

EU Data Protection Directive and U.S.

Safe Harbor program

–

Exposure of data to foreign government

and data subpoenas

–

Data retention issues

• Need for isolation management

• Multi-tenancy

• Logging challenges

• Data ownership issues

(5)

Challenges for Cloud

• Infrastructures and data centers already built • Loss of control

• Shift of employment

• Potential loss of employment • Security – can be a big unknown • SLA’s

• Provider expertise in the cloud space • Portability

• Knowledge transfer • Compliance

• Location

• Lack of standards

• How to bill for services? • Public or private? Hybrid?

(6)

Driving Private Clouds

• Portability

• Confidence as a Service

• What works in the cloud and

what doesn’t?

• Vendor dependency

• IT threat

• Power users circumvent IT

• HR involvement?

• IT may not be involved in those

decisions

• Tangible and intangible ROI

• IT threat

• Security

• Bankruptcy

• Geographic Diversity

calculations

• Standards

• Local regional sovereignty

requirements

(7)

Risk Assessment is Critical

• Type of information • Technology

– Vendor lock-in could be a threat – Open systems

• Vendors are trying to close their systems

• Location of information • Location of information

(8)

Lifecycle of a Data Center

• Inception • Design • Build • Operations • Retire/Retrofit • Retire/Retrofit

(9)

Inception

• We need a data center – Where?

– How big?

– How much power? – How much cooling? – Colo or not

– Cloud or not (public, private, hybrid) – Cloud or not (public, private, hybrid)

(10)

What can go wrong at inception?

• Dreaming too big • Ignoring new trends

• Living too cutting edge with nothing to fall back on • Relying too much on consultants previous designs • Relying on the wrong design build/consulting firms • Relying on the wrong design build/consulting firms

(11)

Example – What’s wrong here?

_{PDU’s add to}

cooling load

Walkway blocked

Open space mixes airflow

No access to cabinets – blocked

(12)

PUE

• Power Usage Effectiveness

• Defacto standard

• Total power / IT load

• Most data centers are in the 1.9-2.6 range

• The closer to 1 the better

• Requires Intelligent PDU’s or some means to measure

USED power

• DCIE (Data Center Infrastructure Effectiveness) is

1/PUE to give percentage

(13)

Green Grid

• The Green Grid is proposing the use of a new metric

• Data center compute efficiency (DCcE)

• and its underlying sub-metric, server compute efficiency (ScE).

• These metrics will enable data center operators to determine the

efficiency of their compute resources, which allows them to identify

areas of inefficiency.

• Using DCcE and ScE, data center operators can continually refine

and increase the efficiency of their compute resources in the same

way that they use power usage effectiveness (PUE) to improve data

center infrastructure.

• CUE (Carbon Usage Effectiveness)

(14)

(15)

Ethernet, Energy Efficient Ethernet, Data Center

Ethernet

• All things Ethernet!

• EEE is a game changer, especially for 10GBASE-T

– Provides a true idle state – Significantly lowers power – Significantly lowers power

• Data Center Ethernet collapses the backbone structure – layer 2

instead of layer 3

• Idea is to increase speed (bridge not route) and provide priority

• Ethernet has a LOT of overhead

(16)

Detailed Design

Original 2D customer floor plan concept

(17)

Customer Case Study

Thermal Analysis Overview

• Hot Aisle/Cold Aisle Configuration

– Based on customer-proposed type and placement of CRAC units, design create an optimal layout for thermal management, which was then run through thermal analysis software

Potential hotspots identified and corrected in the design

Theoretical analysis performed on multiple levels of equipment population to ensure ability to manage increased heat load of future equipment additions

Phase Phase 22 Phase Phase 22

ensure ability to manage increased heat load of future equipment additions

- Findings:

With airflow management practices such as:

• Vertical chimneys on SAN cabinets

Thermal analysis provided to user for communication with cooling vendor to ensure right-sized CRAC system

(18)

(19)

(20)

TIA 942, 942-2, 942-A

• This Standard (942-A) replaces ANSI/TIA-942

dated April 12, 2005 and its addenda.

• This Standard incorporates and refines the

technical content of:

– ANSI/TIA-942, Addendum 1, Data center coaxial

cabling specifications and application

311 distances

312 – ANSI/TIA-942, Addendum 2, Additional guidelines for

(21)

Expanded Topology

(22)

Quick case studies

• The savings behind a great

design

• Must take into account the entire

data center ecosystem

• Customer needs assessment

• Knowledge of all DC products

• Knowledge of standards and

(23)

Traditional 3-Tier

• 3-Tier switch architectures

were common practice

– Core and SAN

switches in the MDA – Aggregation switches

in the MDA, IDA or HDA – Access switches in the HDA – Access switches in the HDA

• Shorter distances

usually accommodated

more than two connectors

without exceeding loss

• No longer adequate for non-blocking, low-latency communication

(24)

(25)

(26)

(27)

(28)

A better design=..

BASE-T Switches in HDA

Aggregate/Core

• Every red line is 24 or 48 channels of copper (primary network) • Every blue line is 24 or 48 channels of secondary copper

(29)

Top of Rack Switching 3 Tier

(30)

With ToR intercabinet connections change to

fibre

(31)

(32)

1, 3 and 5m limitation on passive,

10m on active SFP+

• In order to fill a 48 port switch,

you would need to be able to put

48 servers in the cabinet per

network

• Most data centers don’t allow

cabinet to other cabinet spaghetti

• Impossible in most data centers

– Limited by weight, power and

cooling

(33)

Installed port cost –

data center cabling media

10GBASE-CX4 INFINIBAND QDR FCoE FC/10GbE 10GBASE-Cu 5m 10GBASE-Cu Active 7m Hardware Cable Cost Maintenance 15% of

90 day warranty from electronics manufacturer

- 0 0 0 0 0 0 0 0 0 0 10GBASE-T (30m) 10GBASE-SR 10GBASE-LR 10GBASE-LRM 10GBASE-LX4 (switch to … 10GBASE-CX4 Maintenance 15% of cost YOY 20 YEAR WARRANTY

(34)

(35)

Costs for switching

List Total

Used

ports Unused

78Nexus 2000 32 port 10G $15,000.00 $1,170,000.00 1092 1404 78Redundant Power Supplies $500.00 $39,000.00

312SFP+ Uplink Ports $1,500.00 $468,000.00

Total for 2000's $1,677,000.00

10Nexus 5000 32 port $23,010.00 $230,100.00

39SFP+ Modules $5,200.00 $202,800.00

10Redundant Power Supplies $500.00 $5,000.00

Total for 5000's $437,900.00

2Cisco 7010 Core Switches $79,000.00 $158,000.00 2Redundant Power Supplies $7,500.00 $15,000.00 4Fiber Card for Uplinks $70,000.00 $280,000.00 $453,000.00

Grand Total $2,567,900.00 Does not include software

3 Years Maintenance $1,155,555.00

(36)

Zoned Approach

List Total

34 Nexus 2000 32 port 10G $15,000.00 $510,000.00 34 Redundant Power Supplies $500.00 $17,000.00 136 SFP+ Uplink Ports $1,500.00 $204,000.00

Total for 2000's $731,000.00 Savings $ 946,000.00 5 Nexus 5000 32 port $23,010.00 $115,050.00

20 SFP+ Modules $5,200.00 $104,000.00 5 Redundant Power Supplies $500.00 $2,500.00

Total for 5000's $221,550.00 Savings $ 216,260.00 2 Cisco 7010 Core Switches $79,000.00 $158,000.00

2 Cisco 7010 Core Switches $79,000.00 $158,000.00 2 Redundant Power Supplies $7,500.00 $15,000.00 2 Fiber Card for Uplinks $70,000.00 $140,000.00

$313,000.00 Savings $ 140,000.00 Grand Total $1,265,550.00 Savings $ 1,302,260.00 3 Years Maintenance $569,497.50 $ 586,057.50

(37)

Amount that could be used

for other equipment per

cabinet

• $48,420.71

• This is ONLY 39 cabinets!

• According to our ROI sheet 48 ports per cabinet

(24 each network) for 6A F/UTP and 30m

channels the cost is $105,399.00 or roughly 2 ½

cabinets worth of savings

• Total savings to customer

AFTER CABLING

(38)

Qty Description List Total

Network

Ports Used Unused

720 Nexus 2000 32 port $ 15,000.00 $10,800,000.00 23040 10080 12960 720 Redundant Power Supplies $ 500.00 $360,000.00

5760 SFP+ Uplink Ports $ 1,500.00 $8,640,000.00

Total for 2000's $19,800,000.00

Cost per port $859.38 $8,662,500.00 $11,137,500.00 192 Nexus 5000 32 port $ 23,010.00 $4,417,920.00 6144 2880 0 768 SFP+ Uplink Modules $ 5,200.00 $3,993,600.00

192 Redundant Power Supplies $ 500.00 $96,000.00

Total for 5000's $8,507,520.00

4 Cisco 7010 Core Switches $79,000.00 $316,000.00 4 Redundant Power Supplies $7,500.00 $30,000.00

24 Fiber Card for Uplinks $70,000.00 $1,680,000.00 768 768 0

Total for 7000's $2,026,000.00 Grand Total with top of rack $30,333,520.00

Port Count Total 29952 13728 12960

Nexus 2k's in top of rack

Required 5k's for uplinks

7k's and line cards

Port Count Total 29952 13728 12960

150 Nexus 2000 using required ports $ 15,000.00 $2,250,000.00 4800 4200 600 150 Redundant Power Supplies $ 500.00 $75,000.00

1200 SFP+ Uplink Ports $ 1,500.00 $1,800,000.00

Total for 2000's $4,125,000.00

Cost per port $859.38 $3,609,375.00 $515,625.00 19 Nexus 5000 32 port $ 23,010.00 $437,190.00 608 600 8 76 SFP+ Uplink Modules $ 5,200.00 $395,200.00

19 Redundant Power Supplies $ 500.00 $9,500.00

Total for 5000's $841,890.00

2 Cisco 7010 Core Switches $79,000.00 $158,000.00 2 Redundant Power Supplies $7,500.00 $15,000.00

3 Fiber Card for Uplinks $70,000.00 $210,000.00 96 76 8 1/2 the 7000's required

Zoned Approach - required ports 8400 at 14 servers dual network attached each

5k's required if centralized $25 million savings,

unused ports drop from 12,960 to 616

(39)

Enter New Switch Fabrics

• Switch fabrics typically

only use one or two

tiers of switches

– Interconnection switches in the MDA or IDA

– Access switches in the HDA or EDA

HDA or EDA

– Aggregation switches are usually eliminated

• Provides lower latency and greater bandwidth between any two

points

– Eliminates traffic having to travel through multiple switch layers

(40)

What this looks like- every server

connects to a more centralized switch

In the fabric world- also known as leaf and spine 2 Connector channel is basis for 40GBASE-T

(41)

Cross Connects in the Data Center

• Creates a convenient patching area

– Fiber panels that mirror switch ports connect via fixed links

– Creates an “any to all” configuration

– MACs made at the panels via fiber jumpers

• Can be located in a separate cabinet

– Keeps switches untouched and secure in their own cabinet

• Allows for one-time deployment • Allows for one-time deployment

of fiber from MDA to HDA

– Simplifies adding new equipment

• Fiber can be used for multiple purposes (networking or SAN)

– Swap a server’s fiber connection from networking to SAN using a jumper at the cross connect

(42)

Another View

• One-time deployment of fiber from cross connect in MDA to cross connects at End of Rows in HDA

• Fiber jumper

changes at End of Row

cross connects cross connects

allow fiber to be used for various purposes

• Each cross connect adds connection points and loss

(43)

Higher Speeds Have More Stringent

Insertion Loss Requirements

• Ethernet speeds migrating from 1 and 10 Gb/s to 40 and 100 Gb/s

• Fibre Channel speeds (SANs) migrating from 8 Gb/s to 16 and 32 Gb/s • Distance limitations and channel loss decrease as speeds have increase • Standards specify a maximum channel loss for each application

Fiber Type 1000BASE-SX (1 Gb/s) 10GBASE-SR (10 Gb/s) 40GBASE-SR4 (40 Gb/s) 100GBASE-SR10 (100 Gb/s)

Distance Channel Distance Channel Distance Channel

(m) Loss (dB) (m) Loss (dB) (m) Loss (dB)

OM3 1000 4.5 300 2.6 100 1.9

OM4 1100 4.8 400 2.9 150 1.5

Fiber Type

8 Gb/s Fibre Channel 16 Gb/s Fibre Channel 32 Gb/s Fibre Channel Distance (m) Channel Loss (dB) Distance (m) Channel Loss (dB) Distance (m) Channel Loss (dB) OM3 150 2.0 100 1.86 70 1.87

(44)

Or to put it another way=..

BASE-T Switches in HDA

Aggregate/Core

• If the top of rack switch is removed – the fiber in the top of the cabinet becomes a passive cross connect that can be used for SAN

• The same would hold true if the aggregate end of row switch was removed. This adds additional passive connections in the channel

(45)

Sacrificing copper increases fiber and $$==

• OM4 min recommended in TIA

(46)

Looking at costs

Three Tier Leaf/Spine DAC Leaf/Spine 10GBASE-T

Low density 14 Servers/Cab

Install Cost $ 8,816,885.18 $ 11,786,235.65 $ 8,638,321.02

Average Cost/Server Cab $ 61,228.37 $ 70,156.16 $ 59,988.34

Annual Power cost Networking $ 91,328.26 $ 101,419.78 $ 44,402.69

High Density 40 Servers/Cab High Density 40 Servers/Cab

Install Cost $ 16,295,362.88 $ 23,208,375.30 $ 18,722,843.39

Average Cost/Server Cab $ 113,162.24 $ 138,145.09 $ 130,019.75

(47)

Fabrics in particular

Leaf/Spine DAC Leaf/Spine 10GBASE-T

Low density 14 Servers/Cab

Total Equip/Cabling Cost $ 11,786,235.65 $ 8,638,321.02 Average Cost/Server Cab $ 70,156.16 $ 59,988.34 Annual Power cost Networking $ 101,419.78 $ 44,402.69 Total Cabling Cost $ 1,222,357.82 $ 70,327.30 Total Cabling Cost $ 1,222,357.82 $ 70,327.30 High Density 40 Servers/Cab

Total Equip/Cabling Cost $ 26,394,022.02 $ 21,596,114.19 Average Cost/Server Cab $ 157,107.27 $ 149,973.02 Annual Power cost Networking $ 177,610.75 $ 106,717.82 Total Cabling Cost $ 5,123,942.02 $ 2,078,260.76

(48)

Questions???

• @carriehigbie