Designing the Data Center

(1)

May 4 - 5, 2009

Designing the Data Center

Asef Baddar, RCDD

(2)

Talking points

– What are Data Centers?

– Why Data Centers?

– The components of the Data Center

– Approaching the Data Center design

– Breakdown of the components

– Nuggets of information from TIA-942

(3)

What is a Data Center?

– A place to locate computer, storage and networking

devices safely and securely

– A place that provides un-interruptible power needed to

power a Company’s mission critical devices

– A place that provides a temperature controlled

environment for equipment to operate properly

– A place that provides connectivity to other devices inside

and outside the data center

(4)

Components of the Data Center

– Physical space

– Raised flooring

– Electrical

– Standby Power

– Cooling

– Fire Suppression

– Physical security

– Physical access

– Cabling

(5)

(6)

Why are Data Centers Important?

Critical Applications ¾ ERP ¾ CRM ¾ SCM ¾ Back office ¾ eCommerce Critical Computing Systems ¾ Servers ¾ Storage ¾ Networking Passive connectivity Pass-through Cross-connect

Data Center

Wiring Closet

CEO’s, CFO’s, CIO’s and other senior mgmt don’t spend much time

(7)

Why are Data Centers Important?

50-60% of all IT dollars are spent in the data center!

Willie Sutton

“Because that’s where the money

(8)

(9)

The Approach to designing a Data Center

– KISS – Keep it simple sir

– Flexibility – equipment changes – Scalability – 2K sq.ft. or 2M sq.ft.

– Modularity – logical and master plan for equipment – Standardize – consistency and uniformity

– The Human Factor – promote good habits – Don’t lose your sanity! “Oops” will happen.

(10)

What is tiering?

• Tier 1 – Basic

• Tier 2 – Redundant Components

• Tier 3 – Concurrently Maintainable

• Tier 4 – Fault Tolerant

(11)

(12)

What is “ N “ ?

• N = Maximum capacity needs

• N+1 = Max. plus 1 extra device(s)

• N+2 = Max. plus 2 extra device(s)

• N=3 = Max. plus 3 extra device(s)

• N*2 (2N) = doubles need

• N*3 (3N) = triples need

• Balance Need versus cost and complexity of

redundant systems

(13)

Tiering for availability

3 days, 15 hours and 40 minutes

99 Two nines

8 hours and 46 minutes 99.9

Three nines

52 minutes and 36 seconds 99.99

Four nines

5 minutes and 15 seconds 99.999

Five nines

32 seconds 99.9999

Six nines

Downtime per year Percent

Level of availability

(14)

Flexibility

• Data Centers are not static – use products that can be easily changed or moved

•Panels that accept copper or fiber

•Outlets that can be changed with a cord •Cabinets that can accept various equipment

(15)

1

5

3

4

2

6

7

8

10

9

11

12

vertical

Scalability

(16)

(17)

(18)

(19)

(20)

Understanding the

components

(21)

Physical space

Data Center selection should consider:

– Seismic areas – use earthquake resistant technologies

– Ice storms – ensure backup generators can run for extended periods

– Hurricanes – shelter against high winds – Tornadoes – no windows

– Flooding – build on elevated ground – Landslides – have retention walls

(22)

Physical space continued…

– Polluted air – limit outside air or use high filtration air systems – EMI – build away from sources of EMI

– Vibration – stay away from railroads, airports, major thoroughfares

– Political climate – labor issues and materials delay – Flight paths – distribute servers

(23)

Raised flooring

• Floor height – 24” for plenum is average

– Ensure there is enough ceiling height for heat exhaust

• Weight bearing capacity – should be rated with same strength as sub-floor

• Sub-floor – shall be sealed to prevent concrete dust • ESD – floors shall have static control

• No perforated tiles in main pathways – heavy equipment being transported over them can break them

• Ramps – should have a 1 in 12 incline, be 6’ in width and have landings on both ends or 6’ square for maneuvering equipment

(24)

Electrical

Electrical specification should consider:

• Separate power sources – building system power feed shall be isolated from server equipment power feed

• Bypass – have maintenance bypass options for normal maintenance without taking down equipment

• Pre-wire cabinet locations – adds more upfront cost but less disruptive

• PDU loads – do not load circuits more than 50 percent • Use whips – flexibility for moving power

(25)

Electrical continued…

• Receptacles – limit to 1 per circuit

• Heavy guage wire – 10 AWG is better than 12 AWG

• Run six 10 AWG wires – 3 phase conductors, 1 neutral, 1

equipment ground and 1 isolated ground

• Labeling – label the circuit and location in circuit panel

• Convenience outlets – do not tie into equipment power

(26)

Standby power

Standby power should be specified with the following considerations:

• Sizing of UPS

– Use peak load and not operating load

– Uptime should be a minimum of 20 minutes at full load with generator backup or 2 hours without a generator backup • Generators – should be designed into your Data Center if

uptime will be critical for extended outages in the area • Battery room – ensure adequate ventilation

• Monitoring lights – utilize color coded domes and wire the lights into the various backup systems for instant visual clues such as red for UPS and blue for generator

(27)

Heat is destructive to electronics

“For every 18 degrees above ambient temp. (72F-75F) reduces equipment life by ½” Intel 1987

(28)

Cooling

Specifying cooling should consider: • Fans – balance air flow with noise

• Ton of cooling – melting of 2,000 pounds of ice in 24 hours – 1 ton = 12, 000 BTUs

– 1 watt = 3.41 BTUs

• Perforated tile – can displace approximately 4KW of heat • Ideal temperature – 68 – 72 degrees is ideal

(29)

Cooling continued…

• Containing pressure – seal all tile cutouts, walls and doors • Ideal humidity – 50 percent plus or minus 10%

• Air handlers – have a buffer space around air handlers

• Cabinets – use solid side doors and install blanking panels on cold side intake to increase pressure

• Air change – design for 50 air changes per hour

(30)

(31)

(32)

Hot and cold aisles

HOT

1 tile can output 4KW BTU’s

Front

(33)

Cooling needs chart

19-29 568.3 6,820,000 10,000 682 200 17-25 497.3 5,967,500 10,000 596.75 175 14-22 426.3 5,115,000 10,000 511.5 150 12-18 355.2 4,262,500 10,000 426.25 125 10-14 284.2 3,410,000 10,000 341 100 7-10 213.1 2,557,500 10,000 255.75 75 5-7 142.1 1,705,000 10,000 170.5 50 # of air handlers Tons of cooling Total BTUs DC size (sq.ft.) BTUs/sq.ft. Watts/sq.ft

(34)

Fire Suppression

Fire suppression specification needs to address: – Gases

• Inergen (IG-451) – removes oxygen from the air and ozone friendly • Argonite (IG-55) – reduces oxygen in the air and ozone friendly • FM-200 or HFC-227 –absorbs heat energy and ozone friendly • FE13 or HFC-23 – absorbs heat energy and not ozone friendly – Sprinklers – has a threshold rating of 155 degrees on the bulbs and

you should utilize a dry system in the server room

– Smoke and heat detection – allows a proactive approach

(35)

Physical security

Physical security specification needs to address:

– Door controls – use badge access or combination door locks – Cages – additional physical security of sensitive equipment

– Locking cabinets – key or combination but be aware of cooling implications

– CCTV – place at entrances into DC and at sensitive equipment – Access Policies – have a policy on how access is granted

(36)

Physical space

Physical space specification needs to address: – Aisle size – 5’ for main thoroughfare

– Door size –shall be a minimum of 3 ft wide and 7 ft high, without doorsill, hinged to open outward (code permitting) or slide side-to-side, or be removable. Doors shall be fitted with a lock and have either no center post or a removable center post to facilitate

access for large equipment

– The minimum ceiling height shall be 2.6 m (8.5 ft) from the finished floor to any obstruction such as sprinklers, lighting

fixtures, or cameras. Cooling requirements or racks/cabinets taller than 7 ft may dictate higher ceiling heights. A minimum of 18 in clearance shall be maintained from water sprinkler heads.

(37)

(38)

TIA 942 – Telecommunications

Infrastructure Standard for Data Centers

What kind of information can I find in the document? – Scope

– Definition

– Design Overview

– Cabling system overview

– Cabling system infrastructure

– Telecommunications spaces and Related Topologies – DC cabling systems

– DC cabling pathways – DC redundancy

(39)

Cabling Topologies

(40)

STATUS PHONE WS-X6548-GE-TX12 11121 314 23242526 35363738 4748 4 8 P O R T 1211 910 78 56 34

12 13141516171819202122232410/100/1000 BASE-TETHERNET252627282930313233343536SWITCHING MODULE373839404142434445464847

STATUS PHONE WS-X6548-GE-TX 4748 3738 3536 2526 2324 1 314 1112 12 4 8 P O R T 1211 910 78 56 34 12 131415161718192021222324 252627282930313233343536 4847 4546 4344 4142 3940 3738

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX12 11121 314 23242526 35363738 4748 4 8 P O R T 1211 910 78 56 34 12 24 23 2122 1920 1718 1516 1314 252627282930313233343536 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX 4748 3738 3536 2526 2324 1 314 1112 12 4 8 P O R T 1211 910 78 56 34 12 131415161718192021222324 252627282930313233343536 4847 4546 4344 4142 3940 3738

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX12 11121 314 23242526 35363738 4748 4 8 P O R T 1211 910 78 56 34 12 131415161718192021222324 36 35 3334 3132 2930 2728 2526 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX12 11121 314 23242526 35363738 4748 4 8 P O R T 1211 9 10 78 56 34 12 24 23 2122 1920 1718 1516 1314 252627282930313233343536 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX 4748 3738 3536 2526 2324 1 314 1112 12 4 8 P O R T 1211 910 78 56 34 12 131415161718192021222324 36 35 3334 3132 2930 2728 2526 4847 4546 4344 4142 3940 3738

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX12 11121 314 23242526 35363738 4748 4 8 P O R T 1211 910 78 56 34 1 2 131415161718192021222324 252627282930313233343536 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX12 11121 314 23242526 35363738 4748 4 8 P O R T 1211 9 10 78 56 34 12 24 23 2122 1920 1718 1516 1314 252627282930313233343536 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

(41)

STATUS PHONE WS-X6548-GE-TX 4 74 8 3 738 3536 2 526 232 4 1 314 1112 12 4 8 P O R T 1211 910 78 56 34

STATUS PHONE WS-X6548-GE-TX12 11121 314 232 42 526 35363 738 4 74 8 4 8 P O R T 1211 910 78 56 34 12 24 23 2122 19 20 1718 1516 1314 3635 3334 3132 2930 2728 2526 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX12 11121 314 232 42 526 35363 738 4 74 8 4 8 P O R T 1211 910 78 56 34 12 131415161718192021222324 3635 3334 3132 2930 2728 2526 4847 4546 4344 4142 3940 3738

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX12 11121 314 232 42 526 35363 738 4 74 8 4 8 P O R T 1211 9 10 78 56 34 12 24 23 2122 1920 1718 1516 1314 252627282930313233343635 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX12 11121 314 232 42 526 35363 738 4 74 8 4 8 P O R T 1211 910 78 56 34 12 24 23 2122 1920 1718 1516 1314 3635 3334 3132 2930 2728 2526 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX12 11121 314 232 42 526 35363 738 4 74 8 4 8 P O R T 1211 910 78 56 34 1 2 131415161718192021222324 252627282930313233343635 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX12 11121 314 232 42 526 35363 738 4 74 8 4 8 P O R T 1211 910 78 56 34 12 131415161718192021222324 3635 3334 3132 2930 2728 2526 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X6548-GE-TX 4 74 8 3 738 3536 2 526 232 4 1 314 1112 12 4 8 P O R T 1211 910 78 56 34

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

Patch Cord into Server

(42)

Switch

Zone Distribution Area

FA N STAT US 1 2 3 4 5 6 7 8 9 Power Supply 1 Power Supply 2 Catalyst 6500 SERI ES STATUS PHONE WS-X654 8-GE-TX12 11121314 23242526 35363738 4748 4 8 P O R T 12 11 910 78 56 34 12 24 23 2122 1920 1718 1516 1314 36 35 3334 3132 2930 2728 2526 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X654 8-GE-TX12 11121314 23242526 35363738 4748 4 8 P O R T 12 11 910 78 56 34

STATUS PHONE WS-X654 8-GE-TX12 11121314 23242526 35363738 4748 4 8 P O R T 12 11 910 78 56 34 12 24 23 2122 1920 1718 1516 1314 252627282930313233343536 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE STATUS PHONE WS-X654 8-GE-TX12 11121314 23242526 35363738 4748 4 8 P O R T 12 11 910 78 56 34 12 131415161718192021222324 252627282930313233343536 4847 4546 4344 4142 3940 3738

STATUS PHONE WS-X654 8-GE-TX12 11121314 23242526 35363738 4748 4 8 P O R T 12 11 910 78 56 34

STATUS PHONE WS-X654 8-GE-TX12 11121314 23242526 35363738 4748 4 8 P O R T 12 11 910 78 56 34 12 24 23 2122 1920 1718 1516 1314 36 35 3334 3132 2930 2728 2526 373839404142434445464847

10/100/1000 BASE-TETHERNET SWITCHING MODULE

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

SD

THE SIEMON COMPANY

Patch Cord into Switch

Patch Cord into Server

Servers 48-Port Patch Panels

Horizontal Cabling

(43)

(44)

(45)

942 design guidelines

– In addition to satisfying today’s telecommunication requirements, the horizontal cabling should be planned to reduce ongoing

maintenance and relocation. It should also accommodate future equipment and service changes. Consideration should be given to accommodating a diversity of user applications in order to reduce or eliminate the probability of requiring changes to the horizontal cabling as equipment needs evolve.

– Change your equipment and not your cabling!!! – Minimum of Category 6 is recommended.

(46)

942 design guidelines continued

– For copper cabling, in order to reduce the effect of multiple

connections in close proximity on NEXT loss and return loss, the horizontal distribution area termination should be located at least 15 m (50 ft) from the main distribution area termination. *NOT necessary!

– Overhead cable trays should be suspended from the ceiling

– 75 ohm cable (type 734 , 735) is allowed for application specific applications such as DS-3

– Use a minimum of 4” wide vertical management on both sides of racks and a minimum of 10” wide vertical between racks

(47)

942 design guideline continued

– Power strips shall be labeled with the PDU/panel identifier and circuit breaker number.

– Racks that are supported by the access floor shall be bolted to the cement slab or a metal channel secured to the slab by

threaded rods that penetrate through the floor tiles.

– Sharp edges on the top of the threaded rods shall be covered using domed nuts or other method. Exposed threads under the access floor should be covered using split tubing or other

method.

– Power strips with indicators but no on/off switch or breaker reset button should be used to minimize accidental shut-off.

(48)

(49)

(50)

Utilize pre-terminated Category 6A systems

– Great for quick network deployment

– Excellent Channel to Channel characteristics – Ideal for data center Server Row designs

– Lifetime System Warranty – No field testing required at all

• 100% performance tested at factory to TIA CAT6A – UTP or STP cables secured in braided sleeve

(51)

Utilize pre-terminated MTP design

– Consider OM3 fiber or better

– Great for quick network deployment

– Cabling harness can be changed for new equipment – Excellent Channel to Channel characteristics

– Ideal for data center 4 connector low loss channels – Lifetime System Warranty

– No field testing required at all

• 100% performance tested at Factory

MTP - SC

(52)

Switch or Server Cabinet Options

– 72 x CAT6A UTP ports in 2U

– Minimizes server or switch cabinet rack space – 3 panel options

• Standard Flat • Angled

(53)

Server Cabinet Panel Option

• VertiGO Patch Panel – 12 x CAT6A ports

– Consumes zero rack space

– Provides easy access to servers – Improves airflow

(54)

– Reduces cabling/congestion inside of cabinet – Allows for copper and/or fiber connectivity – Supports 10G connectivity