Sun's Project
Blackbox:
A Modular Approach to
High Density
Datacenters
Jud Cooley Sr. Director, Engineering Sun Modular DatacenterOTuI1.pdf
Who I am
•
Director of Engineering in Sun's Systems Group
•
31 years of computer development experience
> NCR, Celerity Computing, Floating Point Systems, Cray Research BSD, Sun
> Mostly large scale systems, Enterprise class
•
Jobs at Sun
> Product development
> Physical Science Lab (Optics, Reliability Physics, Prognostics) > Large Central Engineering organization, small focused teams
•
Have been fortunate to work on several paradigm
What I will cover
•
A few Data Center trends
•
Sun's Project Blackbox
•
Applications and Challenges
•
Industry Acceptance
•
Implications
•
Future
First, this is the product
•
20' container, air shippable
•
S20 (8 standard racks), D20 (4 std, 3 deep racks)
•
400 kW power delivery, 200 kW heat removal
•
208 VAC version, 415 VAC version
•
25 RU dedicated
•
Insulation, dehumidifier, fire suppression system
•
Active environmental monitoring
•
World-wide safety, EMI, fire certifications
> UL/EN 60950-1, CSA, CE Mark, GOST, EN 55022/55024, FCC Class A, ICES-003 Class A, VCCI, CNS-13438, AS/NZS 3548, C-Tick, GOST-R, BSMI, MIC, ...
Some Data Center trends
•
This is not a comprehensive list
•
Which is the bigger problem: space or power?
> It is all measured by cost
•
Free cooling?
•
High density is good, right?
•
What is the right scale to operate at? (Virtualization)
•
Facilties Mgrs and DC Mgrs now talk to each other
•
Commodity vs. differentiation: Vendor lock-in
•
Corporate Social Responsibility
•
Every aspect of operation is being examined
Switch Communications Super NAP
• 407,000 square feet of space
• 250 MVA Switch owned substation
• 146 MVA of generator capacity
• 84 MVA of UPS supply
• 30,000 tons of system plus system cooling
• 4,500,000 CFM
• 30 cooling towers
• 100% heat containment using thermal-scif™
• Designed for 1500 watts per sq. ft. density
• 7000+ cabinets
• Armed 24/7/365 military trained
• Custom servers
• The rack is the computer
Microsoft
• Gen 1 – traditional bricks-n-mortar
• Gen 3 – water cooled containers
• Gen 2 – optimizations
• Gen 4 – air cooled containers
Gen 4
Sun Datacenters based on modularity
• Scalable, repeatable, modular architecture
> Modular right-sized power and cooling
> Simplified, flexible cabling and plumbing
• Real-time energy monitoring
• Scale cost with use
• Easy to build and expand
Entering a New Age of Engineered
Datacenters
Entering a New Age of Engineered
Datacenters
Eco (Ecology and Economy)
•
Mandates: RoHS, WEEE, REACH, EnergyStar, LEED, etc.
•
Total power use and efficiency (PUE)
•
Carbon footprint
•
Material and design re-use
Some Data Center trends
•
This is not a comprehensive list
•
Which is the bigger problem: space or power?
> It is all measured by cost
•
Free cooling?
•
High density is good, right?
•
What is the right scale to operate at?
(Virtualization)
•
Facilties Mgrs and DC Mgrs now talk to each other
•
Commodity vs. differentiation: Vendor lock-in
•
Corporate Social Responsibility
•
Every aspect of operation is being examined
Sun Modular Datacenter
(aka Project Blackbox)
• 2005 CTO question: “What is the largest computer I can deliver?”
• Optimize for: General purpose, Enterprise capability
• Target new market segments, new customers, find sweet spot
• Use cases: rapid deployment, harsh environment, mobility, DR, temporary use, distributed datacenters, incremental high density space, rotational use
• Configure once, repeat often
• Lights out, remote management
• It's a computer, It's a building:
> UL/EMI/RoHS etc. certifications
> Building codes, fire safety, weather protection, ingress/egress,
Sun Modular Datacenter Features
•
What's included with Project Blackbox platform
> Structurally upgraded container
> Internal power infrastructure provides redundant 12.5kW power per rack. 25KW option for 7 payload racks
> Internal cooling system with heat exchangers, fans, & plumbing
> Eight 19” racks with cable management system & rack sliding tool
> Control infrastructure including internal sensor monitoring, lighting, air filters, & dehumidifier
•
What's not included with Project Blackbox Platform
> Payload (servers, disk, storage, tape, cables, network routers/switches, etc)
> Chiller & external plumbing and facility infrastructure
> Power generator, transformer, UPS, external connections and infrastructure
Sun MDC Cooling Efficiency
> Unique, closed-loop water cooling system
– Water can carry as much as 3,500 times more heat than air in
the same volume
– Self-contained, closed-loop air circulation system results in far
Sun MDC Site Requirements
• Input Power: 200kW
> Low & high voltage support
> 190 - 220 VAC, 3 Phase, 50 Hz/60 Hz > 380 - 415 VAC, 3 Phase, 50 Hz/60 Hz
• Cooling: chilled water
> 18°C to 22°C (64°F to 72°F) nominal input temperature
• Networking: standard options
> Ethernet, Fibre, or bulk pass-through
• Space: flat, level area
> Minimum: 9.14m x 5.49m (30ft x 18ft)
• Can be indoor or outdoor
> Typical options:
> Secure warehouse space;
Sun MDC Physical & Structural Detail
•
Starts with ISO 1 TEU shipping container
> 6.06m x 2.44m x 2.59m (20ft x 8ft x 8.5ft) (L x W x H)
> Outfitted with internal structural beams, insulation and paint
> Integrated cable trays and interior lighting; exterior connections for power, water, & network
> External lift points unchanged
•
Less than 34,000 lbs. fully configured
•
8 racks in two rows of 4
> Service access in central aisle
•
Front and rear doors
Sun MDC External Connections
Chilled water Chilled water connection connection Water Water supply supply Water Water return return Electric Electric Feeder Feeder Entranc Entranc e e External External Power Power Disconnect Disconnect Switch Switch Network Network Connection Connection Panel Panel•
Redundant water, network & power connections located on
Sun MDC Network Connectivity
•
High bandwidth needs are met through bulk
pass-through ports.
•
Infiniband or other high
bandwidth port
•
9x2 configured ports per
side
•
36 total
•
Or, remove “seal”
material and put in as
many as fit
Sun MDC Power
•
Capacity = 200kW
> 190 to 220 VAC 3-phase
50/60Hz, 600A service (Japan and North America)
> 380 to 415 VAC 3-phase 50/60Hz, 300A service (International)
> Dual grid capable with
redundant power distribution panels
•
Power power per rack
> 12.5kW or 25kW for 7 racks
Sun MDC Rack Detail
• Eight 40RU racks, 48.26cm x 78.1cm (19in x 30 in)
• Seven payload racks (servers, storage, etc.)
• One network & control rack with 25RU for network switch, etc.
> Also includes alarm control box, EPO control box, and de-humidifier
• Each rack includes an overhead articulated cable-management arm
Racks lock to floor for transportation
Racks lock to floor for transportation
Sun MDC Rack and Payload Access
• Racks slide into central (service) aisle for serviccing and installing payload w/ overhead cable management system
Sun MDC Deep Rack Version
•
Model D20
Sun MDC Mobility
Designed to move
ISO 20’ steel shipping container, rated >2X max expected weight load
Flexible coolant supply lines to Heat Exchangers
Tested for durability
> Shake tested to 6.7 on Richter scale
> NEMA 3R standard for weather resistance > IP 44 certified for particulates & moisture
Suitable for
indoor & outdoor deployment
-29 C to 54 C (-20 F to 130 F)
Sun MDC Installation Example - UCSD
• Multi-department
collaborative use via CALIT2 and Project Greenlight
• Fiber backbone across campus and to Super Computer Center
• World-class Research
environment for Green IT and energy efficient computing
Sun MDC Installation Example – Internet
Archive
•
Internet
backbone
•
Inter-building
courtyard
•
Room for 3
units
Sun MDC Installation Example - Hansen
Sun MDC Installation Example -
MTS/Russia
•
Distributed Datacenters
Sun MDC Installation Example - SLAC
•
Raid deployment, highly dense
Challenges
•
Datacenter and Facilities Managers must talk and agree
•
Breaking a paradigm has political risk
•
Could cost as much as a building if not done right (but faster!)
•
“Authority Having Jurisdiction” (AHJ)
•
Access in bad weather
•
Defferred Maintenance model
Industry Acceptance of Containers
•
IBM has said they will
•
Green Grid Container Task Force has been established to
create some guidance on Pros and Cons
Implications
•
Modularity comes in many sizes
> So does Interconnectedness
•
Rack-Rack = 2 m
•
Rack Cluster to Central Switch = 10 m
•
DC room = 30 m
•
Campus, City, Nation, ...
Other implications
•
EMI “tight”
•
Physically breaks clusters into 300 RU increments spaced
about 5-10 m apart
•
<200 kW per “node”
Future of Sun MDC and Containers
• Sun is committed to the General Purpose container business
• Microsoft building datacenters with specialized containers/servers >
http://loosebolts.wordpress.com/2008/12/02/our-vision-for-generation-4-modular-data-centers-one-way-of-getting-it-just-right/
• Both models have value
• Possible Future direction for Sun: > 40' containers
> Very High Density > Very High Power > Extreme hardening
