Direct liquid cooling by bull. Laurent Cargemel Server Design & Development June 27th, 2012

(1)

Laurent Cargemel – Server Design & Development June 27th, 2012

(2)

Green datacenter characteristics

-

PUE small as possible → 1

-

Energy reuse : warmer liquid may open new usage

-

_{65°C outlet temperatures open « central heating » market}

-

Reduce liquid cooling footprint within the data-center

-

_{Mobull, new racks…}

-

Today Bull focuses on smaller PUE by using liquid

cooling

-

_{Current DLC offer}

-

₂

nd

generation improvements

(3)

-

Water cooled blade don’t need ice-cold water

-

Cooling with hot water improves significantly the PUE

Water cooling & PUE

Cool doors 40 kW/rack Room 20°C Water 7-12°C PUE 1.5-1.6 27°C 14-19°C 1.4-1.5

Direct Liquid Cooling 60 kW/rack Room up to 27°C Water 30-40°C PUE 1.1-1.2 A/C Air-Cooled 10(-20) kW/rack Room 20°C A/C water 7-12°C PUE ≥ 1.7

(4)

42U Bull rack (19’’)

-

Cooperative Power Chassis (CPC)

- _{Partitioning of water cooling and AC voltage for safety reasons} - _{54V DC to the chassis}

-

Chassis

- _{18 bi-sockets blades}

- _{Switch IB & Ethernet embedded}

- _7U

- _{100% water cooled}⇒_{fan less} - _Noiseless…

-

Hydraulic Chassis (HYC)

- _{Provide water cooling for the 5 chassis}

- _{Redundant 2N at rack level (Cheaper than at blade level…)}

Bullx direct liquid cooling technology

2 Bi-sockets nodes per blade

100% liquid cooled

(5)

Bullx dlc – Main features

-

Bullx dlc technology allows to implement more powerful blades

-

_{Double EP node blades, Single EP node blade plus accelerators…}

•

_{2kW blade with 1 EP node plus custom accelerators}

•

_{It supports largest MIC adaptors}

-

Maintainability remains as it was in an air flow design

-

_{Direct access to CPU sockets}

-

_{Direct access to Dimms}

-

_{Direct access to disks}

-

Maximum rack water inlet temperature:

Type of blade

CPU power

Node power

_{Water T°}

_(customer

loop)

2 Bi-sockets nodes blade

EP 130W

820W

35°C

EP 115W

790W

40°C

1 EP node with GPU

MIC 300W

1400W

40°C

(6)

Bullx dlc – Very accurate Blade cooling control (1)

-

Blade Water inlet temperature is regulated by setting appropriate inlet valve

openness

-

Water Redundancy is provided by putting 2 Hydraulic Controller per rack. One

HYC is active at a time

-

Regulation worst case is when one active HYC fails:

- _{Tcase CPU variation is much smaller than water inlet temperature variation : 2-3°C} - _{Figure shows real test results}

Influence d'une élévation de T °eau sur la T ° des CPU

-4 -2 0 2 4 6 8 10 50 100 150 200 250 300 T e m p s (s ) D eg ré ( °C ) Tcase CPU

(7)

-

Other source of variation:

-

_{Start at warm or cold water condition,}

blade power variations (+/- 1°C

around reference temperature)

-

Water flow is regulated depending

on the rack configuration

• _{Resulting measured water flow}

variation : (1,4 à 1,7 l/min)

• _{Related Tcase CPU variation (1,4°C)}

Bullx dlc – Blade cooling control accuracy (2)

T c a s e C P U e n fo n c tio n d e s v a ria tio n m a x im u m d e d é b it d a n s le s la m e s A r m o i r e 4 0 k W - 4 5 P l a q u e s fr o i d e s W i n x e n - T e a u c l i e n t 3 5 ° C 0 0 ,2 0 ,4 0 ,6 0 ,8 1 1 ,2 1 ,4 1 ,6 1 , 4 1 , 4 5 1 , 5 1 ,5 5 1 ,6 1 , 6 5 1 , 7 D é b it (l/m in ) Ec ar t / n om in al (° C) E c art Tc as e C P U 0 2000 4000 6000 8000 10000 12000 20 22 24 26 28 30 32 34 20 22 24 26 28 30 32 34

W ater tem perature control Scrutation 10s Kp=20 Ki=0,5 Regul 30°C

T° inlet blade Rac k c onsumption Time (s) Te m pe ra tu re s (° C ) - P ow er (k W ) 0 10 20 30 40 50 60 70 80 0 50 100 150 200 250 300 1 2 3 4 5

Blade water flow control (differential pressure)

From 5 to 1 chassis, and from 1 to 5 chassis

Number of chassis Differential pressure Time (min) D iff é re nt ia l p re ss ur e ( kP a ) N um b e r o f c ha ss is

(8)

Bullx dlc technology 2

nd

generation (1)

-

_{Today, a good solution is based on free cooling (warm water system)}

-

_{Tomorrow, a warmer outlet temperature may help for energy reuse}

-

_{Here is the split of the temperature difference between CPU case and}

water

•

_{2 main candidates for an optimization are the heat spreader geometry and the}

heat exchange at cold plate level

Losses by exchange type

Margin 10%

Conduction 45% Series water path

13%

TIM 13%

Exchange w. Fluid 19%

(9)

Bullx dlc technology 2

nd

generation (2)

-

_{Example of heat exchange optimization at}

coldplate level

•

_{More turbulence involves the heat exchange}

between cold plate en fluid

•

_{Several simulations done with different}

shapes of fins

Fluid speed spectrum (m/s)

Cold plate bullx Exchange area

Fluid path

(10)

0,6 0,8 1,0 1,2 1,4 1,6 1,8 2,0 2,2 2,4 2,6 0,0 2,0 4,0 6,0 8,0 10,0 12,0 14,0

Critère 2: Performances thermiques "Pouvoir caloprteur" à 45°C

ΔT (i n-ou t) [° C] ΔTi-o (eau)_45°C ΔTi-o (T-10)_45°C ΔTi-o (T-10/ZnO)_45°C ΔTi-o (Eau/CuO)_45°C

Bullx dlc – research activities (1)

-

Experiment new fluid : Nano-fluid

- _{Copper nano-particles fluid has been tested : 1°C}

improvement might be expected

-

1-2 kW Blade immersion while saving easy access

to the different removable parts

-

1-2 kW blade without liquid connectors

Illustration 2: Nanofluides vus au microscope électronique: (a) éthylène glycol + cuivre à 0,5 %; (b) eau + alumine ; (c)

(11)

Bullx dlc – research activities (2)

-

Water cooling with no liquid in the board

-

Advantage : No hydraulic disconnection during maintenance operations (security & cost

improvement)

-

Technical stakes : Electronic components are far from the cooling liquid

- _{Thermal : The usage of heat pipes is mandatory to} drive heat toward the thermal exchange surface of the blade

- _{Mechanical : Need perfect alignment between blade} and cold plate in order to use very thin high

performance TIM

- _{Mechanical Test vehicle has been done : first} results are promising

Design of a 400mm high performance heat pipe

Pla qu e f ro id e TIM CPU CPU CPU CPU Coque du châssis Coque de la lame Poignée amovible Zone d'appui

Dissipateur à caloduc caloduc

Zone de contact Effort Re fro id iss em en t Pla qu e f ro id e TIM CPU CPU CPU CPU Coque du châssis Coque de la lame Poignée amovible Zone d'appui

Dissipateur à caloduc caloduc

Zone de contact Effort Re fro id iss em en t

(12)

Make datacenter easier

-

Current Mobull

-

Trends: Improve rack (AC cabling, water pipes included…), 2

nd

AC main connection + Fire allarm and

protection + Access control

Hydraulic pipes

Thermal insulation

Floor Racks layout uses warm and

cold corridors scheme

Shock absorber is needed to insure full container transportation:

(13)