High Density Rack Configurations and their Associated Cooling Issues

High Density Rack

Configurations and their

Associated Cooling Issues

High Density Server Configurations

The Fact

High-density server configurations represent the cutting edge of server management. With standardised chassis pre-configured with network switching and power supply, users of blade servers benefit from several features:

•

• •

•

But, the last fact is oversold. This efficiency is measured in single digit percentages and is overwhelmed by the increased power consumed by the latest generation of CPUs. An analogy would be that the latest Airbus 380 will use relatively more efficient wing and engine design – perhaps measured in a couple of percentage points. But this 580-passenger plane will still use significantly more power than a 120 passenger Airbus 319 – by a factor of 4.

The Fiction

By constantly reducing server footprints, more servers can be fitted to a single rack. Customers are lured by overzealous hardware sales people on a concept that they can pack more processing power in a given space, cutting datacentre costs. This is true, furthermore, most datacentres have modular, upgradeable power systems, meaning power delivery to individual racks can be high.

The problem is not space or power to an individual rack – it is the overall impact of high-power racks on the datacentre cooling. Only six years ago, the market leading datacentre company – Exodus – broke all previous records by delivering a datacentre with the capacity to cool 25watts per ft2 (270watts per m2). Today, most datacentres are built to the physical limit of standard air conditioning systems – approximately 100watts per ft2 (1000watts per m2).

Unlike Moore’s Law – this will not continue to double every two years. Laws of Physics, including the transfer of energy, have been around for significantly longer than the transistor.

Customers who push their air-conditioning systems over certain limits are faced with one of two expensive scenarios:

•

•

The Reality

Blade server technology has benefits that ensure their survival and may even transform datacentres of companies that use large quantities of servers:

Standardised components for rapid scaling (up and down) and rapid replacement.

Management tools allowing for instant auto-configuration and hot-swapping.

Lower TCO (but only at high blade saturation of chassis).

More efficient infrastructure reducing overall electricity consumed per unit of calculation power (ie: two very efficient power supply units for a whole rack, a shared SCSI and network bus, reduced

ancillary components). _{Hot servers – these will run inefficiently and}

breakdown sooner. Also, in peak conditions, they will reach unsustainable temperatures and force shutdown. This could be caused by reboots, upgrades, peak utilization, DDOS attacks or (in load balanced systems) by being temporarily overloaded.

Capital upgrades – expensive supplementary cooling systems will have to be installed, in some cases this will be physically impossible as the heat will still need to be dissipated (ie: a corporate datacentre on the 32nd floor of Canary Wharf Tower will have a hard time dissipating increased heat build-up).

• •

•

After working on several high-density solutions, IXEurope welcomes customers with high power needs. However, there are specific design, suite build, and rack layout considerations to review as customers push the physical limits of heat transfer.

This report touches on these issues and highlights the need for customer and supplier to plan current and future needs as closely as possible. This is divided into broad sub-sections:

•

•

•

Basic Configurations

“Heat is energy and cannot be destroyed, merely transferred or state changed”.

Though not officially ‘high-power’, customers whose initial configurations are averaging 1000watts per m2 should follow some basic recommendations to maximize

performance of systems and avoid some typical pitfalls of a inefficiently planned datacentre.

Back to Back, Front to Front

The majority of equipment today has air intake grilles at the front, which allow air to be drawn in and across the internal components, and then exhausted to the rear.

If the racks are configured in the conventional manner i.e. all facing the same way, the exhaust of the first row will be drawn into the air intake of the second row and so on. To avoid a cumulative heat build up, the racks should be arranged to face alternate directions i.e. Front to front, back to back. The front-to-front row has floor tile grilles delivering conditioned air to the Cold corridor. This air is drawn in by both rows of racks and then subsequently exhausts into the Hot corridor.

Through convection and the negative air pressure caused by the air intake of the Computer Room Air Condition unit (CRAC), this air returns to the CRAC. Any equipment located in the return air path shall experience the cumulative heat load of the suite. This is why the racks or equipment are limited to 47U or to the height of the CRAC return air intake and the available return air path area. 22˚C Intake Intake CR A C CR A C Raised Floor

Discharge Supply Air Plenum Perforated tiles

45˚C 40˚C 34˚C Cold Aisle Hot Aisle Physical Footprint Cooling Footprint Easier to manage.

More software management instead of physical reconfiguring and cabling.

Better use of infrastructure.

Basic Power Configurations (Suite averaging 1000watts per m2).

High-Power Configurations (Suite averaging from 1000-2000watts per m2).

Extremely High-Power Configurations (Suites with racks 4000watts per rack).

The CRAC units are self-regulating in relation to cooling load. The amount of cooling required is determined by measuring the temperature of the return air. To ensure precise cooling response to the suite heat load, we do not install vented floor tiles to the Hot corridor, as this would dilute the return air. Under floor static pressure decreases with air velocity therefore no vents or tiles shall be installed within 2400mm of a CRAC.

The airflow paths will be more effective if the rack rows are at 90 degrees to the CRAC and in line with the hot corridors.

Efficient Spacing of Racks

In addition to employing a Hot/Cold corridor layout, the Project Manager needs to ensure that manufacturers’ recommendations have been met. An example would be for the IBM Netbay 42 Enterprise rack, which requires 914mm to the rear and 1524mm to the front at a maximum weight of 932 Kg.

HP has similar requirements, which are necessary to allow for efficient drawing of cool air into the rack and exhausting of hot air. Not meeting these standards will impede airflow and will void manufacturer’s warranties.

Currently, HP recommends dedicating 1200mm to the front and 600mm to the rear of a rack for a row of high-power racks so that there is adequate room for floor venting in front of all racks.

In an existing infrastructure with 1200mm front and back, HP recommends spacing out high power racks, especially as the powerful ventilation systems impact other racks of equipment. They also recommend an average floor spacing of 24sf/rack and a thermal analysis to position floor vents correctly. Nonetheless, overall datacentre load should not surpass 1800watts/rack so that a standard datacentre configuration is not stressed.

Efficient Use of Sub-Floor Air Plenum • • • Intake HVAC Raised Floor Discharge

Supply Air Plenum Perforated

Tiles

HVAC

Cold Aisle Hot Aisle Cold Aisle

Intake

HVAC

Discharge

Data Center Thermal Modelling BL30p 30KW Analysis > 40 33.75 27.5 21.25 <15 Base Temp. Scale Case Model CM75 Power Density 75W per Sq Ft

Recommended 9000 Rack 10000 Rack 9000 Rack 10000 Rack 10000 Rack 9000 Rack 10000 Rack 9000 Rack CRAC CRAC

Use of the sub-floor area.

Positioning and control of air vents. Avoiding unwanted airflow.

Use of Sub-Floor Area

Suite cabling containment should be designed to ensure that sub floor containment does not obstruct

airflows or free air space to ventilation grilles. High-level cable containment shall be designed and recorded on the multi layered design and build drawings. Solid cable trunking may be used at high-level however solid trunking shall not be used within the floor void if analysis deems it to cause an airflow restriction. As the airflow patterns are a major concern, then IXEurope will implement as much high level cable containment as reasonably possible with consideration given to future possible supplementary cooling. The final design shall be approved a n d signed off by the client before the suite build commences.

Positioning and Control of Air Vents

Clients and IXEurope shall also determine where and how many adjustable ventilated floor grilles are installed so as to deliver the appropriate volume of air to the server air intake and control the balance of the air conditioning system. Refer to the separate “Guidelines for the use of vented tiles”.

Vented tiles vary in design and so can vary from 400 CFM to 1000 CFM. The perforated tiles have a lower CFM capacity than the steel adjustable vent tiles. The

adjustable tiles are used to balance the suite. Avoiding Unwanted Airflow

It should be noted that vented floor tiles should never placed within 2400mm of the CRAC’s as it is possible that room air can be drawn down into the “cold” floor void plenum through the Venturi effect. High load equipment must be positioned approx 8000mm – 11000mm of the CRAC run.

A careful study of the volume of air required by each rack should be undertaken, as the velocity and pressure of the air within the floor void plenum will dictate how much air passes through the vented tile. It is for this reason that the highest load equipment should not always be placed the closest to the CRAC.

5% open vent tile results in cooler inlet temperature.

95% open vent tile results in hotter inlet temperature.

95% open tile 5% open tile

HV

A

C

Supply Air Plenum High-velocity discharge

Tile cut outs shall be kept to a minimum and any points of air leakage blocked, to maximise plenum air pressure. As we need to deliver the cold air to the front of the equipment via the vented floor tiles, the floor tile cut outs for cable access shall be sealed to maintain pressure at the vents.

Rack Configuration

“The closer a surface is placed to the negative side of a fan, the greater the air resistance, thus the less air moved. The higher the heat output of a device, the greater the amount of air is required to be moved to transport the energy (heat) away from the device.” Conventional rack configuration assumed that sufficient free air space existed within the rack and in front of the server to allow adequate airflow to the equipment in a vertical manner. If this free air space was present then glass front doors created a chimney effect with the cold air delivered from the floor void. Unfortunately today’s servers almost fill the footprint of a 900 x 600 mm rack and so have very little air space to the front or rear of the server. The close proximity of the front door increases air resistance on the internal fan and so reduces the volume of air moved. The close proximity of the rear door causes turbulence at the exhaust fan of the server and so by increasing the positive pressure, further increases the resistance on the server fan.

To ensure the least amount of air resistance then doors should not be fitted however if doors are required then both the front and rear doors should be fully ventilated. It should be noted that standard glass doors with ventilation strips either side have a free air area of only 12 % whereas a ventilated steel door typically has a free air area of 53%. This will provide sufficient airflow to the front of the server however you will still have 47% of the rear door reflecting the exhaust air. This reflected air becomes turbulent and so blanking plates need to be installed between the servers to prevent warm air short-circuiting to the front of the server.

This includes the gap between the top server and the top of the rack. This is not only to stop warm air short-circuiting but also to ensure the rack top extract fans do not create negative pressure to the front of the servers. When installing rack extractor fans we must ensure that fans have a rating at least equal to the rating of the internal server fans so as to ensure a matched volume of air moved. As the exhaust air will exit the rack both through the door and up through the top of the rack we must install deeper racks of 1000mm deep and extract fans to the rack top rear section.

As we have established, the extractor fan efficiency is determined by its proximity to a solid surface thus for high load racks the top fan box may need to be extended. It is also possible to supply racks with extract fans built into the rear door. Racks can be modified so that ductwork can be installed from the rear of the server to a cut out in the rear door thus offering no resistance to the exhaust air. This would restrict flexibility of equipment changes in the future.

In the event of a Utility company power failure, the CRAC’s shall experience a brief interruption while the generator system assumes the load. During this time the server fans shall obviously continue to run and so the rack extract fans are imperative in assisting them to remove the exhaust air.

The castors or jacking feet at the base of the rack will cause a gap of approx 30 mm and so, to prevent short circuits, this should have a plinth trim plate fitted. An example of the volumes discussed. Each IBM eserver BladeCenter requires 500 cubic feet per minute (CFM) and states operating parameters 0f 10C – 35C, 8% - 80% Relative Humidity (RH) at less than 914 m altitude. Equipment Configuration

“Any gains achieved through careful planning of the suite are lost if the same principles are not applied to the equipment”

The prime objective of the cooling system is to transfer the energy within the server to an unimpeded air path where it can be conditioned and recycled into the room. We have seen how the exhaust air exits the rack through the rear and top of the rack. Careful planning of the equipment configuration will prevent high load equipment impacting the environment of other items of equipment. This applies to both vertical and lateral planning, as high load equipment will have a vertical impact within the rack and lateral impact in the return air path to the CRAC.

Typically, server manufacturers design the internal components to operate at 10C - 35C however this should be checked and those with a lower operating limit be placed lower in the rack. Although the servers

use internal fans to draw air across the internal components, there will also be radiant heat from the server casing. This radiant heat will effect the equipment adjacent to it. To minimise the effect on adjacent equipment, gaps should be left between the mounts and the highest load equipment should be mounted as high as safely possible.

As stated all gaps must be blanked off. The servers draw air in from the front. Cabling must not obstruct this area. The gaps can therefore be filled with cable management containment.

As cables at the front of the rack can obstruct the inlet airflow and cables and cable arms to the rear can obstruct the server exhaust. The deeper racks will not only help with turbulence but also facilitate cable management to the rear of the rack.

Where a high load item of equipment has been identified, radiant heat transfer can be assisted by 1U 19” mounted fan trays. A small number of servers have exhaust vents to the side. In this case the width of the rack shall be increased and the same criteria applied to the side as we have applied to the rear. Careful planning is required to ensure that side venting servers do not vent towards each other.

Although the primary means of delivering air to the servers is via the front inlet vents, a limited amount of air from the pressurised floor plenum can be used to assist with radiant heat transfer and provide positive upward desired airflow

internal rack recirculation

cold air short circuit

hot isle air short circuit any ISS product

any ISS product

any ISS product

any ISS product

front door 65% open 35% restriction

rear door 65% open 35% restriction Internal Recirculation Hot Air Short Circuit Cold Air Short Circuit Blanking Panels Without Blanking Panels (undesired airflow)

With Blanking Panels

(desired airflow) Rear door 65% opening 35% restriction Front door 65% opening 35% restriction

air pressure to the rear to assist in top vented exhaust air. The first 4U of the 19” mounts must be kept clear of equipment and not used for storage of other materials. An example of the loads discussed. Each IBM eserver BladeCenter has an output of 1365 Btu/h (400W) 13650 Btu/h (4000W) depending on configuration. The individual weights and total rack weight must be recorded. This is to ensure that heavy items to the upper part of the rack to not affect the rack stability and Health & Safety issues have been observed in relation to manual lifting. The total weight of a rack or freestanding equipment must come within the restriction of the raised floor. Point load is the pressure applied to an area of 25mm2 and the Uniform Distributed Load (UDL) to an area of 1000mm2. If an item of equipment has four feet then the point load for each foot is the total weight divided by four. Care should be taken when two items of heavy equipment have a foot each on one tile. If the equipment exceeds the parameters of the raised floor tile and/or mezzanine/slab then a structural frame shall have to be considered. When moving the equipment into place the rolling load of the raised floor needs observed and so spreader sheets should be used. The client is obliged to comply with the Waste Electrical and Electronic Equipment (WEEE) Directive, The Restriction of the use of certain Hazardous Substances in electrical and electronic equipment (ROHS) Directive and the European Union electromagnetic Compatibility (EMC) regulations, which include the Harmonic Current Emissions standard EN 61000-3-2 and bear the appropriate CE m a r k . T h e c l i e n t s a re a l s o responsible for t h e Portable Appliance Testing (PAT) of their own equipment.

High-Power Configurations

“Fact: For a given data center (at 225 W/ft2) with an A/C failure, the reaction time to reduce load is less than ~ 35 seconds to prevent redlining & shut-down”

Datacentre configurations over 100watts/ft2 or 1000watts/m2 are being requested on a regular basis. However, after analysing most requests, we have come to several conclusions:

•

•

At average power levels in excess of 100watt/m2, datacentre design and layout passes from ‘efficient optimisation’ to ‘mission critical’. All best practices as listed in section 1.2 above must be implemented and monitored regularly.

In addition, suite specific upgrades must be costed in. The most common upgrades would be:

•

•

HVAC HVAC

Intake Intake

Ceiling Return Air Plenum

HVAC Return Air Grills

Suspended Ceiling

Supply Air Plenum

Tailor-made – Customers requesting high power solutions are, in every case, requesting a nonstandard solution. IXEurope has qualified engineers who are happy to provide

consultancy to assist in planning and costing out tailor made solutions.

Cost/Benefit analysis – In most cases, the capital expenditures and ongoing running costs of a highpower suite are large and are a key factor of the decision to invest.

Assisted Air-Handling here, systems to rapidlyextract excess hot air are layered onto existing systems to specifically support the general datacentre. APC recommends installing either a ceiling void general extraction system or, in extreme cases, direct from the rack air extraction. Closed Air-Handling – here, tailor-built air handling systems are installed after closing a BusinessSuite off from the shared air-supply plenum.

Future Considerations

There are a number of products on the market such as the fan assisted rear doors; Rack tops already mentioned; Water-cooled racks; High-level fan coil units etc. Fan assisted racks improve the air flow through that particular rack but careful planning of the suite and subsequently, the datacentre is still to be employed. The potential risk from water-cooled racks is still to be assessed. High-level fan coils systems are installed to take warm air from the hot corridor cool it and then discharge it into the cold corridor. Whereas these units do not use water as a medium but gases, you still have to install condense drip trays and drainage. Fan assisted floor tile vents can deliver more air to the cold corridor but overall balance must be monitored

To accommodate future requirements, in addition to reducing the distance between runs of CRAC’s, IXEurope is investigating the feasibility of installing additional extract ductwork above the Hot Corridor. This extract system shall remove approx 50% of the warm air in the return air path, deliver it to an external AHU where the temperature shall be reduced and then the air returned to the return air intake of the CRAC’s. Suitable controls shall be implemented so that the supplement system can be regulated to support changing load cultures. By removing a portion of the air and treating it remotely, we can implement Humidity regulation, outside of the technical space and introduce an element of free cooling, when the external conditions are suitable.

Where a traditional office floor has been converted to a Datacentre you will normally have a suspended ceiling that acted as a return air plenum for the ceiling mounted air conditioning. This can be reused as above as a hot air return path thus separating the two air flows and reducing air mixing.

Project Management

The IXEurope Project manager shall work closely with the client to design and implement the suite build. The PM shall request the specification of all equipment to be installed in relation to electrical load/weight/heat output, so that he/she can ensure that the correct power delivery is installed, the weight distribution is within the raised floor parameters and that the racks are configured to meet the client demands. The PM shall assist the client in collecting the data in its many formats, i.e. 1 watt = 3.413 Btu/h, 12000 Btu/h = 1 ton of cooling. It should be noted that IXEurope observe strict phase balancing and so shall not supply a Single Phase and Neutral (SPN) supply in excess of 32 Amps. Manufacturers can supply Three Phase and Neutral (TPN) for equipment that requires such power.

In addition to floor plan layout, IXEurope shall also produce rack elevation details in conjunction with the client so as to ensure the equipment is distributed for cooling concerns and that the stability of the rack has not been compromised.

Ongoing Management

Due to the extreme limits of air handling systems, regular monitoring of suite thermics need to be carried. Risks of hot spots developing are increased and the impact not only more extreme but potentially unstable. Any suite changes – cabling, server installation, server utilization rates, movement of doors, blanking plates, etc. needs review on a regular basis.

Extreme Situations

Customers working solutions uniquely based on blade or dense server configurations may run all systems over 4KVA per rack. In this situation, the only options are: •

•

The closing comment for this paper is that high density equipment such as Blade servers have not changed the laws of Physics. The power draw to processing ratio remains the same however by packing it all into one box has vastly improved the management of the system. As Datacentre operators we shall continue to manage the same density of processing capacity by following the guidelines within this paper .

Reference Material

If the system has the following characteristics

Use the following base cooling approach

With the following solution for high density enclosures

Under 3kW per rack average, with very high ceilings or under 100kW total power High average per-rack power or over 100kW total power

Alternate high density solution for mainframe environment

Very Large Air Circulation Zones – In these cases, standard specifications of 50 to 60 ft2 of space per rack become necessary to avoid hot air short circuits.

Rack Cooling – The industry has been promoting rack cooling systems. At the moment, IXEurope does not see a cost benefit in implementation. In general, there are two key risks – rack cooling systems naturally have the potential for risk, with a single point of failure that will need regular

servicing and additional risks of condensation. Suppliers such as APC and Liebert claim to have this under control. Confirmed results are as yet unavailable.

IBM BladeCenter Planning and installation guide (August 2003)

DELL White paper Rack Impacts on cooling for high-density servers (Aug 2002)

APC White papers #4 rev 4, #5 rev 3, and #55 rev 1 HP Optimising Datacenters for high density

computing 02/2004

Liebert IT White paper Managing Extreme heat 2004

The UpTime institute High Density Trends version 1 European Power Supply Manufacturers Association (EPSMA) guidelines EN-61000-3-2

EMC Act 1998

About IXEurope

IXEurope provides outsourced datacentre services in four countries UK, Germany, France and Switzerland -through its network of datacentres.

As a specialist in datacentre services, IXEurope offers its customers access to flexible outsourced solutions while they maintain overall control of their projects. By continually focusing on quality, IXEurope gives clients cost effective access to expertise and exceptional service levels.

IXEurope delivers its services to systems integrators and IT consultants; hosting and managed service providers; and network service providers. Working in partnership with our solution providers, IXEurope datacentres house IT and networking systems for global enterprises, financial institutions, government agencies and e-commerce and mid-sized companies.

For more information please visit the website at:

Contacting the Company

IXEurope Datacentres London West & London City

T: +44 (0) 20 7689 1400 E: [email protected] Paris T: +33 (0) 1 48 17 65 00 E: [email protected] Geneva T: +41 (0) 22 544 98 00 E: [email protected]

Frankfurt & Mörfelden

T: +49 (0) 69 920 42-0 E: [email protected]

Düsseldorf

T: +49 (0) 211 77 058-0 E: [email protected]

Zürich 1 & Zürich 2

T: +41 (0) 1 355 69 00 E: [email protected] IXEurope Headquarters T: +44 (0) 800 073 0766 F: +44 (0) 208 476 7821 E: [email protected]

A: PO Box 485, West Drayton UB7 0NJ