Alcatel-Lucent Modular Cooling Solution

(1)

T E C H N O L O G Y W H I T E P A P E R

Current heat exchange methods for cooling data centers are inherently inefficient in meeting the requirements imposed by today’s dense, high-speed computing equipment, and they can contribute considerable costs to operating expenditures. The Alcatel-Lucent Modular Cooling Solution (MCS) can reduce energy consumption (with respect to transport of waste heat to the location’s heat sink) while offering a coefficient of performance (COP) much greater than current approaches. Using a building’s chilled water secondary loop the MCS can operate with as little as 10 percent of the electrical power used by a comparable Computer Room Air Conditioner (CRAC). This paper presents the results of tests performed on the Alcatel-Lucent MCS to determine the effects of pump unit power loss on the system and data center equipment.

Alcatel-Lucent Modular Cooling Solution

Redundancy test results for pumped, two-phase modular cooling system

(2)

The Challenges of Data Center Cooling

Data center cooling continues to be a major challenge for data center operators. The densely-spaced, high-speed hardware that provides the computing power to enable a variety of data center operations places significant demands on data center cooling and power systems. In fact, recent reports estimate that as much as half the energy used by a data center is dedicated to powering cooling systems¹. In an effort to operate more cost-effectively and efficiently, data center operators continue to search for new ways to reduce energy consumption and address environmental and green computing initiatives.

But traditional air cooling approaches prevent data center operators from optimizing data center real estate, since they do not deliver adequate cooling unless equipment is spaced far apart. At the same time, air cooling systems add operational costs because they require installation of expensive raised floors to be used as cool air channels. As more computing equipment is installed, more cooling units are required, which places even more strain on space and power. And additional costs are incurred when operators have to reconfigure the data center for aisle containment (either hot or cold) or in-line heat removal, dedicated cabinets, and IT hardware upgrades.

Compounding the issue is the fact that operators must also navigate through a variety of cooling solutions from multiple vendors, while keeping an eye on industry efforts around emerging standards designed to define the requirements for a “green data center”. Unfortunately, in this constantly changing environment, operators need a solution that can be applied today rather than tomorrow.

And that solution must align with a range of business objectives, including cost-effective operation, optimization of energy efficiencies, reduced management complexity, as well as the need to meet green computing initiatives.

The Alcatel-Lucent Modular Cooling Solution (MCS) offers data center operators with large data centers or high density electronic equipment an end-to-end system that can reduce total cooling costs by up to 50 percent, reduce data center energy consumption by up to 90 percent (with respect to transport of waste heat to the location’s heat sink), and offer performance far exceeding traditional approaches to data center cooling.

With the Alcatel-Lucent MCS, data center operators can:

• Decrease energy costs by virtually eliminating the need for supplemental air conditioning units

• Maximize space utilization by cooling more servers per cabinet and more cabinets per data center

• Eliminate the need for hot and cold aisles and isolated high power cabinets/racks

• Minimize the need for specially conditioned rooms

• Address eco-sustainability requirements/issues by reducing a data center’s carbon footprint, energy consumption, and noise output

Alcatel-Lucent Modular Cooling Solution

The Alcatel-Lucent MCS uses phase change or two-phase modular cooling to enable cooling as close to the heat source as possible. Rather than dispersing waste heat through the data center, it transfers it into a liquid coolant and pipes it outside the data center (Figure 1). This is accomplished by:

• Placing an evaporator inside the cabinet, frame or rack — close to the source of the heat

• Pumping refrigerant at low pressure through micro-channel heat exchangers as a liquid

• Converting the liquid to a gas within the evaporator (this is the phase change, at which point the heat is captured)

• Returning the gas to the pumping unit, where it is re-condensed to liquid

• Removing rejected heat via a chilled water system or outdoor condenser unit

1 “Cooling the datacenter: “hippie engineering” meets modern IT”, Amy Coombs, Ars Technica, April 2010, http://arstechnica.com/business/

news/2010/04/hippie-engineering-meets-information-technology.ars

(4)

Figure 1. Cool uniform air distribution with Alcatel-Lucent’s Modular Cooling Solution

This approach offers a major improvement over traditional methods that rely on forcing ambient air through the data center to keep equipment cool. With cooling coils much closer to the heat source, phase change/two-phase modular cooling is far superior in transporting heat away from equipment² than other methods. As a result, data center operators benefit from:

• Lower energy requirements, operating expense, and risk

• Improved real estate utilization

• Improved ability to meet green computing/environmental initiatives

To validate the benefits of phase change/two-phase modular cooling, Alcatel-Lucent deployed the Alcatel-Lucent MCS at its Naperville Modular Cooling Lab in Illinois and subjected it to a variety of tests. The first series of tests were designed to determine the effects pump unit power loss and variations in environmental humidity would have on the system and data center equipment³. The test results summarized in this document describe the effects a loss of a series of Modular Cooling Units (MCUs), similar to what may occur in a partial outage of a redundant system, will have on the Alcatel-Lucent system and data center equipment.

This experiment was performed on the Naperville prototype MCUs and the Alcatel-Lucent MCS as a whole. All testing was conducted while IT equipment within the frames/cabinets was under full power. It should be noted that the lab was designed to provide for a wide range of experimentation.

Field site conditions and equipment configurations will vary greatly and may not reflect conditions at the Naperville lab.

Testing MCUs in the Alcatel-Lucent MCS

The Alcatel-Lucent Modular Cooling Lab in Naperville, Illinois, has 10 equipment frames fully loaded with IT computing hardware. The operating environment is 324 ft² (30 m²) with a ceiling height of nine feet (2.74 m), and it is thermally isolated from any benefits that could be offered by the building cooling system. All perforations and venting in walls, floors and ceiling have been sealed. In addition, ceiling tiles are foil backed to provide an acceptable level of humidity control.

As a result, this lab is, for the most part, a sealed and isolated environmental enclosure. This enables experimentation without interaction or assistance from any outside cooling source.⁴

2 For more information about the Alcatel-Lucent MCS see the Alcatel-Lucent white paper “Keeping Cool In The Data Center: A High-Efficiency Modular Solution.

3 For details of the environmental humidity tests, see the Alcatel-Lucent white paper “Alcatel-Lucent Modular Cooling Solution: Pumped, two-phase modular cooling humidity variation test results”

4 For more details about the Alcatel-Lucent Naperville Modular Cooling Lab, see the Alcatel-Lucent white paper “Alcatel-Lucent Modular Cooling Solution: Pumped, two-phase modular cooling humidity variation test results”

(5)

Equipment frames are configured with the standard hot aisle/cold aisle arrangement with typical cabinet spacing. The lab’s small size and the very high heat load in the room are not typical of most data centers. Therefore, to prevent condensation, the cooling system regulates the refrigerant temperature to maintain a temperature above the dew point temperature. But it is still very difficult to accommodate changing temperatures by maintaining a steady relative humidity (RH) level. To ensure conditions for this experiment could be correlated with typical site conditions, the system was set to maintain a room humidity target of 45–50 percent RH.

Obviously, not having to add humidity will lower operating costs. But, more importantly, low humidity conditions are optimal for the Alcatel-Lucent MCS or any cooling solution because condensation issues are greatly minimized.

The MCUs of the Alcatel-Lucent MCS are designed to be installed within cabinet enclosures, and they have been tested to and comply with NEBS Level 3 (including NEBS Zone 4 Earthquake) ETSI, UL, IC, CE and CSA standards. With this approach, the robust hardware can endure and continue to operate in typical telecom industry environments.

There are 30 MCUs installed in the equipment frames in the Modular Cooling Lab. Figure 2 shows some of the equipment frames, as well as the pump unit, MCUs, overhead distribution manifold, and flexible refrigerant lines.

Pump unit configuration

The Alcatel-Lucent MCS in the Modular Cooling Lab is supported by only one chilled water pump unit. Note that this pump unit is equipped with two pumps. The second pump is a redundant pump that will kick in if the first pump fails. Some data center designs may require a high level of system redundancy. For these data centers, it may be prudent to supply the MCUs within an equipment frame, or alternating frames, with separate cooling loops, each with their own pump unit. In this way, if a catastrophic pump unit outage does occur, the remaining operational MCS systems will continue maintaining appropriate operating conditions and the macroscopic thermal impact will be limited.

A redundant configuration requires appropriate sizing to ensure that cooling capacity lost by an out of service pump unit is recovered by the remaining operational MCS systems. To make this possible, the site should be designed to allow for enough of a cooling capacity margin to protect against pump unit outage. Therefore, a fully redundant system should split the heat load between multiple pump units. It is imperative that appropriate operational conditions exist to support proper unit operation (chilled water temperature, flow and site humidity control). A chilled water secondary cooling loop used in the site design may look similar to the one presented in Figure 3.

Figure 2. Equipment frames in the Alcatel-Lucent Modular Cooling Lab

(6)

Figure 3. Redundant MCS site configuration

Test conditions

To simulate a pump unit outage on a one pump system, half (50 percent) of the MCUs in the Modular Cooling Lab were removed from operation. As a result, the single pump unit continued to receive the same, full thermal load as it had with all MCUs operational.

This created a scenario similar to the macroscopic dynamics of a multi-MCS installation with multiple pump units. In a multi-unit environment, the heat exhausted from out of service MCUs will be rejected into the room,

thereby causing the room temperature to rise slightly. Assuming the site uses an Alcatel-Lucent MCS as the primary cooling solution, the rise in ambient temperature will slightly increase the temperature of the intake air flowing through the operational MCUs. Therefore, the cooling capacity of those units will increase to the point at which captured heat is rejected from the non-operating MCUs and thermal stabilization occurs. Results will be the same whether a chilled water or compressorized secondary cooling loop system is deployed.

Provisions for this experiment included monitoring ambient room conditions during experimentation.

To simulate a loss of 50 percent capacity, the supply refrigerant lines to half of the cooling units were disconnected in a pattern shown in Figure 4.

Circulating refrigerant supplied to alternating cabinets.

Pump Unit Pump Unit

Building Chilled H2O Modular

Cooling Units

Figure 4. Modular Cooling Lab configuration for pump unit failure experiments

1

2

1

2

1

2

1

2

1

Pump Unit 5C-238 Naperville Modular Cooling Solution Lab

RPP

2

Number of MCUs taken out of service

(7)

This alternating pattern represents a method of connection to a redundant system in the field with alternating MCUs within the frame supported by alternating pump units.

The equipment frames were run under these test conditions for a period of 22 hours to allow sufficient time for the room temperature to stabilize.

Temperature data was recorded throughout the experiment, along with sound pressure levels. At the end of the test interval, supply line refrigerant connections were re-established and conditions monitored for an additional period of 12 hours. Figure 5 shows the top of the cabinets in the Modular Cooling Lab, as well as the manifold and the supply lines that were disconnected. This is an experimental lab; aesthetics are greatly improved in a customer installation.

Test results

With half of the cooling units disconnected there was an obvious change in the cabinet temperature profile, exactly as expected. An additional heat load of 1.8 kW in the room was created by the steam generating humidifier that was operating constantly at full capacity. As the room temperature rose, the air had a greater capacity to retain moisture and the percentage of RH dropped. To adjust for the drop in percentage of RH, the humidifier generated more steam. This was an extremely dynamic condition until stabilization occurred.

Testing took place in the Chicago area in February, at which time the average outdoor temperature was 22°F (–5.6°C) and the outside air was very dry. Alcatel-Lucent no longer controls indoor humidity in the building in which the Modular Cooling Lab is located. As a result the humidity in the hall just outside of the lab was 18 percent RH. The walls, floors and ceiling within the lab quickly wicked the moisture away, so it was an uphill battle to keep the room humidity near 50 percent RH in these conditions.

A psychometric chart can best relay the impact of maintaining a specific humidity level as temperature rose in the lab. With initial start conditions of 68°F (20°C) and 47 percent RH the corresponding humidity ratio was 0.011 grams water/gram dry air. During the test, the average room temperature rose to 79°F (26°C) with a humidity

of 42 percent RH after stabilization.

This resulted in a humidity ratio of 0.016 grams water/gram dry air. As a result, there was 50 percent more moisture suspended in the room during the test. By the conclusion of the test, the dehumidifier had to remove this moisture and, therefore, recovery time was lengthened as the system maintained the refrigerant temperature above dew point.

Figure 6 is an infrared image of the exhaust side of the hot air aisle in the lab. The temperature scale along the right side indicates the temperature

Figure 5. Disconnected MCU supply lines for the Alcatel-Lucent MCS pump unit failure test

Figure 6. Infrared image of the exhaust side of the hot air aisle in the Modular Cooling Lab during testing

40°C

35°C

30°C

25°C

20°C

15°C 39.1°C

17.0°C

(8)

gradient, with blues on the cool end and yellow/white with increased temperature. The operational MCUs were exhausting air at 64.4°F (18°C), while the non-operational MCUs were exhausting unconditioned server exhaust air at up to 100°F (38°C). The MCU fans were operational. Note the pattern created by alternating MCU disconnection.

Figure 7 is the corresponding digital image of the same angle shown in Figure 6. All cabinet doors remained closed during test.

Data gathered during the experiment is presented in Table 1. The room sound pressure level increased as the fans within the servers sped up to compensate for the increased room temperature. When the experiment concluded, sound levels dropped to their initial state. The introduction of the Alcatel-Lucent MCS can lower acoustic emissions and server power consumption. This was proven at the Alcatel-Lucent Plano IPTV lab where sound levels dropped by 20 dB_(A) after the Alcatel-Lucent MCS was installed. (A-weighting attempts to match the response of the human ear.)

Table 1. Summary of data collected during testing

EXPERIMENTAL DATA

Initial Sound Pressure Level 73.1 dBA

Sound Pressure Level During Experimentation 75.4 dBA

Final Sound Pressure Level 73.1 dBA

Chilled Water Supply Pressure 42 psig (2.9 bar)

Chilled Water Return Pressure 27 psig (1.86 bar)

Thermocouples were placed in the exhaust aisle approximately 3.5 feet (1 m) above the floor in the center of the aisle and carefully placed so that no MCU would exhaust air directly on them (Figure 8).

This placement enabled a temperature measurement of mixed aisle air at approximately waist high.

Figure 8. Temperature measurement was conducted with thermocouples in the exhaust aisle

Intake Aisle 2 Exhaust Aisle Intake Aisle 1

Cab 3 Cab 1 Cab 2 Cab 4 Cab 5 Cab 8Cab 7

Cab 6

Cab 9 Cab 10

Figure 7. Digital image of the exhaust side of the hot air aisle in the Modular Cooling Lab during testing

(9)

The green line shown in Figure 9 is a measurement taken at the pump unit, which is located in the server intake aisle approximately 5.5’ ( 1.7 m) above the floor.

Figure 9. Temperature profile measurements taken at the pump unit in the server intake aisle during testing

The Modular Cooling Lab is also equipped with temperature sensors in the ceiling above the intake and exhaust aisles (Figure 10). These ceiling mounted probes are connected to a monitoring system offered as an option with the Alcatel-Lucent MCS, along with a refrigerant leak detection system, which is also connected to the monitoring system.

In the Modular Cooling Lab, the system is set up to monitor and trigger alarms for high room temperature, refrigerant leak, pump outage and status of a fan unit. If an alarm is triggered, the system is configured to send an e-mail and text message.

Figure 10. Temperature sensors in the ceiling above the intake and exhaust aisles in the Modular Cooling Lab

Temperature sensors above exhaust aisle

(10)

Data recorded for a two day interval from ceiling mounted sensors is presented in Figures 11, 12 and 13. Temperatures are presented in °F. The figures show that stabilization occurred quickly as the system dynamically adapted to changing environmental conditions. The system can be configured to display English or SI units, and these figures display English units.

Figure 11. Two day intake aisle temperature profile

Figure 12. Two day exhaust aisle Sensor 2 temperature profile

Figure 13. Two day exhaust aisle Sensor 1 temperature profile

(11)

Recovery time, when all system components are brought back into operation, is important in these types of scenarios. Had the room percent RH been dryer (i.e., moisture was not added) recovery would have occurred more quickly. The system was controlling moisture to prevent condensation from forming on components. As indicated in Figures 14, 15, 16 and 17 the room temperature fully recovered in approximately 50 minutes.

Figure 14. Exhaust aisle Sensor 1 two hour recovery window

Figure 15. Exhaust aisle Sensor 2 two hour recovery window

Figure 16. Intake aisle 1 sensor two hour recovery window

(12)

Figure 17. Intake aisle 2 sensor two hour recovery window

This recovery time may vary somewhat depending on how heavily each MCU is thermally loaded, the temperature of the chilled water with a chilled water pump unit, room humidity and level of loading for each pump. However, the recovery time is very quick compared to other cooling meth- odologies and the system provides a more consistent temperature throughout the data center.

Conclusion

The Alcatel-Lucent MCS is a highly dynamic solution that can continually adjust and adapt to its operating environment. And it is capable of handling data center cooling requirements under a variety of operating conditions.

Testing of the system in the Alcatel-Lucent Modular Cooling Lab with 50 percent of MCUs removed from service revealed that the solution can maintain proper temperature control in the event of a partial system outage and minimize impact to the rack mounted equipment. The rack mounted equipment was never exposed to inlet air temperatures greater than 80°F (26.7°C), which is well below the maximum operating intake air temperature of 95°F (35°C) specified by many IT hardware manufacturers.

For data centers that require a high degree of redundancy (beyond that provided by the redundant pump in the pump unit) a site should be designed to allow enough cooling capacity margin for protection against the complete failure of a pump unit . A fully redundant system would split heat load between multiple pumping units. It is imperative that appropriate operational conditions exist to support proper pump unit operation (chilled water temperature and flow). And when high availability is required, proper planning is necessary to ensure system capacity is not exceeded, as would be the case if a pump unit was removed from operation.

For data center operators with large data centers or high density electronic equipment, the Alcatel- Lucent MCS is an end-to-end system that can address immediate cooling requirements. It can be relied upon to maintain cooling levels in even the most demanding data center conditions. In addition, it can reduce total cooling costs by up to 50 percent, reduce data center energy consumption by up to 90 percent (with respect to transport of waste heat to the location’s heat sink), and offer a coefficient of performance (COP) far greater than traditional approaches to data center cooling.

Acronyms

COP coefficient of performance MCU Modular Cooling Unit RH relative humidity

(13)

About the Author

Bill Scofield Bell Labs

Thermal/Environmental R&D Alcatel-Lucent

Bill Scofield is a Bell Laboratories innovator and inventor responsible for Thermal/Environmental Design, Analysis & Assessment. Bill is the principal Designer and Inventor of the Alcatel-Lucent Modular Cooling Solution. He is also developing several other leading edge cooling technologies from the component level through site level. He has a long history working directly with global customers on-site to improve their deployed hardware’s thermal performance and reliability. This coupled with his background in hardware regulatory compliance requirements and standards for environmental conditions enable a clear understanding of customer needs. His efforts have enabled improvement in both reliability and performance of customer equipment thereby minimizing service outages.

Bill’s expertise and his background in hardware compliance requirements have guided the design of the Alcatel-Lucent Modular Cooling Units to conform to telecom’s most stringent hardware requirements, including meeting NEBS Level 3. Bill has five issued patents, eight pending patents and refrigeration certification issued by the US Environmental Protection Agency. He earned both Masters and Bachelors of Science degrees in Mechanical and Aerospace Engineering from the Illinois Institute of Technology.

(14)

www.alcatel-lucent.com Alcatel, Lucent, Alcatel-Lucent and the Alcatel-Lucent logo are trademarks of Alcatel-Lucent. All other trademarks are the property of their respective owners.

SBG5677100402 (05)

Alcatel-Lucent Modular Cooling Solution