Saving energy in the Brookhaven National Laboratory Computing Center: Cold aisle containment implementation and computational fluid dynamics modeling
Student Intern
Earth and Environmental Engineering, Columbia University, New York, NY 10027
Another Intern
Chemical Engineering, University of Rochester, Rochester, NY 14627
Lab Mentor
Energy Utilities Department, Brookhaven National Lab, Upton, NY 11973
Another Mentor
Abstract
This research project in the Energy Utilities Department is a study of energy efficiency
improvement in the Data Center building at Brookhaven National Laboratory. The Data Center
houses rows of large computer servers. These servers, like all computers, give off a substantial
amount of heat and must always be kept cool to run efficiently. The total electricity usage of data
centers, primarily computer power and A/C units, accounted for 1.6% of all US electricity usage
in 2006, and is projected to increase by 12% annually.1,2 Due to the growing importance of data centers in an increasingly computer-dependent world and the large amount of cooling necessary
to keep them running, the DOE has recently committed to a new energy efficiency standard for
data centers that requires a 30% reduction in energy usage from their expected energy
requirements, based on size.3 This study focuses on the modeling and implementation of curtains and baffles to save cooling energy by containing the cold air entering the server rooms. These
curtains ensure that none of the cold air is cycled back into the A/C units without first passing
through and cooling the servers. A 3D model was made of the airflow and temperature
distribution in the data center using Ansys CFX, a Computational Flow Dynamics software. A
temporary trial of the containment was performed using plastic sheets to verify the computer
model and identify major benefits and problems with containment. The trial showed that
containing two of three aisles with incoming cold air led to an 8.9°F decrease in the hottest inlet temperature to the computers. The model supported this excellent result and can now be used to
model other curtain placements and configurations to find the most efficient and inexpensive
I. Background
A. Data Centers and Energy Usage
Data centers are extremely important, large users of energy. In 2006, they accounted for
about 1.6% of all US electricity usage,1,2 and this figure is projected to grow by 12% annually.1 Data centers use this energy not only to run the extremely powerful computers housed there, but
also to power air conditioning units throughout the facility to prevent the computers from
overheating. Of the 23.8 million kilowatt-hours of energy that the Brookhaven Data Center
consumes each year, about 60% powers the computer servers and 40% goes to cooling.4 While it is nearly impossible to reduce the amount of energy used by the computers without reducing the
Center’s computing capabilities, there are a number of techniques that can be employed to cool
the servers more efficiently. Lowering the amount of energy required to cool a data center would
yield large energy savings.
B. Air Flow in a Data Center and Cold Aisle Containment
In many data centers, including the Data Center at Brookhaven National Laboratory, the
air conditioning units pump cold air into an area below the floor called the plenum, as shown in
Figure 1. This cold air comes up through porous floor tiles into an aisle with computer racks
facing inward, called the cold aisle. This air in the cold aisle goes through the small fans in the
computers and into the neighboring outflow aisle, called the hot aisle, where all the server racks
are back-to-back. The air then rises up from the hot aisle and eventually reenters the air
Figure 1. Diagram of typical airflow inside a data center
There are two main inefficiencies that result from this. The first and most significant
source of inefficiency is the fact that the recirculated cold air wastes fan energy because this air
does nothing to cool the actual servers, but it leaves the cold aisle and enters to the air
conditioner unit simply to be blown back to the cold aisle it came from. The second source of
inefficiency is the fact that the entire room is being cooled. Any cooling leakage that the room
experiences, for example from an open door to an adjacent room or the sun’s radiation on a hot
day, will require that the air conditioners work harder to counteract those effects and chill the
entire volume of the room.
Both of these sources of inefficiency can be corrected with containment. In the
containment model, shown in Figure 2, the top of the cold aisle is blocked off. In a
three-dimensional view of the aisles, the two sides of the cold aisles would also be shown as blocked
off. This containment does not allow any cold air to escape the cold aisle without first passing
through the servers. This solves the first problem because it makes the system such that the air
must first pass through and cool a server to return to the air conditioner and be pumped into the
cooled has been reduced to simply the cold aisles, instead of the entire room. The cold aisles will
therefore become colder, while the ceiling above the cold aisles, where the air returns to the air
conditioner, will be hotter. Using cold aisle containment is one of the most energy efficient
options and often the least expensive method of reducing cooling energy because it does not
require the purchase of a new, more efficient air conditioner and its purpose is simply to make
sure the cold air is channeled only to where it can best be used.
Figure 2. Diagram of airflow inside a data center with cold aisle containment
C. Proven Benefits of Cold Aisle Containment
Cold aisle containment, as well as containment in the plenum and within individual
server racks, can significantly reduce the cooling load of a server room. The national laboratory
at Savannah River reduced their data center’s cooling load by 43% and had a payback of just two
and a half months from their containment initiative.5 Lawrence Berkley National Laboratory also implemented a large-scale containment project throughout their data center with a payback of
two to four years.6 Due to the large reduction in cooling requirements of their existing servers, Lawrence Berkley added new servers to rooms that were previously at maximum capacity
capacity. They estimate the total increase in cooling capacity at 21%.6 The alternative to adding this quantity of new servers would have likely been a large construction project to add an
additional server room extension to the data center.
II. Methods
A. Overview of the Study
This study is comprised of both a real-world experimental component and a computer
modeling component. As a real-world experiment, temporary cold aisle containment baffles were
put in the RCF room. Temperature, pressure, and other useful data were taken at strategic points
along the hot and cold aisles and at the air conditioners before and after the containment
experiment. The data taken before the containment experiment were used to build a
three-dimensional computer model of airflow and temperature distribution through the room without
and with the inclusion of containment structures, modeled like those used in the real-world
experiment. The results of the “after containment” computer model were then compared to the
results found in the actual experiment in order to validate the model.
B. Overview of the Room
The entirety of this study is focused on Brookhaven National Laboratory’s RHIC
Computing Facility (RCF) room, which is a room in the Computing Center dedicated to the
analysis of data from Brookhaven’s Relativistic Heavy Ion Collider (RHIC). The RCF room
contains three cold aisles, three hot aisles, and two primary air conditioning units (AC1 and
AC7). Additionally, some of the computer racks have small “top-hat” air conditioning units,
which sit directly on top of the rack. These units take hot air in from the top of the hot aisle, cool
C. Real-world Experiment
The temporary cold aisle containment curtains, consisting of thin plastic sheets, were
placed on the top and sides of two of the three cold aisles of the RCF room. Pictures of the RCF
room before and after the containment was implemented can be seen in Figures 5-6 and Figures
7-8 respectively. The containment was implemented under careful observation for approximately
12 hours before removal.
Figures 5 and 6. Pictures of a cold aisle before containment
D. Computer Model
A three-dimensional k-epsilon turbulent airflow computer model of the RCF server room was
built using a Computational Fluid Dynamics modeling (CFD) software called Ansys CFX. The
model was built in two configurations, one without containment, shown in Figure 9, and one
with containment, shown in Figure 10. The before containment model was built based on
temperature and pressure data in the server room. Adiabatic wall boundaries were added to the
room model where the actual containment sheets were hung in the real-world experiment to
create the after containment model. No other inputs of boundary conditions, temperatures, or
pressures were changed in creating the after containment model. These computer models were
built in order to compare airflow and heat distribution in the before and after containment
scenarios.
III. Data and Results
A. Real-World Experiment Data and Analysis
The real-world experiment yielded excellent results and showed significantly colder cold
aisles, which will result in energy savings if permanently implemented. Figures 12 and 13 show
floor plan views of the data center near the top of the computer server racks with a color contour
of temperatures before and after containment, respectively. Each small black box represents a
location where temperature is monitored. As shown in these contour maps, the cold aisles are
significantly colder after containment. In fact, the experimental data shows that the containment
resulted in an 8.9° F decrease in the hottest cold aisle temperature. Since the air conditioning
units are ideally set so that the hottest temperature in a cold aisle is the maximum allowable
operating temperature of the servers, a reduction in hottest cold aisle temperature directly leads
to a decrease in chilling requirements. It is also important to note that AC7, one of the two air
conditioner units in the room, significantly reduced its chilled water intake during the experiment
and was cooling at approximately 65% capacity. This and the very low cold isle temperatures both suggest that if containment was implemented, this air conditioner unit could be turned off
permanently or both units’ cooling could be significantly reduced, greatly decreasing energy
Figures 12 and 13. Temperature floor plan views of RCF server room before and after containment B. Computer Model Validation
In validating the computer model, two main regions were compared for temperature. The
first is the server inlet air temperatures at the top of the servers in the cold aisles that were
contained. This is an important temperature comparison because if this model were used to
predict results of containments that were not experimentally tested, an approximation of the
decrease in top of cold aisle temperature would indicate how much less cooling would be
required after containment implementation. Figures 14 and 15 show the experimental
temperature data uncontained and contained respectively, while Figures 16 and 17 show the
model prediction uncontained and contained respectively. The black boxes highlight the two cold
aisles that had containment in the contained model for comparison. Comparing the uncontained
scenario in Figures 14 and 16, the temperature profiles match very well because the experimental
data in Figure 14 was used to build the model shown in Figure 16. The true test of the model’s
the actual experimental data taken in the containment scenario, shown in Figure 15. These
temperature gradients also match very well and support the model’s accuracy.
Figures 14 and 16. Uncontained scenario of experimental data and computer model, respectively
Figures 15 and 17. Contained scenario of experimental data and computer model, respectively
The second focus of comparison was the temperature of the air returning to each of the
two air conditioner units. Table 1 shows the uncontained and contained temperatures of return air
to the air conditioners in both the experimental data and the model predictions. The model’s
return air predictions are consistently colder than the experimentally measured temperatures.
This could likely be fixed by adjusting boundary conditions of the model slightly. Despite
to contained are quite close to the experimentally measured values. In both the experiment and
the model, AC1 return air became significantly warmer while AC7 return air became slightly
colder. Since the change in temperature was predicted well by the model, the overall colder
predicted temperatures are not a significant issue. It is assumed that if this model is implemented
for a containment that was not experimentally tested, there will still be uncontained temperature
data from the room. This experimental uncontained value of return air temperature, if warmer
than the model’s predicted uncontained
value, can be used to adjust the contained
model prediction and determine a more
accurate temperature value as long as the
change in temperature predicted by the
model is accurate.
C. Computer Model Data and Analysis
Figures 17 and 18 show the temperature cross-sections of the room in the uncontained
and contained computer models. The cold aisles in the uncontained model have a large
temperature gradient from bottom to top, signifying that the bottom servers must be overcooled
in order to reach the desired temperature in the upper servers. In the containment model, the
temperature remains consistent and colder throughout the aisle, so the air conditioners can cool
all the servers to the desired temperature without needing to overcool the bottom half of the
room. Another important observation in Figures 17 and 18 is the difference in temperature of the
ceiling above the racks. The ceiling has become warmer because less air from the cold aisles is
escaping the cold aisles and returning to the air conditioning units.
Real-world Model AC1 Uncontained 81.9° F 71.7° F Contained 88.7° F 80.9° F Change +6.8° F +9.2° F AC7 Uncontained 81.9° F 76.4° F Contained 77.1° F 74.7° F Change -4.8° F -1.7° F
Figures 17 (top) and 18 (bottom). Temperature cross-sections before and after containment, respectively
This warmer ceiling temperature led to a higher return air temperature to AC1, as shown
in Table 1. This can be seen when comparing the uncontained AC1 in Figure 19 with the
contained AC1 in Figure 20, and means that less cold air is recycling back into AC1. AC7 return
air actually became colder in the contained model. Figure 21 from before containment and
Figure 22 after containment show that when the cold aisle is contained, there is too much cold air
being pumped into the cold aisle, implying that an air conditioning unit should be turned down or
off completely if the aisle is contained. The excess of cold air in the aisle causes some of the cold
air to leak out of the 2-foot opening left in containment for emergency access to the aisle. As
shown in Figure 24 of the containment model, this cold air leakage travels along the floor and
directly into AC7. This colder temperature entering AC7 caused the air conditioner to greatly
reduce its chilled water intake during the experiment. As shown in the experimental data, this
aisles, as the cold aisles were colder than they had been previously. The slight drop in
temperature of the AC7 return air can be seen by comparing Figures 23 and 24.
Figures 19 and 20. Temperature contours near AC1 before and after containment
Figures 23 and 24. Opposite view of temperature contours near AC7 before and after containment
IV. Conclusions and Recommendations
When compared with experimental data, the computer model successfully showed the
trends and locations of temperature increases and decreases throughout the server room. This
computer model can now be used to model the room with only one air conditioner unit turned on,
to model other geometries of containment, or to model other server rooms in the data center. This
will give a general idea of what containment is needed and how the room can be best contained
to optimize energy savings and reduce installation costs.
Both the computer model and the experimental data show that cold aisle containment
should be implemented in the RCF server room to save cooling energy. Recommissioning is an
important step after implementing cold aisle containment in the data center. As shown in the
analysis of AC7, there will likely be overcooling once containment is implemented, so the room
must be analyzed and each air conditioner should be considered separately for a reduction in
chilled water usage, an increase in set point temperature, or a decrease in fan speed with the
addition of a Variable Frequency Drive unit. Some air conditioning units may even be turned off
with no negative impact on server inlet temperatures. This reanalysis step after implementation
Cold aisle containment, when implemented correctly, can lead to significant energy
savings, both in other national laboratories and in the RCF room at Brookhaven National
Laboratory. With a fluid flow model that is capable of predicting temperature distributions and
cooling loads associated with different configurations, Brookhaven National Laboratory would
benefit greatly from a containment project in their data center.
V. References 1
ENERGY STAR Program, US Environmental Protection Agency. “Report to Congress on
Server and Data Center Energy Efficiency Public Law 109-431.” Aug 2007.
2
US Energy Information Administration. <http://www.eia.gov/totalenergy/data/
annual/showtext.cfm?t=ptb0801>
3
ASHRAE. “Recommendations for Meeting Energy Efficiency Requirements for New Federal
Data Centers.” Aug 2011.
4 Lizardos, Brookhaven National Lab. “Computer Facilities Energy Efficiency Study: Building
Nos. 459 and 515.” Dec 2011.
5 Federal Energy Management Program, US Dept of Energy. “Retro-Commissioning Increases
Data Center Efficiency at Low- Cost: Success at Savannah River Site (SRS) at Low-Cost.” Dec
2010.
6 Federal Energy Management Program, US Dept of Energy. “Data Center Airflow Management
VI. Acknowledgements
