Data Center Combined Heat and Power (CHP):
Benefits and Implementation
by
Sam Brewer
I. Common Causes of Failure
Major trends in the United States have reached an intersection between green energy innovation and the need to fuel a growing population. In the foreseeable future the industry will be forced to change how it powers data centers and other critical infrastructure. One needs to consider the following things:
1. The environmental impacts of the current power generation infrastructure are unsustainable and undesirable.
2. The central electric grid is a common point of failure for our economy and risk planning must include preparation for extended outages.
3. The natural gas resource base is large and growing. Distribution is simple, fault tolerant, highly parallel and much more environmentally friendly than coal or uranium.
4. Smaller scale, low maintenance, power generation technologies are available that can be fueled by natural gas.
5. Electricity is a feedstock or factor of production for every single modern unit of economic activity. An interruption in electricity flow immediately equates to losses and this must be accounted for in IT planning.
The Fukushima Nuclear Disaster provides a study of what can happen in a disaster when economic infrastructure is wholly dependent on immediate electrical power: The nuclear reactors in Fukushima were designed to produce electrical power for
consumption in Japan. Their failure to operate today can be traced to a single, not obvious, and highly concerning fact: Once a nuclear plant stops producing electrical power it requires continuous auxiliary electrical power to prevent the release of radioactive waste. In the case of three reactors at Fukushima, the March 11, 2011 tsunami washed away the fuel tanks for the backup generators. Thus electrical power could not be provided to the emergency support systems in the plant, highlighting the irony that a plant designed to produce electrical power also requires electrical power in order to function.
The same is true for the United States’ current electrical grid - once a plant disconnects from the grid, it needs another source of electrical power in order to start back up and reconnect. Our infrastructure can tolerate very minor unplanned events and still operate correctly. However, once a major event occurs, failures can cascade through the
system, amplify, and cause total system failure for an extended period.
It is important to ask ourselves in the context of the 21st century if we can ignore the possibility that man-made or natural disasters will disrupt the supply of electricity for an extended period of time. Even if your textbook tells you to ignore it, is it ethical or fiscally responsible to do so? Do your customers expect you to ignore these events during planning for risk?
In the past decade the US has been marred with brownouts and rolling blackouts. The most notable was the regional northeastern blackout of August 14, 2003, which was a cascading failure caused by a single fault within the system. The power grid in the
United States has been considered one of the most reliable in the world, but the grid is being stretched to its maximum, and additional cascade utility grid failure events are foreseeable. According to a report by the American Society of Civil Engineers (ASCE)1,
investment in transmission infrastructure has been stagnant or on the decline for the past 30 years. It would cost an estimated $1.5 to $2.0 trillion dollars of investment in new generation and transmission infrastructure to meet our needs by the year 2030 if we were to use a growth trajectory and technology mix similar to that of the past century.
What will the future state of our electricity grid be? The simple answer is, we do not know, but what we do know is that the grid is already stretched to its maximum, and everyday we are trying to milk one more kilowatt of electricity out of it. The ending result of this is a less reliable electric grid. In a worst case scenario, we may see rolling and extended blackouts on a regular basis in our highest stressed area.
With data centers dependency on clean reliable electricity to operate, scenarios of brownouts and rolling and prolonged blackouts are far from ideal. The cost of losing power to a data center can be in the order of millions of dollars per minute. Companies that are highly reliant on their data centers cannot afford to be exposed to outside forces in this regard.
II. The Legacy of Electric Supply and Distribution
A. Setting the Stage - The Past, Present, and Hopeful Future of Energy
Electricity is a refined form of energy that is not present on earth in a naturally usable form. It must be processed and refined from a primary energy source in order to be used. During the industrialization of the United States, the preferred method of delivering electricity was through centralized production and distribution via line infrastructure to the point of use. This system evolved because it was economic to transport fuel to a central point of power generation because fuel was typically heavy, bulky or special (nuclear), and energy conversion equipment (power generation) was not sufficiently technologically advanced or compact for users.
B. Coal Example
Legacy Electric Supply and Distribution - Coal Example
1
In this example using coal as a primary energy source to provide electric power, there are four distinct industries that have a major hand in providing the customer electric power. Each of these industries have capital requirements and environmental impacts and have now been made obsolete by two major milestones:
1. Easy Natural Gas Recovery and Distribution - Not only are vast quantities of natural gas able to be recovered from new deposits, but the techniques to extract the gas and the distribution infrastructure all the way to the end user are mature in the United States. The resource base is huge and the delivery capacity is massive.
2. Smaller Power Generation Equipment that is Easily Maintained - Various
technologies have been developed that make the use of natural gas to generate electricity at the point of use viable.
These two events have created an environment whereby drastic increases in cost effectiveness and reliability of electric power can now be implemented by eliminating two major steps in the supply and distribution chain.
C. Future Electric Supply and Distribution Chain
The number of major industries required to provide reliable electricity in this example have been cut in half, from four to two. The supply chain has been made highly secure against a large scale loss of electrical power through the shift of control to the end user.
D. Current Data Center Design:
The electricity demands of a data center are extremely high. According to the US EPA Combined Heat and Power Partnership2, data centers can have an energy density of
20-100 watts per square foot (sqft), meaning that a 10,000 sqft data center may need between 200 kw and 1000 kw of electricity. Current high performance servers in research labs have power densities well over 1000 watts per square foot. The utility infrastructure would likely need a generous amount of time to upgrade to make this kind of application possible. Couple this with the fact that we are quickly increasing our reliance on cloud computing, and we can be confident that data center and electricity demand is only going to grow.
Data centers typically rely on backup diesel generators during extended power outages. Backup diesel generators are reliable in the short term and great for quick starts upon the loss of power, however they are not designed to run for days at a time. The high emissions produced from these backup diesel generators can also pose a problem in some regions of the United States. There may be issues with getting the permits just to run them, let alone run them long enough to sustain your data center in a long term utility outage. Choosing to run a backup generator beyond permit levels may result in heavy fines for exceeding permits.
Data centers have two separate utility feeds, UPS backup for short duration sags, swells, and power losses, and backup diesel generators. This type of arrangement is beneficial for short term electrical outages, but is not designed to address the energy needs of the future. With the current state of our electric grid, our data centers are at an ever increasing risk of not being able to maintain our contracted “uptimes.” The resulting reduction in uptime reliability will likely lead to lost revenue and heavy fines or
assessments when data centers do not meet their letters of agreement (LOA).
III. CHP
A. SolutionThere is a solution to the obvious weaknesses in the electricity distribution infrastructure and future growth of our data centers. The solution lies in a proven integration technique called Combined Heat and Power (CHP), or Combined Cooling, Heat, and Power
(CCHP). Hereafter, we will simply refer to CHP when referring to both of these systems. Combined cooling, heat, and power is ideal for data centers due to their constant power and cooling demands.
CHP is utilized in many applications for distributed generation (DG), and there are many benefits associated with its use. CHP combines a power production component, a heating component for distributed heating if needed, and or a cooling component if needed. The generator can be either a reciprocating generator, gas turbine, micro-turbine, or fuel cell. The heating and cooling portions of the system are produced by the waste heat from the electricity generation that is transferred through a heat exchanger to produce hot water or steam that will be distributed through a system for process heat, or used in an absorption or adsorption chiller to produce chilled water. For data center application purposes, we will focus on the use of gas turbines for electricity and
absorption chillers to meet the cooling demands of data centers. Many of the benefits to a CHP system include:
● Reduced energy costs
● Reduced reliability on the electric grid ● Increased data center uptime
● Greater flexibility in data center build-out ● Greater flexibility in data center expansion ● Reduced carbon footprint
● Increased reliability for the grid
The following table is a block diagram representation of power and cooling infrastructure for both legacy and CHP powered data centers.
Legacy Data Center ● Grid Dependent ● 10.8 tons CO2 / kW
IT per year
Green Data Center CHP Retrofit
Option 1 (All Sizes) ● Grid Independent ● 4.0-5.3 tons CO2 /
Green Data Center New Build Option 1 (0.2-2.0 MW) ● Grid Independent ● 4.0-5.3 tons CO2 / kW IT per year
Green Data Center New Build
Option 2 (2.0-200 MW) ● Grid Independent ● 3.5-4.5 tons CO2 /
kW IT per year
As one considers the design of a new data center or existing data center, it is important to ask the following questions. An affirmative answer to any of these questions should be an indication to consider an onsite, distributed CHP system.
● Is it important to reduce operational energy costs?
● Is it important to maintain operation for long duration utility blackout events? ● Is it a management or institutional focus to reduce carbon footprint?
● Will it be costly or infeasible to bring more electric infrastructure to the data center?
● Do I believe that the electric utility will be less stable in the future?
According to the Uptime Institute’s Data Center Site Infrastructure Tier Standard: Topology3, a tier IV data center should have no incidents of more than any single four
hour outage in any five year period, or 0.8 hours, or forty-eight (48) minutes, in any one year. This is based on the utilization of two separate utility feeds, UPS backup, and standby emergency diesel generators. In a report from the US EPA Combined Heat and
Power Partnership4, with a CHP system and one utility feed, system outage time drops
to forty-three (43) minutes per year, and with CHP and two utility feeds, outage time drops to seven seconds per year.
With business’ expectations of high data center reliability, it is very important to understand the reality of current data center reliability models, and to explore
technologies like CHP that will not only increase the reliability of your data center, but also reduce long term operating costs and environmental impacts. The most important thing when deciding to explore the option of implementing a CHP system in a facility is to consult with experts in this field. There are several companies available to step you through the initial process of feasibility studies, design, construction estimates, and installation. There are also many governmental agencies and NGO’s that are working very hard to increase awareness and the implementation of CHP in the United States. Below is a list of a few resources for newcomers to CHP:
● EPA Combined Heat and Power Partnership, http://www.epa.gov/chp/
● US Department of Energy’s Clean Energy Application Centers
● US Department of Energy, Energy Efficiency and Renewable Energy,
http://www1.eere.energy.gov/femp/technologies/derchp_chpbasics.html
● Texas Combined Heat and Power Initiative, http://www.texaschpi.org/
● NYSERDA, http://www.nyserda.ny.gov/en/Page-Sections/Research-and-Development/Combined-Heat-and-Power.aspx
● United States Clean Heat and Power Association, http://www.uschpa.org/
During the initial review of a data center’s needs and what CHP can provide, here are some things to keep in mind, that may be helpful when partnering with a company for designing and installing your CHP system.
● Is this for a new data center or a retrofit? ● How large is the data center?
● Is this a stand alone data center or part of a larger multi use facility? ● What are the power, cooling, and heating needs of the facility? ● What are the reliability needs?
● What is the potential for expansion?
● What resources are available as fuel sources? ● How important is public image to potential clients?
All of this information is very important to know as you move ahead in partnering with a group that will work with you to design and install your CHP system.
B. Energy Cost Comparison - for the accountant
Energy Type Coal -
Appalachian Natural Gas Primary Electricity Refined 4http://www.epa.gov/chp/documents/datactr_whitepaper.pdf
Primary Source Source Product Commodity Cost ($/MMBTU) $2.405 $2.126 $11.00-$20.007
Transportation and Distribution Cost ($/MMBTU)
$0.808 $1.00-$2.00 $6.00-$30.00
Broker or Sales Cost ($/MMBTU)
$0.20-$0.60 $0.20-$0.50 $0.60-$2.00 Total Energy Cost
($/MMBTU)
$3.40-$3.80 $3.32-$4.62 $17.60-$52.00
C. Energy Systems Technology - for the engineer
Let us assume that the decision has been made that a new data center shall generate its own electricity from a primary fuel source - at this point, we have to perform a qualitative and quantitative technical factor analysis of our competing choices. Let us assume that we entertain building different types of power plants for a fictional data center of the following type:
Critical IT and Facility Load - 1000 kW
Cooling Requirement - ~300 tons continuous Uptime Institute Rating - Tier II
Case 1 2 3 4 5
Prime Mover Steam Generator / Thermal Boiler
MicroTurbine Reciprocating Engine
Fuel Cell Grid Data Center (reference)
Module Uptime / Availability
~90% ~99.5% ~92%-97% ~99.5% N/A
Module Size 1000 kW 200 kW 1000 kW 200 kW N/A
Time to Start 2 hours 1 minute 10 seconds 12 hours+ N/A Fuel Coal Natural Gas Natural Gas Natural Gas Electric
5 Data taken from CME Group public website on 4/1/2012 for May 2012 delivery, min contract 12000
BTU/lb
6Data taken from CME Group public website on 4/1/2012 for May 2012 delivery to Henry Hub
7Data taken from CME Group public website on 4/1/2012 for April - July 2012 delivery to NYISO Zone A 8Data derived from US EIA public website - http://www.eia.gov/cneaf/coal/page/trans/ratesntrends.html
Cooling Steam Absorber Exhaust Absorber Exhaust Absorber Hot Water Absorber from FC Stack Electric Electrical Conversion Efficiency (HHV) ~22% ~30% ~33% ~40%(+) N/A - Grid DOE estimates 28.8% Annual Fuel Consumption 136,300 MMBTU 100,000 MMBTU 90,900 MMBTU 75,000 MMBTU 11,400,000 kWh Annual O&M Staff Hours 24,000 200 12,000 8,000 0
O&M Hourly Cost (complexity level) $50 $60 $60 $80 N/A Annual Energy Cost (average from table 1) $490,000 $390,000 $360,000 $300,000 $1,350,000 Annual O&M Staff Costs $1,200,000 $12,000 $720,000 $640,000 $0
Annual Parts and Overhaul Reserve $125,000 $150,000 $125,000 $400,000 $50,000 Annual Cost to Operate $1,815,000 $552,000 $1,205,000 $1,040,000 $1,400,000 System Initial Cost $2,500,000 $3,000,000 $2,500,000 $8,000,000 $1,500,000
* All costs and system performance data have been taken from publicly available information from representative manufacturers literature when possible. Assumptions and estimates have been made in certain categories where information is not readily available.
D. The Bottom Line - for the CFO
Case 1 2 3 4 5
Prime Mover Steam Generator / Thermal Boiler
MicroTurbine Reciprocating Engine
Fuel Cell Grid Data Center (reference) Investment Required (above grid DC reference) $1,000,000 $1,500,000 $1,000,000 $6,500,000
-Annual Savings negative $848,000 $195,000 $360,000 -NPV (5%, 20
years)
-E. The Environmental Impact
Case 1 2 3 4 5
Prime Mover Steam Generator / Thermal Boiler
MicroTurbine Reciprocating Engine
Fuel Cell
Grid Data Center (reference) Annual CO2 Emissions (metric tons) 13,200 5,300 4,800 4,000 10,800 Annual NOx Emissions (metric tons) 16.7 1.6 12.4 0.3 12.5
If the United States were to move to some sort of carbon tax, a logical tax rate might be on the order of $50/metric ton initially. If this comes to pass, a savings of 5000 metric tons of carbon per year results in additional savings of $250,000 per year.
IV. Implementation and CHP Retrofit
Assuming that after all technical research has been completed and the user had made the decision to move ahead with a CHP system retrofit for a data center, the following steps can be taken to move toward project completion:
A. Engage a specialist in data center infrastructure integration who can design, build, and maintain your system
During this step, you should perform a market analysis and seek out references for possible system integrators that you want to spearhead the design and implementation of the retrofit. After you have determined what your core technology will include for the data center (fuel cell, turbine, etc.), it would be wise to approach a firm with experience in that core technology.
B. Engage potential partners and interview them
Some questions you might want to ask:
a) How many installations of this core technology have you performed or designed? b) Do you offer operation and maintenance contracts?
c) May I visit some of your customers?
d) Would you mind performing a walk-through of my facility and giving me your initial impression?
C. Create a workable design and obtain an integration contract
After making a decision on a partner; be prepared to spend about 1-2% of the projected project amount to secure a design and a maximum price guarantee for the project. The report deliverable should address the following:
● Increase in reliability over legacy systems ● Energy intensity impact
● On site electricity generation ● Environmental impact
● Operational cost impact
● Total project implementation cost
During this phase, specific issues will have to be addressed depending on your current infrastructure layout.
Location of Data Center - whether it be the 30th floor of an urban high rise or a
suburban commercial park, knowing the existing location will drive design choices early in the process. New options, even for dense urban areas are now attractive. One key option for dense locations such as high rises is the hybrid UPS MicroTurbine. This practical innovation can be located in boiler rooms in urban buildings or a mechanical building in an office park, and provide a secure electric feed to the data center.
Key Innovation Example
Hybrid UPS Turbine technology, developed by Capstone Turbine Corporation, allows the functions of a merged UPS and turbine power to be integrated into a single package that can be integrated into a CHP system. One new possibility is that this allows data centers in high rises to work with building owners to locate their secure UPS equipment in a boiler or mechanical room - freeing up valuable real estate for additional IT equipment.
Cooling infrastructure - most CHP systems will produce chilled water as a cooling medium and most data centers can use chilled water via some means; for example:
● DX CRAC Units - it is difficult to get a CHP system to actively cool a refrigerant, thus, if you have a data center floor or rack cooling system based on refrigerant, an interface or design approach must be adapted to this. Some of your options include adding coils to existing CRAC units to supply both refrigerant and chilled water, replacing all CRAC units with chilled water variants, or performing an optimization study to shut down redundant refrigerant CRAC units and then replacing only those CRACs that must be kept in constant operation with chilled water.
● In rack cooling - most data centers can eliminate CRAC units entirely or
substantially by switching to in-rack cooling gear via a sidecar or rear-door heat exchanger. This approach substantially increases white space for additional IT equipment. These systems can be used with refrigerant or chilled water - the refrigerant type units usually have a central refrigerant to chilled water interface heat exchanger that allows easy compatibility with a CHP system.
Electric infrastructure topology - there are very few “standard” electrical systems that one encounters when visiting data centers, however, some general trends are typically evident and can be integrated with-in a non-invasive manner with no down time. In any event, an expert can advise you on how to “harden” your electric infrastructure against single points of failure and long duration utility blackouts.
D. Execute a contract to integrate the system with your partner
A properly designed CHP system for an existing facility with a critical electric IT load of between 100kW and 2000kW should take between 9 and 18 months to fully integrate into the facility and should have a design life of up to 30 years.
V. Summary
Beyond reducing costs and saving money, CHP in your data center is a practical solution that comes with many added benefits mentioned here. The review of whether CHP is feasible for your facility through the evaluation of your location, electricity cost, reliability requirements, and environmental impacts is only a portion of implementing CHP into your data center. It is important to take the time and effort necessary to learn about CHP and conduct the evaluations necessary in determining its benefits so that you can insure your facility is the most robust and sustainable to operate.
About the Authors
Justin Grau Google Inc
Justin Grau specializes in electrical
distribution and water chemistry at Google’s data center in Council Bluffs, Iowa.
Justin has a background in nuclear power production, with over eight years in the U.S. Navy as a Reactor Operator and Reactor Operator Instructor. He worked for over two years at a commercial nuclear power plant and has a BS in Nuclear Technologies. He resides in Omaha with his wife and son, and is currently working toward his MBA.
Sam Brewer GEM Energy
Sam Brewer works in distributed power systems for GEM Energy, an advanced energy technology integration firm that specializes in mission critical
infrastructure, leveraging
environmentally preferred technology. Sam is a former United States Air Force Pilot and has a BS in Nuclear
Engineering from Rensselaer Polytechnic Institute. He is a DOE certified Data Center Energy
Practitioner He resides in Saratoga Springs, NY with his family.
© 2012 Justin Grau and Sam Brewer. All rights reserved. INTELLECTUAL PROPERTY DISCLAIMER
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY AND IS PROVIDED “AS IS” WITH NO WARRANTIES WHATSOEVER INCLUDING ANY WARRANTY OF
MERCHANTABILITY, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION, OR SAMPLE.
NO LICENSE, EXPRESS OR IMPLIED, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED OR INTENDED HEREBY.
