10 Steps to Holistic Data Center Design Michael Salvador
Technical Solutions Manager for Data Centers at Belden
Overview
Designing a data center can be a daunting and complicated process. There are many considerations and decisions that ultimately impact the cost to build, operate and scale the data center.
In the past, IT departments provided an estimate of the equipment and power needs required for their various systems. Many of these estimates were either inaccurate speculations based on current needs with no forethought given to the IT roadmap and business growth, or they were overinflated based on worse-case scenarios. Facilities then used these estimates to either build exactly what was requested or to once again overinflate the design in attempts to protect themselves. This outdated process resulted in large inefficient data centers that were costly to operate and virtually impossible to upgrade.
The rising cost to construct and operate a data center calls for a better approach to data center design. A holistic design approach provides significant benefit to data center managers, facilities, customers and the business as a whole.
A central aspect of successful holistic design involves carefully considering all of the variables with an eye on operational expenditure (OPEX) over the life of the data center, not just on the capital expenditure (CAPEX) to build the data center in the first place. Holistic design must also account for business growth and evolving technology, incorporating systems and innovative technologies that ensure an agile, scalable and highly robust data center that supports both current and future needs.
Successful holistic data center design requires following key steps that essentially take into account all business requirements, technology innovations, and energy and operational savings, while identifying and eliminating any ineffective decisions and operational waste.
Table of Contents
Overview . . . . 1
Step 1: Including All Players . . . . 1, 2 Step 2: Setting Ground Rules . . . . 2
Step 3: Determining Availability & Redundancy . . . . 3
Step 4: Gathering Requirements . . . . 3
Step 5: Balancing CAPEX and OPEX . . . . 4
Step 6: Selecting the Right Equipment . . . . 4
Step 7: Designing Equipment Areas . . . . 5
Step 8: Designing the Overall Space . . . . 6
Step 9: Constructing and Commissioning . . . . 6
Step 10: Ongoing Post-Construction Review . . . . 6
Conclusion . . . . 6
In most business environments, boards of directors or C-level executives formulate a strategic business plan with goals based on customer feedback, market conditions, opportunities and corporate vision. These goals are communicated to various business units, who in turn develop plans and requirements to meet the goals. Whatever the business type, virtually all of the requirements identified to meet the business plan involve some form of IT-related service or resource that must ultimately be supported by the data center. Consequently, the data center has become a catalyst for business and a vital part of the business ecosystem.
Once each business unit identifies their requirements, IT departments review those needs and identify the technologies that enable business units to successfully carry out their plans. IT must then determine the equipment, power and space required to support the technologies. Facilities then use this information to establish their own list of requirements needed to support IT—all while keeping operational costs in check. The end result enables the business units and the strategic business plan to either succeed or fail.
Step 1:
Step 1:
the early design phase identifies the actual requirements and future business direction needed to properly design the data center. Trusted advisors and outside design consultants and contractors should also be invited to the discussion. During this process, published meeting minutes should be distributed to all players, and recap meetings should be held at regular intervals. Primary and alternate representatives should be picked from every team, and the overall success of the project should be part of their responsibilities.
While teams can vary from business to business, following is typical list of the players involved in successful holistic data center design:
• Project Sponsor – Typically this should be a C- level person (i.e., CIO or CTO) or other business level executive.
• Project Manager – Depending on the overall scope, project management should likely be this player’s sole responsibility. The project manager is the overseer of meetings and the individual responsible for keeping the project and discussions on track. The project manager can be internal to the company or part of an outside architectural and engineering team.
• Business Unit Representatives – This could be one or multiple persons responsible for relating their business unit requirements and roadmap of future direction. • IT Technologists – These individuals can
include networking (i.e., LAN and WAN) teams, server teams for each operating system, storage area network (SAN) teams and data center support teams.
• Facilities Management – Any area of facilities operations and maintenance that is applicable should be included in the process, including security, real estate and property management.
• Purchasing/Procurement – These are individuals that negotiate all aspects of the data center purchases.
• Outsourced Services – Whether an architectural and engineering firm, individual consultant, general contractor or integrator (or a combination thereof), most organizations employ outside help at various times during the design process. Engaging these individuals depends on an organization’s resources, skill level and project scope. However, to remain cost effective, organizations need to keep mark-ups and change orders to an absolute minimum. For larger projects, the customer
may bid the project as a whole with an established percentage over cost to the general contractor, rather than dealing with separate subcontractor bids and potential surprises. This practice also puts onus on the general contractor to fully participate in the design process and understand all requirements, allowing only customer-initiated change orders to be billed and significantly lowering contingency budgets from as much as 15% to as low as 2%. • Trusted Advisors – These individuals are
typically expert representatives from vendors and manufacturers who can help with the design process surrounding their specific solutions (i.e., switches, cabling, power and cooling solutions). Including these individuals in the process is at the discretion of each discipline.
Step 2:
Step 2:
Setting Ground Rules
While setting ground rules may seem like a daunting, political and sometimes ugly task, it can help ensure respect for all players and professionalism that will go a long way in creating a superior product and successful outcome. Remember the following tips when moving ahead with plans for a holistic data center design:
Step 3:
Step 3:
Step 4:
Step 4:
Determining Availability & Redundancy
Gathering Requirements
Understanding and determining the data center availability and redundancy is a critical part of the holistic design process, and one that can significantly impact cost. Critical to this is determining which of the Uptime Institute’s tier ratings best suits the business needs.
• Tier 1: Non-redundant capacity components with an availability of ~99.671% and 28.8 hours of downtime per year
• Tier 2: Tier 1 + redundant capacity components with an availability of ~99.741% and 22 hours of downtime per year
• Tier 3: Tier 1 + Tier 2 + dual-powered equipment and multiple uplinks with an availability of ~99.982% and 1.6 hours of downtime per year.
• Tier 4: Tier 1 + Tier 2 + Tier 3 + all components fully fault-tolerant including uplinks, storage, chillers, HVAC systems and servers with everything dual-powered and an availability of ~99.995% with 0.4 hours of downtime per year
Many businesses believe they need 100% uptime, 365 days a year, 24 hours per day — essentially Tier 4. This may or may not be true for your business; but the initial cost to build a data center at each tier must be considered. Following are some conservative estimates of cost increases from tier to tier.
• Tier 1 to Tier 2: ~ 30% increase in project CAPEX • Tier 2 to Tier 3: ~ 50% increase in project CAPEX • Tier 3 to Tier 4: ~ 22% increase in project CAPEX
Big business with enormous data centers often create multiple data center “halls” that each meet a specific tier classification for ‘right size’ availability and redundancy. This tactic can save on construction costs and allows a business to achieve lower operational cost.
More detailed information on the data center tier ratings can be found by visiting the Uptime Institute’s website, www.uptimeinstitute.com.
When it is time to start gathering the requirements for the data center design, there are a few points to consider.
First of all, it is important to keep an eye on what the requirements will be once the data center actually goes live ; not on what the requirements are today. Each business unit should communicate their 3 to 5 year plan for growth — IT should understand each equipment manufacturer’s roadmap for this time period. In addition, anything that will ultimately be virtualized, outsourced to a service provider or moved to the public cloud should not be included in the requirements.
The next step is to create a redundancy and dependency grid for each application and system. This is an excellent tool for ascertaining the size, space, step functions for expansion, tier classification(s) and dependencies required for each application and system. The grid can get rather large and complicated; but it will provide cost saving to the company in the long run by preventing over designing the data center. The redundancy and dependency grid can also help determine the growth of the data center and allow for a more modular approach to building the data center.
Step 5:
Balancing CAPEX and OPEX
When selecting the latest IT equipment or method to power or cool the data center, it is important to take an approach that balances total cost to build (CAPEX) versus the total cost of operating the data center (OPEX). For example, it is not wise to select an energy-saving technology without considering the cost to operate and maintain the technology. Following are questions to answer when selecting a technology for the data center:
• What additional training or people will be required to operate the technology? • What are the costs related to maintenance and upkeep?
• What is the lifespan of the equipment and how does it affect my depreciation expense? Closely examining CAPEX and OPEX helps to make informed, intelligent choices and if necessary, provides the reasoning for spending more up front to ultimately achieve operational savings.
Step 6:
Step 6:
Selecting the Right Equipment
Determining the most efficient servers with the most efficient power supplies is an important step to holistic data center design. Examining the power consumption of the servers and finding the ‘sweet spot’ for number and speed of the CPUs, memory and correct sizing of the power supply is part of this process.
Selecting the correct platform to support a specific application is also important. How an application uses a server must be understood to effectively balance energy savings and performance. For example, some applications are input/output (I/O) intense and may need a server that has lower compute processing capabilities. On the other hand, some applications may require more compute processing and less I/O. Another factor is power supply size. Selecting a power supply that is too large will not be as
Selecting the Right Equipment
Step 7:
Step 7:
Designing Equipment Areas
Whether the data center site has been selected or not, there are several philosophies when it comes to deciding how the equipment areas of the data center should be built . As discussed previously, tier level support is part of this equation. Some key points to consider are cable plant design and limitations in relation to switch location, business unit pods or shared pods, cabinet design and who will be allowed to perform moves, adds and changes in the data center. These points need to be tailored to meet the needs of the business.
In summary, apply standardization, modularization and industrialization to as many components and pieces of the data center as possible. Understand the future applications and technology that will likely be deployed, and build in adaptability and flexibility to support them where possible. If you stay ahead of the curve and begin planning the pods or modules that will be built, it can allow for quick rollout of the IT equipment and the ability to double check calculations prior to construction. Also try to maximize the power load within a pod or equipment area before building out the next module. This allows the facility to operate at the maximum possible energy efficiency.
Understanding cable plant technology and capabilities is a key component when designing the cable plant layout. Following are questions to answer when determining the cable plant:
• Does the business require high bandwidth capabilities?
• What network topology design will be deployed (i.e., fabric switching or traditional switching)?
• What are the advantages and implications of using certain cable media types? • What are the distance limitations of the
data center design?
• What configuration will be used for access level switching (i.e., top of rack, end or row, middle of row or centralized)? • What will be required in the future? When designing the data center, avoid exclusive cabinets and pods of equipment. Having a standardized configuration for a cabinet or pod provides benefits such as creating a specialized customer part number with a manufacturer or distributor, and ease of budgeting and ordering for future expansion. The configuration can be reviewed and adjusted when technology changes occur. It is also smart to look for
ways to eliminate on-site construction labor, shipping material disposal and transportation costs by taking advantage of manufacturer or distributor services to pre-configure as many components as possible. Prior to any request for pricing, make sure to inform contractors of the use of these services or any other labor-saving technology, such as pre-terminated cables.
The easiest place to start is with a basic server cabinet. Using the data from the requirements gathering phase, determine what the cabinet will look like.
Considerations might include:
• Width, depth and number of RU spaces • Type of doors (i.e., single or split) • Cable entry (i.e., top or bottom)
• Ability to modify the cabinet or re-deploy as another type of cabinet
• Power distribution unit (PDU) size, intelligence, type and number of outlets • Cable management
• Equipment positioning • Cabinet color
These same considerations apply for networking cabinets, storage cabinets and almost any other cabinet used to house IT
equipment. Keeping the number of types of cabinets to a minimum can help simplify design and expansion of the data center. By deploying a modular approach using as many standardized cabinets as possible, the data center becomes industrialized and predictable with better port and space utilization. Modular design involves planning for the entire space, power and cooling requirements based on the company’s growth over the life of the facility. The design is broken into modules that can be added as the business requirements evolve. All shared infrastructure should be sized with the absolute final design in mind — it is important to understand that modules will need to be plugged into this infrastructure with little or no interruption of services as growth occurs. This allows for better cost efficiencies and ease of operation.
When designing equipment areas, DCIM is ideal for “what if” scenario planning. Once the data center is up and running, DCIM will also support asset inventory, cable plant connectivity, and power management and reporting. While it can seem rather early in the process to select a DCIM package, it can be a time consuming process to ensure that the package meets all requirements needed to serve the business. It also takes a significant amount of time to set up the DCIM templates and standard information to be tracked.
Step 9
Step 9:
Step 10:
Step 10:
Constructing and Commissioning
Ongoing Post-Construction Review
Regular team meetings should occur during construction and
commissioning. Invite the various leaders and supervisors of all involved trades to these meetings. It is critical that the design team attends these meetings. Detailed status and problem reporting should occur to keep everyone apprised of the project, coordinate overlapping trades and provide a team atmosphere for the project. During this phase, be careful of scope creep and changes to the design. The same holistic principles should take place to resolve any design problems and construction conflicts that arise.
After the data center has gone live, it is important to continually review which techniques, technologies and strategies are performing as expected. For example, determine how successfully or unsuccessfully
In reviewing, also consider what the team would do differently as the data center scales out or for the next data center build. Regular data center capacity discussion meetings are also important so that the next phases
planned. These same meetings should include all of the internal team members and select outside team members as necessary. Discussion should also include status regarding any new business initiatives that
Designing the Overall Space
With the requirements determined, IT can then present their requirements to the rest of the team and begin discussions to finalize power, cooling and space requirements. If a space has not been selected, this information can be useful for identifying potential sites or spaces. If the space has been selected, discussions begin to review the types of power distribution, cooling, economization and containment. During these discussions, full team participation is imperative. Adjustments and compromises will occur; but when the full design is completed, every player should understand what is being proposed and built. There should be a sign-off on the design from each team leader and other key participants in the project.
