R&M Data Center
Handbook
Preface
The modern data center, along with the IT infrastructure, is the nerve center of an enterprise. Of key importance is not the data center's size but its capability to provide high availability and peak performance, 7 days a week, 365 days a year.
High-quality products alone are not enough to ensure continuous, reliable operation in the data center, nor is it enough just to replace existing components with high-performance products. Of much greater importance is an integrated and forward-looking planning process that is based on detailed analyses to determine specific requirements and provide efficient solutions. Planning is the relevant factor in the construction of new systems as well as for expansions or relocations. In all cases, the data cabling system is the bedrock of all communication channels and therefore merits special attention.
Data centers can have different basic structures, which in some cases are not standard-compliant, and it is essential to apply case-specific relevant parameters to create an efficient and economically viable nerve center. Today more than ever, an efficient and reliable data center is a distinct competitive advantage. In addition to company-specific requirements, legal security standards and their impact need to be taken into account; these include directives like the SOX agreement, which provide the basic legal premises for data center planning. Furthermore, various standardization bodies are addressing cabling standards in data centers.
This handbook is designed to provide you with important knowledge and to point out relationships between the multitude of challenges in the data center.
Table of Contents
1. Introduction to Data Centers……….. 5
1.1. Technical Terms………... 5 1.2. General Information………. 5 1.3. Standards………... 8 1.4. Availability………. 8 1.5. Market Segmentation………. 9 1.6. Cabling Systems……… 10
1.7. Intelligent Cable Management Solutions……… 12
1.8. Network Hardware and Virtualization……….. 12
1.9. Data Center Energy Consumption……… 13
1.10. Energy Efficiency – Initiatives and Organizations……… 14
1.11. Green IT……… 15
1.12. Security Aspects……… 16
2. Planning and Designing Data Centers……….. 20
2.1. Data Center Types………. 20
2.1.1. Business Models and Services……….. 20
2.1.2. Typology Overview……….. 21
2.1.3. Number and Size of Data Centers……… 22
2.2. Classes (Downtime and Redundancy)………. 23
2.2.1. Tiers I – IV………. 23
2.2.2. Classification Impact on Communications Cabling………. 24
2.3. Governance, Risk Management and Compliance………. 26
2.3.1. IT Governance……….. 27
2.3.2. IT Risk Management……… 28
2.3.3. IT Compliance……….. 28
2.3.4. Standards and Regulations……… 28
2.3.5. Certifications and Audits………. 33
2.3.6. Potential Risks……….. 34
2.4. Customer Perspective……….. 36
2.4.1. Data Center Operators / Decision-Makers……….. 36
2.4.2. Motivation of the Customer……… 39
2.4.3. Expectations of Customers……… 40
2.4.4. In-House or Outsourced Data Center……….. 43
2.5. Aspects of the Planning of a Data Center………... 44
2.5.1. External Planning Support……….. 44
2.5.2. Further Considerations for Planning………. 45
3. Data Center Overview………. 46
3.1. Standards for Data Centers………. 46
3.1.1. Overview of Relevant Standards……….. 46
3.1.2. ISO/IEC 24764………. 47
3.1.3. EN 50173-5……….. 49
3.1.4. TIA-942………. 50
3.2. Data Center Layout……… 51
3.2.1. Standard Requirements………. 51
3.2.2. Room Concepts………... 52
3.3. Network Hierarchy………. 54
3.3.1. Three Tier Network………. 54
3.3.2. Access Layer……… 55
3.3.3. Aggregation / Distribution Layer……… 55
3.3.4. Core Layer / Backbone……….. 56
3.3.5. Advantages of Hierarchical Networks……….. 57
3.4. Cabling Architecture in the Data Center………. 57
3.4.1. Top of Rack (ToR)………... 58
3.4.2. End of Row (EoR) / Dual End of Row……….. 60
3.4.3. Middle of Row (MoR)……….. 61
3.4.4. Two Row Switching………. 62
3.4.5. Other Variants……….. 62
3.5. Data Center Infrastructure……….. 66
3.5.1. Power Supply, Shielding and Grounding………. 66
3.5.2. Cooling, Hot and Cold Aisles………. 69
3.5.3. Hollow / Double Floors and Hollow Ceilings……… 72
3.5.4. Cable Runs and Routing………. 73
3.5.5. Basic Protection and Security……… 75
3.6. Active Components / Network Hardware……… 78
3.6.1. Introduction to Active Components………... 79
3.6.2. IT Infrastructure Basics (Server and Storage Systems)……… 79
3.6.3. Network Infrastructure Basics (NICs, Switches, Routers & Firewalls)……… 84
3.6.4. Connection Technologies / Interfaces (RJ45, SC, LC, MPO, GBICs/SFPs)………. 87
3.6.5. Energy Requirements of Copper and Fiber Optic Interfaces………... 89
3.6.6. Trends……… 90
3.7. Virtualization………... 90
3.7.1. Implementing Server / Storage / Client Virtualization……… 91
3.7.2. Converged Networks, Effect on Cabling……….. 93
3.7.3. Trends……… 94
3.8. Transmission Protocols……….. 95
3.8.1. Implementation (OSI & TCP/IP, Protocols)………. 95
3.8.2. Ethernet IEEE 802.3……… 98
3.8.3. Fibre Channel (FC)……… 102
3.8.4. Fibre Channel over Ethernet (FCoE)……….. 104
3.8.5. iSCSI & InfiniBand………. 105
3.8.6. Protocols for Redundant Paths……… 106
3.8.7. Data Center Bridging………. 108
3.9. Transmission Media……… 109
3.9.1. Coax and Twinax Cables ………. 110
3.9.2. Twisted Copper Cables (Twisted Pair)……….. 110
3.9.3. Plug Connectors for Twisted Copper Cables……… 113
3.9.4. Glass Fiber Cables (Fiber optic)………. 116
3.9.5. Multimode, OM3/4 ……… 116
3.9.6 Single mode, OS1/2 ...117
3.9.7. Plug Connectors for Glass Fiber Cables………... 118
3.10. Implementations and Analysis……… 124
3.10.1. Connection Technology for 40/100 gigabit Ethernet (MPO/MTP®)……….. 126
3.10.2. Migration Path to 40/100 gigabit Ethernet……… 129
3.10.3. Power over Ethernet (PoE/PoEplus)………. 135
3.10.4. Short Links ……… 139
3.10.5. Transmission Capacities of Class EA and FA Cabling Systems……… 143
3.10.6. EMC Behavior in Shielded and Unshielded Cabling Systems……….. 148
4. Appendix………..156 References………. 156 Standards………... 156
All information provided in this handbook has been presented and verified to the best of our knowledge and in good faith. Neither the authors nor the publisher shall have any liability with respect to any damage caused directly or indirectly by the information contained in this book.
All rights reserved. Dissemination and reproduction of this publication, texts or images, or extracts thereof, for any purpose and in any form whatsoever without the express written approval of the publisher is in breach of copyright and liable to prosecution. This includes copying, translating, use for teaching purposes or in electronic media.
This book contains registered trademarks, registered trade names, and product names, and even if not specifically indicated as such, the corresponding regulations on protection apply.
1.
Introduction to Data Centers
According to Gartner, cloud computing, social media and context-aware computing will be shaping the economy and IT in the coming years. Cloud computing is already changing the IT industry today, and there will be more and more services and products entering the market. Outsourcing projects and subtasks and procuring IT resources and services by means of the cloud (World Wide Web) is becoming a standard process. The attraction of services provided by the cloud is also the result of their considerable cost and efficiency advantages as compared to traditional IT infrastructures. Server and storage virtualization help to optimize and concentrate IT infrastructures and save on resources.
Consequently, high demands are placed on data center performance and availability, and server operations are required around the clock. A breakdown in a company's computer room, however, can prove fatal as well, and it always results in enormous costs and dissatisfied users and customers.
1.1. Technical Terms
There are no standard, consistently used terms in the data center world, and the companies' creativity is endless. We recommend always specifying what the supplier implies or what the customer expects.
The terms
data center – datacenter – data processing center – computer room – server room – server cabinet – collocation center – IT room – and more
stand for the physical structure that is designed to house and operate servers. Interpretations may vary. The term campus (or MAN) is also used for data centers on a larger area.
We define "data center" as follows:
A data center is a building or premise that houses the central data processing equipment (i.e. servers and infrastructure required for operation) of one or several companies or organizations. The data center must consist of at least one separate room featuring independent power supply and climate control.
Consequently, there is a distinction between a data center, and specific server cabinets or specific servers.
1.2. General Information
The data center is the core of a company; it creates the technical and infrastructural conditions required for virtually all business processes in a company. It secures the installed servers and components in use, protects them from external dangers and provides the infrastructure required for continued reliable operation. The physical structure provides protection against unauthorized access, fire and natural disasters. The necessary power supply and climate control ensure reliability.
Accurate, economical planning plus the sustainable operation of a modern data center are steps companies need to take to meet the requirements of availability. The enormous infrastructural requirements involved lead many companies to opt for an external supplier and outsource their data center needs.
Outsourcing companies (such as IBM, HP, CSC, etc.) offer their customers the use of data center infrastructures in which they manage the hardware as well as the system software. This offers the advantage of improved control of IT costs, combined with the highest security standards and the latest technology. In outsourcing, the systems and applications of a company are first migrated to the new data center under a migration project, and are then run in the data center of the outsourcing company.
Along with data security, high availability and operational stability are top priorities.
The data center must have the ability to run all the necessary applications, required server types and server classes. Due to the increasing number of real-time applications, the number of server variants is soaring as well.
For the general performance of a server service it doesn’t matter if the service, i.e. the software, that is offered, • runs with other services on the same server;
• is processed by a separate server computer or even;
This last situation is especially common with public WWW servers, because the Internet sites that are frequented, such as search engines, large web shops or auction houses, generate a large quantity of data for processing. In this case, so-called load-balancing systems come into play. They are programmed to distribute incoming queries to several physical servers.
The following basic types of server services exist:
File Server
A file server is a computer attached to a network that provides shared access to computer files for workstations that are connected to the computer network. This means that users can send each other files and share them via a central exchange server. The special feature offered by file servers is that using them is as transparent as using a local file system on workstation computers. Management of access rights is important, for not all users should have access to all the files.
Print Server
A print server or printer server allows several users or client computers shared access to a printer or printers. The biggest challenge is to automatically provide the client computer with the appropriate printer driver for their respective operating system so that they can use the printer without having to install the driver software locally.
Mail Server
A mail server, or message transfer agent (MTA), does not necessarily have to be installed on an Internet provider's server, but can operate on the local network of a company. First, it is often an advantage when employees can communicate with each other by e-mail, and second, it is sometimes necessary to operate an internal e-mail server just because Internet access is heavily restricted for security reasons, or communication between workstations and external mail servers is not allowed in the first place.
Web Server
The primary function of a web server (HTTP server) is to deliver web pages from a network to clients on request. Usually, this network is the public Internet (World Wide Web). This form of information transfer is becoming more and more popular in local company networks (intranet). The client uses a browser to display, request and explore web pages, and also to mouse-click to follow the hyperlinks contained within those pages, which are links to other websites, files, etc. on the same or a different server.
Directory Services (DNS server)
Directory services are becoming increasingly important in IT. A directory in this context is not a file system but an information structure, a standardized catalog of users, computers, peripherals and rights in a network. Information can be accessed network-wide via an entry in this directory, which means that directory services are a practical basis for numerous services, making work easier for administrators and life easier for users in large network environments. Here are some examples:
• Automatic software distribution and installation • Roaming user profiles
• Single sign-on services
• Rights management based on computer, clients and properties Application Server
An application server allows users on a network to share software applications on the server.
• This can be a server in the LAN, which runs applications that are frequently used by clients (instead of pure file server services).
• Or it runs the application/business logic of a program (e.g. SAP R/3®) in a three-tier architecture (see section 3.3.1.), interacts with database servers and manages multiple client access.
• Or it is a web application server (WAS) for web applications, dynamically generating HTML pages (or providing web services) for an Internet or intranet connection.
Web services are automatic communication services over the Internet (or intranet) which can be found, interpreted and used through established and standardized procedures.
Simple Application Services
In an application server’s simplest form, it stores the database of an application. It is loaded onto the client's random access memory (RAM) and launched there. This process differs only slightly from a file server. The application just needs to know that any necessary additional components or configuration data are not located on the computer it launches from, but on the server from where it was loaded.
This configuration makes sense when a high number of single-user application programs exist. It offers two advantages: first, it just requires one program installation on the server instead of several installations on each workstation, and second, costs are lower – usually only one software license per computer is required, for the computer that runs the program. Now, when an application is used by several computers, but not at the same time, the software can be installed on the server. It can be used by all computers and the licensing fee only needs to be paid once. And updating software is easier too.
Client–Server Applications
In the more complex forms of application servers, parts of a program – in some cases the entire program – are launched directly on the server. In the case of large databases, for example, the actual data and the basic data management software are usually stored on the server. The components on client computers are called front ends, i.e. software components that provide an interface to the actual database.
Network
High-performance, fail-safe networks are required in order to enable client-server as well as server-server communication. Investing in server technology is not very profitable if the network is not "qualitatively suited". Network components and transmission protocols are discussed in sections 3.6. and 3.8.
Network Structures and Availability One way to increase the availability of a network is to design it for redundancy, meaning that several connections are provided to connect servers, and possibly storage systems, as well as network components to the network. This ensures that, in case of system, interface or connection failure, data traffic can still be maintained. This is of critical importance especially for the backbone, which is literally the backbone of the infrastructure.
This is why data center classifications were defined to specify different levels of network reliability. These are discussed in greater detail in section 2.2.
Access for Remote Users
In addition to the local network, many enterprises have remote users that must be connected. There are two categories of remote users: those working at a remote regional company office and those with mobile access or working from home. Two company sites are usually connected by means of a virtual private network (VPN), also called a site-to-site scenario.
A VPN connection can also be established between individual users and the company's head office, which would constitute a site-to-end scenario. An SSL-encrypted connection is sufficient for providing access for one single user if this is necessary.
VPN Connections
In VPN connections, an automatic VPN tunnel is established and network data is transported between the company network and the client using encryption. In general, VPN connections support any kind of network traffic, i.e. terminal service sessions, e-mails, print data, file transfers and other data exchange. To provide the user with fast access to the network resources, the transmission of the respective data packets must be prioritized in comparison with the rest of the network traffic. These requirements can be met with high-end routers. With regards to operational reliability of Wide Area Network (WAN) links, these connections should also be laid out redundantly.
Storage Architectures
A storage server conventionally holds a sufficient number of local hard disk drives (DAS: Direct Attached Storage).
Even with high numbers of hard disks there are rarely any technical problems since modern servers can easily access several dozens hard disks thanks to RAID controllers (Redundant Array of Independent Disks).
There is a trend emerging in data centers towards storage consolidation. This means that a storage network (SAN: Storage Area Network) is installed which includes one or several storage systems for the servers to store data.
A storage-consolidated environment offers the following advantages: • Better manageability (particularly for capacity extension etc.) • Higher availability (requiring additional measures)
• Extended functions such as snapshot-ting, cloning, mirroring, etc SAN, NAS, iSCSI
Different technologies exist for accessing central storage resources:
• SAN: Even though the term SAN, Storage Area Network, is basically used for all kinds of storage networks independent of their technology, it is generally used today as a synonym for Fibre Channel SAN. Fibre Channel is currently the most widely used technology for storage networking. It is discussed in greater detail in section 3.8.3.
• NAS: Network Attached Storage, as the name suggests, is a storage system that can be connected directly to an existing LAN. NAS systems are primarily file servers. A point to mention is that there is no block-level access (exclusively assigned storage area) to a file share, so in general database files cannot be stored.
• iSCSI: iSCSI, Internet Small Computer System Interface, allows block-level access to an accessible storage resource on the network as if it was locally connected through SCSI. Professional NAS systems support iSCSI access (e.g. through a Network Appliance), so that the storage areas of these systems can also be used by database servers. On the server side, iSCSI can be used with conventional network cards. It is a cost-effective technology that is spreading fast.
However, the dominant technology in data center environments today is Fibre Channel, mainly because high-end storage systems have only been available with Fibre Channel connectivity so far.
1.3. Standards
The U.S. TIA-942 standard, the international ISO/IEC 24764 standard and the European EN 50173-5 standard define the basic infrastructure in data centers. They cover cabling systems, 19" technologies, power supply, climate control, grounding, etc., and also classify data centers into availability tiers.
A detailed comparison of the different standards is provided in section 3.1.
1.4. Availability
Just a few years ago, there were still many enterprises which could have survived a failure of several hours of their IT infrastructure. Today, the number of companies who absolutely rely on continuous availability of their IT is very high and growing.
A study carried out by the Meta Group showed that the damage caused by the breakdown of a company's key IT systems for more than 10 days can put the company out of business in the following three to five years – with a likelihood of 50 percent.
There are a number of classifications with regards to reliability and availability of data centers, for example, those issued by the Uptime Institute, BSI and BITKOM (overview in 2.1.2).
The availability rating of a company's communications infrastructure is of crucial importance in developing, up-grading and analyzing an IT concept in today’s world. Consequently, the basic question is: "What is the maximum acceptable IT outage time for a company?"
These growing requirements for availability not only affect the IT infrastructure itself but also place high demands on the continuous securing of environmental conditions and supply. Redundancy in cooling and power supplies, multiple paths and interruption-free system maintenance have become standard for highly available IT infrastructures.
Before planning and coordinating technical components so as to achieve the targeted availability, there are additional points regarding risk assessment and site selection that must be considered.
These points include all possible site risks, be they of a geographical (air traffic, floods, etc.) or political (wars, conflicts, terrorist attacks, etc.) nature or determined by the surroundings (fire-hazards due to gas stations, chemical storages, etc.), which could have an effect on the likelihood of a potential failure. Furthermore, potentially criminal behavior of company employees and from outside the company should also be taken into consideration.
Achieving high availability on the one hand presupposes an analysis of technical options and solutions, but it also requires the operator to plan and implement a comprehensive organizational structure, including the employment of trained service staff and the provision of spare parts and maintenance contracts. Precise instructions as to how to behave in case of failure or emergency are also part of this list.
The term "availability" denotes the probability that a system is functioning as planned at a given point in time.
1.5. Market Segmentation
We can often read about Google’s gigantic data centers. However, Google is no longer the only company running large server farms. Many companies like to keep their number of servers a secret, giving rise to speculation. Most of these are smaller companies but there are also some extremely big ones among them.
The following companies can assumed to be running more than 50,000 servers.
• Google: There has been much speculation from the very beginning about the number of servers Google runs. According to estimates, the number presently amounts to over 500,000.
• Microsoft: The last time Microsoft disclosed their number of servers in operation was in the second quarter of 2008, and that number was 218,000 servers. In the meantime, it can be assumed that as a result of the new data center in Chicago, that number has grown to over 300,000. Provided the Chicago data center stays in operation, the number of Microsoft servers will continue to increase rapidly.
• Amazon: The company owns one of the biggest online shops in the world. Amazon reveals only little about their data center operations. However, one known fact is that they spent more than 86 million dollars on servers from Rackable.
• eBay: 160 million people are active on the platform of this large online auction house, plus another 443 million Skype users. This requires a gigantic data center. In total, more than 8.5 petabytes of eBay data are stored. It is difficult to estimate the number of servers, but it is safe to say that eBay is a member of the 50,000-server club.
Tier level Introduced Requirements
Tier I In the 1960s
Single path for power and cooling distribution, no redundant components, 99.671% availability
Tier II In the 1970s
Single path for power and cooling distribution, includes redundant components, 99.749% availability
Tier III End of the 1980s
Multiple power and cooling distribution paths but with only one
active path, concurrently maintainable, 99.982% availability
Tier IV 1994
Multiple active power and cooling distribution paths, includes redundant components,
fault-tolerant, 99.995% availability
• Yahoo!: The Yahoo! data center is significantly smaller than that of Google or Microsoft. Yet, it can still be assumed that Yahoo! runs more than 50,000 servers to support their free hosting operation, the paid hosting services and the Yahoo! store.
• HP/EDS: This company operates a huge data center. Company documentation reveals that EDS runs more than 380,000 servers in 180 data centers.
• IBM: The area covered by the IBM data center measures 8 million square meters. This tells us that a very high number of servers is at work for the company and their customers.
• Facebook: The only information available is that Facebook has one data center with more than 10,000 servers. Since Facebook has over 200 million users storing over 40 billion of pictures, it can safely be assumed that the number of servers is higher than 50,000. Facebook has plans to use a data center architecture that is similar to Google.
Google, eBay and Microsoft are known users of container data centers, which were a trend topic in 2010, according to Gartner. Other terms used to describe container-type data centers are:
modular container-sized data center – scalable modular data center (SMDC) – modular data center (MDC) – cube – black box – portable modular data center (PMDC) – ultra-modular data center – data center in-a-box – plug-and-play data center – performance-optimized data center – and more
Data center segmentation by center size in Germany, according to the German Federal Environment Agency (Bundesamt für Umwelt):
Data center type Server
cabinet Server room Small center Medium center Large center Total
Total number of servers
installed 160,000 340,000 260,000 220,000 300,000 1,280,000
Percentage of server
clusters 12.5 % 26.6 % 20.3 % 17.2 % 23.4 % 100 %
Number of data centers 33,000 18,000 1,750 370 50 53,170
Percentage of all data
centers 62.0 % 33.9 % 3.3 % 0.7 % 0.1 % 100 %
Data Center Inventory in Germany, as of November 2010 See section 2.1.2. for additional data center types.
1.6. Cabling Systems
The communications cabling system is essential to the availability of IT applications in data centers. Without a high-performance cabling system, servers, switches, routers, storage devices and other equipment cannot communicate with each other and exchange, process and store data.
However, cabling systems have often grown historically and they are not fully capable of meeting today's requirements.
This is because today's requirements for data centers are high: • High channel densities
• High transmission speeds
• Interruption-free hardware changes • Ventilation aspects
• Support
Careful, foresighted planning plus the structuring of communications cabling systems should therefore be high on the list of responsibilities for data center operators. Furthermore, related regulations such as Basel II and SOX (U.S.) stipulate consistently stringent transparency.
System Selection
Due to the requirement of maximum high availability and ever increasing data transmission rates, quality demands on cabling components for data centers are considerably higher than on those for LAN products. Quality assurance thus starts in the early stages of planning, when selecting systems that meet the performance requirements listed here.
• Cable design of copper and fiber optic systems • Bandwidth capacity of copper and fiber optic systems • Insertion and return loss budget of fiber optic systems • EMC immunity of copper systems
• Migration capacity to next higher speed classes • 19" cabinet design and cable management
Both fiber optic and copper offer the option of industrially preassembled cabling components for turnkey systems in plug-and-play installations. These installations provide the highest possible reproducible quality, very good transmission characteristics and high operational reliability. Given the high requirements for availability and operational stability, shielded systems are the best choice for copper systems.
Global standards require a minimum of Class EA copper cabling. The choice of supplier is also something that requires careful consideration. The main requirements of a reliable supplier are – in addition to the quality of cabling components – professional expertise, experience with data centers and lasting supply performance.
Structure
Data centers are subject to constant change, as a result of the short life cycles of active components. To avoid having to perform major changes in the cabling system with the introduction of every new device, a well-structured, transparent physical cabling infrastructure that is separated from the architecture and that connects the various premises running the devices in a consistent, end-to-end structure is recommended.
Data center layouts and cabling architecture are discussed in sections 3.2. and 3.4. respectively.
Redundancy and Reliability
The requirement of high availability necessitates the redundancy of connections and components. It must be possible to replace hardware without interrupting operation, and in case a link fails, an alternative path must take over and run the application without any glitches. Proper planning of a comprehensive cabling platform is therefore essential, taking into account factors like bending radii, performance reliability and easy and reliable assembly during operation.
The availability of applications can be further increased by using preassembled cabling systems, which also reduces the amount of time installation staff need to spend in the data center's security area for both the initial installation as well as expansions or changes. This further enhances operational reliability.
It is also important that all products be tested and documented under compliance with a quality management system.
Like the data center cabling itself, connections used for communication between data centers (e.g. redundant data centers and back-up data centers) and for outsourcing and the backup storage of data at a different location, need to be redundant as well. The same applies to the integration of MAN and WAN provider networks.
Installation
In order to be qualified for installation and patching work in data centers, technicians need to be trained in the specifications of the cabling system.
Suppliers like R&M provide comprehensive quality assurance throughout the entire value chain – from the manufacturing of components, through professional installation and implementation up to professional mainte-nance of the cabling system.
Another advantage of the above-mentioned factory-assembled cabling systems is the time saved during installation. Moreover, when extending capacity by adding IT equipment, the devices can be cabled together quickly using preassembled cabling, thus ensuring that the equipment and the associated IT applications are in operation in the shortest time; the same applies to hardware.
Cabinet systems with a minimum width of 800 mm are recommended with regard to the 19" server cabinet. They allow a consistent cable manage-ment system to be set up in a vertical and horizontal direction. Cabinet depth is usually determined by the passive and active components to be installed. A cabinet depth of 1000 mm to 1200 mm is recommended for the installation of active components.
Ideally, a server cabinet has a modular structure, ensuring reliability at manageable costs. A modular cabinet can be dismantled or moved to a different location if needed. Modularity is also important with containment solutions and climate control concepts.
Equally important is stability. Given the high packing densities of modern server systems and storage solutions incl. the power supply units, capacity loads of over 1000 kg are required for server racks. This means that cabinet floors and sliding rails must also be designed for high loads. Loads of 150 kg per floor tile or rail are feasible today.
Cable management is another important aspect. With growing trans-mission speeds, it is absolutely essential to run power and data cables separately in copper cabling systems so as to avoid interference.
Due to increasing server performance and higher packing density in racks, ventilation concepts such as perforated doors and insulation between hot and cold areas in the rack are more and more important. Further productivity-improving and energy-optimized solutions can be achieved by means of cold/hot aisle approaches, which are part of the rack solution.
Documentation and Labeling
A further essential prerequisite for a good cabling system administration and a sound planning of upgrades and extensions is meticulous, continuously updated documentation. There are numerous options available, from individual Excel lists to elaborate software-based documentation tools. It is absolutely essential that the docu-mentation reflect the current state and the cabling that is actually installed at any given point in time.
Related to documentation is the labeling of the cables; it should be unambiguous, easy to read and readable even in poor visibility conditions. Here too, numerous options exist, up to barcode-based identification labels. Which option is best depends on specific data center requirements. Maintaining uniform, company-wide nomenclature is also important. To ensure unambiguous cable labeling, central data administration is recommended.
1.7. Intelligent Cable Management Solutions
Data and power supply cables need to be planned in advance and their routing must be documented to allow immediate response to updates in concepts and requirements. The cable routing on cable racks and runs needs to be planned meticulously. Raised floors are often used for cable routing and allow maintenance work to be carried out in the data center without the need for any structural work. Further information appears in sections 3.5.3. and 3.5.4.
Cable penetration seals, however, often prove to be a weak point. Like walls, ceilings and doors, penetration seals need to fulfill all safety requirements with respect to fire, gas and water protection. To allow for swift and efficient changes and retrofittings in the cabling, they also need to be flexible.
1.8. Network Hardware and Virtualization
A comprehensive examination of a data center must include server security needs and the network. Many companies have already converted their phone systems to Voice over IP (VoIP), and the virtualization of servers and storage systems is spreading rapidly. Virtualized clients will be the next step.
This development means that business-critical services will be run over data links, which also carry the power to the terminal devices via Power over Ethernet (PoE). Along with the growing importance of network technology to ensure uninterrupted business operations, security requirements are growing in this area too.
The rack is the basis for dimensioning servers as well as components in network technology. Since active components are standardized to 19", network cabinets are usually based on the same platform. Requirements with regard to stability, fire protection and access control are comparable as well.
Network infrastructures in buildings are generally designed to last for 10 years, therefore foresighted planning in the provisioning of network cabinets is recommendable. Accessories should be flexible to make sure that future developments can be easily integrated. In terms of interior construction, racks differ considerably.
Due to the frequent switching at connection points, i.e. ports, of network components, cables in network cabinets need to be replaced more frequently than those in server cabinets. These processes, also called MACs (Moves, Adds, Changes) and the increasing density of ports make good cable management even more important. Top and bottom panels in the rack are subject to these demands as well. Easy cable entry in these areas make extensions easier and ensure short cable runs. Cable ducts and management panels ensure clean, fine distribution in the rack. In cable management, particular attention has to be paid to the stability of components. Minimum bending radii of cables must also be considered.
Climate control is another aspect which is rapidly gaining in importance with network cabinets. Switches and routers are increasingly powerful, thereby producing more heat. It is important to allow for expansion when selecting climate control devices. They range from passive cooling through the top panels, venting devices, double-walled housings, ventilators, top cooling units and cooling units between the racks.
Find further information on network hardware and virtualization in sections 3.6 and 3.7. respectively.
1.9. Data Center Energy Consumption
Greenpeace claims that at current growth rates, data centers and telecommunication networks will consume about 1’963 billion kilowatt hours of electricity in 2020, which would translate into a threefold increase within 10 years. According to Greenpeace, that’s more than the current electricity consumption of France, Germany, Canada and Brazil combined.
The powerful emergence of Cloud Computing and the use of the Internet as an operating system and not just a way to access stored data are reported to generate an enormous increase in the energy needs of the IT industry.
Data centers are bursting at their seams. In their studies, Gartner analysts predict that data centers will reach their limits in the next three years, with respect to energy consumption and space requirements. In their opinion, this is mostly due to technical developments such as new processors in multi-core design, blade servers and virtualization, which push density higher and higher. According to the analysts, density will increase at least tenfold in the next ten years, driving energy consumption for both operation and cooling to considerably higher levels. This is particularly true for data centers that have been in operation for a longer period of time and use outdated technologies. In some of these, as much as 70% of the energy consumption is required just for cooling.
Theenergyconsumptionofprocessorstodayamounts to approx. 140 watts, that of servers approx. 500 watts and that of blade servers approx. 6300 watts.
Each watt of computing performance requires 1 watt of cooling. The table on the left summarizes the actual energy consumption.
Conclusion:
1 watt IT savings = 3 watts of total savings!
In 2003, the average energy consumption per server cabinet amounted to 1.7 kW, in 2006 it was already up to 6.0 kW and in 2008itwas8.0kW.Today,energy consumption is 15 kW for a cabinet containing the maximum number of servers and 20 kW with the maximum number of blade servers. The overall energy consumption in a data center is shown in the pie chart right:
Energy consumption at server level 1 watt
DC/DC conversion + 0.18 watt
AC/DC conversion + 0.31 watt
Power distribution + 0.04 watt
Uninterruptible power supply, UPS + 0.14 watt
Cooling + 1.07 watt
Building switchgear/transformer + 0.10 watt
Total energy consumption 2.84 watts
1.10. Energy Efficiency – Initiatives and Organizations
There are numerous measures for increasing energy efficiency in data centers. The chain of cause and effect starts with applications, continues with IT hardware and ends with the power supply and cooling systems.
A very important point is that measures taken at the very beginning of this chain, namely for causes, are the most effective ones. When an application is no longer needed and the server in question is turned off then less power is consumed, losses in the uninterruptible power supply decrease and so does the cooling load.
Virtualization – a way out of the trap
Virtualization is one of the most effective tools for a cost-efficient green computing solution. By partitioning physical servers into several virtual machines to process applications, companies can increase their server productivity and downsize their extensive server farms.
This approach is so effective and energy-efficient that the Californian utility corporation PG & E offers incentive rebates of 300 to 600 U.S. dollars for each server that is saved thanks to Sun or VMware virtualization products. These rebate programs compare the energy consumption of existing systems with that of systems in operation after virtualization. The refunds are paid once the qualified server consolidation project is implemented. They are calculated on the basis of the resulting net reduction in kilowatt hours (at a rate of 8 Cents per kilowatt hour). The maximum rebate is 4 million U.S. dollars or 50 percent of the project costs.
By implementing a virtual abstraction level to run different operating systems and applications, an individual server can be cloned and thus used more productively. On the strength of virtualization, energy savings can be increased in practice by a factor of three to five – and even more in combination with a consolidation to high-performance multi-processor systems.
Cooling – great potential for savings
A rethinking process is also taking place in the area of cooling. The cooling system of a data center is turning into a major design criterion. The continuous increase in processor perfor-mance is leading to a growing demand for energy, which in turn leads to considerably higher cooling loads. This means that it makes sense to cool partitioned sections in the data center individually, in accordance with the specific way heat is generated in the area.
The challenge is to break the vicious circle of an increased need of energy leading to more heat, which in turn has to be cooled, consuming a lot of energy. Only an integrated, overall design for a data center and its cooling system allows performance requirements for productivity, availability and operational stability to be synchronized with an energy-efficient use of hardware.
In some data centers, construction design focuses more on aesthetics than on efficiency, an example being the hot aisle/cold aisle constructions. Water or liquid cooling can have an enormous impact on energy efficiency, and is 3000 times more efficient than air cooling. Cooling is further discussed in section 3.5.2.
Key Efficiency Benchmarks
Different approaches are available for evaluating the efficient use of energy in a data center. The approach chosen by the Green Grid organization works with two key benchmarks: Power Usage Efficiency (PUE) and Data Center Infrastructure Efficiency (DCIE). While PUE determines the efficiency of the energy used, the DCIE value rates the effectiveness of the energy used in data centers. The two values are calculated using total facility power and IT equipment power. The DCIE value is the quotient of IT equipment power and total facility power, and is the reciprocal of the PUE value. The DCIE thus equals 1/PUE and is expressed as a percentage.
A DCIE of 30 percent means that only 30 percent of the energy is used to power the IT equipment. This would result in a PUE value of 3.3. The closer this ratio gets to the value of 1, the more efficiently the data center uses its energy. Google can claim a PUE value of 1.21 for 6 of their largest facilities.
Total facility power includes the energy used to power distribution switch board, the uninterruptible power supply (UPS), the cooling system, climate control and all IT equipment, i.e. computers, servers, and associated communication devices and peripherals.
PUE DCiE Level of
efficiency 3.0 33% very inefficient 2.5 40% inefficient 2.0 50% average 1.5 67% efficient 1.2 83% very efficient
In addition to the existing key benchmarks, the Green Grid is introducing two new metrics, whose goal is to support IT departments in optimizing and expanding their data centers. These new metrics are called Carbon Usage Effectiveness (CUE) and Water Usage Effectiveness (WUE). CUE will help managers determine the amount of greenhouse gas emissions generated from the IT gear in a data center facility. Similarly, WUE will help managers determine the amount of water used to operate IT systems.
Initiatives and Organizations
The Green Grid is an American industry association of suppliers of supercomputers and chips, dedicated to the concept of environment-friendly IT, also called green IT. The consortium was founded by the companies AMD, APC, Cisco, Dell, Hewlett Packard, IBM, Microsoft, Rackable Systems, Sun Microsystems and VMware, and its members are IT companies and professionals seeking to improve energy efficiency in data centers. The Green Grid develops manufacturer-independent standards (IEEE P802.3az Energy Efficient Energy Task Force), and measuring systems and processes to lower energy consumption in data centers.
The European Code of Conduct on Data Centres Energy Efficiency was introduced by the European Commission in November 2008 to curb excessive energy consumption in data center environments. The code comprises a series of voluntary guidelines, recommendations and examples of best practices to improve energy efficiency. Data centers implementing an improvement program recognized by the EU commission may use the code's logo. Every year, data centers that are especially successful receive an award. In the medium-term, quantitative minimum requirements will also be defined.
The SPEC Server Benchmark Test of the U.S. American Standard Performance Evaluation Corporation (SPEC) has kicked off the efficiency competition in the IT hardware section. Manufacturers are focusing on producing and testing configurations that are as efficient as possible.
1.11. Green IT
Green IT is about reducing operational costs while at the same time enhancing a data center's productivity. Targeted are measurable, short-term results. A top priority for IT managers in organizations is the efficiency of data centers.
Financial service providers, with their enormous need of computing power, are not the only ones who can profit from green IT. Today, companies in all industrial sectors are paying much more attention to their electricity bills. According to a study by Jonathan Koomey, a professor at the Lawrence Berkeley National Laboratory of the Stanford University, energy costs for servers and data centers have doubled in the last five years.
In 2010, the U.S. Environmental Protection Agency (EPA) projected that the electricity consumption of data centers would double again in the coming five years, generating costs of another 7.4 billion dollars per year. These bills are processed in the finance departments of the companies. The pressure is now on IT managers to reduce energy costs, which in turn results in environmental benefits.
Along with the automotive and real estate industries, ICT is one of the key starting points for the improvement of climate protection. According to a study1 by the Gartner group, ICT accounts for 2 to 2.5 percent of the global CO2 emissions, about the same as the aviation industry. The study states that a quarter of these emissions are caused by large data centers and their constant need for cooling. In office buildings, IT equipment power normally accounts for over 20 percent, in some offices for over 70 percent, of the energy used.
In addition to the energy demand for production and operation of hardware, i.e. computers, monitors, printers and phones, the materials used and production methods also need to be taken into account. This latter area includes the issue of hazardous substances, whether such substances are involved in the production or if any poisonous substances such as lead or bromine are contained in the end product and released during use or disposal. More detailed specifications are provided by the RoHS guideline of the EU (RestrictionofHazardousSubstances).
1.12. Security Aspects
In many countries, legislation stipulates that IT systems are an integral part of corporate processes and thus no longer only a tool to advance a company's success, but an element that is bound by law and in itself part of the corporate purpose. Laws and regulations such as the German Supervision and Transparency in the Area of Enterprise Act (KonTraG), the Basel II Accord or the Sarbanes-Oxley-Act (SOX) almost entirely integrate the company's own IT into the main corporate processes. As a consequence, IT managers have to deal with significant liability risks. Companies and organizations find themselves facing IT-related complications such as customer claims for compensation, productivity losses and negative effects on the corporate image, to name but a few.
The need to act and set up IT systems that are more secure and available is therefore a question of existential importance for companies. The first step is a comprehensive analysis phase to specify weaknesses in IT structures, in order to determine the exact requirements for IT security. The next step is the planning process, taking all potential hazards in the environment of the data center into consideration and providing additional safeguards if necessary. A detailed plan must be worked out in advance to avoid a rude awakening, defining room allocations, transport paths, room heights, cable laying routes, raised floor heights and telecommunications systems.
Taking a broader view of IT security reveals that it goes beyond purely logical and technical security. In addition to firewalls, virus protection and storage concepts, it is essential to protect IT structures from physical damage. Regardless of the required protection class, ranging from basic protection to high availability with minimum downtime, it is essential to work out a requirement-specific IT security concept.
Cost-effective IT security solutions are modular so that they can meet specific requirements in a flexible manner. They are scalable so that they can grow in line with the company and, most of all, they are comprehensive to ensure that in the event of any hazard, the protection is actually in place. It is therefore paramount to know the potential hazards beforehand, as a basis for implementing specific security solution.
Fire Risk
Only 20 percent or so of all fires start in the server room, or its environment. Nearly 80 percent of fires start outside IT structures. This risk therefore needs to be examined on two levels. Protection against fire originating in the server room can be covered by early fire detector systems (EFD), a fire-alarm and extinguishing systems. These systems can also be designed redundantly – so that false alarms can be avoided. EFD systems suck ambient air from racks by means of active smoke extraction systems which detect even the smallest, invisible smoke particles. Digital particle counters, as used in laser technology, can also be applied here.
Due to high air speeds in climate-controlled rooms, the smoke is rapidly diluted, meaning that the EFD systems must have a sufficiently high detection sensibility. Disturbances can be avoided or kept away with filters and intelligent signal-processing algorithms. Professional suppliers also offer these systems in a version that combines fire-alarm and extinguishing systems, which can be easily integrated in 19" server racks for space efficiency. Using non-poisonous extinguishing gases, fires can be put out in
their pyrolysis phase (fire ignition phase), effectively minimizing potential damage and preventing the fire from spreading further. Extinguishing gas works much faster than foam, powder or water, causes no damage and leaves no residue. In modern systems, gas cartridges can even be replaced and activated without the help of technicians. In addition to FM-200 and noble gas (e.g. argon), nitrogen, inergen or carbon dioxide are also used to smother fires through oxygen removal. Then there are extinguishing gases which put out fires by absorbing heat, e.g. the new NovecTM 1230. Their advantage is that only small quantities are required.
In addition to the use of extinguishing gases, the oxygen level can be reduced (called inertization) as a parallel process in fire-hazard rooms such as data centers. The ambient air is split into its individual components via an air decomposition system, reducing the oxygen level to approx. 15 percent – providing a means of early fire prevention. This oxygen reduction does not mean that people cannot enter the data center, as it is principally non-hazardous to the humans. Both EFD and fire-alarm and extinguishing systems are now available from leading manufacturers in space-saving, installation-friendly 1U versions, showing that effective protection is not a question of space.
Fire in the data center
In the evening of March 1, 2011, a fire broke out in the high-performance
data center in Berlin. For security reasons, all servers were shut down and the entire space was flooded with
CO2 to smother the fire!
Water Risk
An often neglected danger for IT systems is water. It is usually not a threat in the case of pipe leaks or floods, but represents a danger in the form of extinguishing water in the event of the above-discussed fire threats. The damage caused by the fire is often less severe than the damage caused by the water used to extinguish it.
This means that IT rooms need to stay watertight during the fire-fighting process, and they must be resistant to long-standing stagnant water that, for example, occurs in a flood. Watertightness should be proven and independently certified to be EN 60529-compliant. Protection from stagnant water over a period of 72 hours is the current level of technology with high-availability systems. The latest developments allow data centers to be equipped with wireless sensors that are able to detect leaks early, send warning signals and automatically close doors if required. This is particularly important when highly efficient liquid-cooling systems are used for racks.
Smoke Risk
Even if a fire is not raging in the immediate vicinity of a data center, smoke still presents a risk of severe damage to IT equipment. In particular, burning plastics such as PVC create poisonous and corrosive smoke gases. Burning one kilogram of PVC releases approx. 360 liters of hydrochloric acid gas, producing up to 4500 cubic meters of smoke gas. This can destroy the IT facility in little time, a fact that reduces the "mean time between failure" (MTBF) substantially. MTBF is the predicted elapsed time between inherent failures of a system during operation.
Reliable protection can only be provided by hermetically-sealed server rooms which are able to resist these hazardous gases and protect their valuable components against the threat. Smoke gas resistance that is tested in accordance with EN 18095 is essential. In Germany, the level of water and gas resistance is defined based on the IP categorization. A data center should have a protection class of IP56.
Power Supply Risk
Uninterruptible power supply systems, called UPS systems, take over when the power network fails. Modern UPS systems (online systems) operate continually, supplying consumers via their power circuits. This means that the brief yet dangerous change-over time can be eliminated. The UPS system simply and reliably bridges the time until the power network is up and running again. Thanks to integrated batteries, UPS systems also operate continuously if power is off for a longer period of time. UPS systems are classified in accordance with EN 50091-3 and EN 62040-3 VFI. For a reliable breakdown protection, equipment used in data centers should fulfill the highest quality class 1 VFI-SS-111.
UPS systems are divided into the single-phase group and the multi-phase group of 19" inserts and stand-alone units of different performance classes. The units provide perfect sinus voltage and effectively balance out voltage peaks and "noise". Particularly user-friendly systems can be extended as required and retrofitted during operation.
If, however, the power supply network stays offline for several hours, even the best batteries run out of power. This is where emergency power systems come into play. These are fully self-sufficient systems that independently generate the power needed to keep the data center running and recharge the batteries in the UPS system. These emergency systems are usually diesel engines that start up in the event of a power failure once the power supply has been taken over by the UPS systems.
New research indicates that in the future these diesel engines could be driven with resource-conserving fuels such as vegetable oil, similar to a cogeneration unit (heat and power plant). This would mean that the units can continuously generate power in an environmentally-friendly manner without additional CO2 emissions, and this power could even be sold profitably when it is not required for the data center operation. Fuel cells will also become increasingly important as power sources for emergency power systems. Fuel cells reduce total cost of ownership (TCO) and offer distinct advantages over battery-buffered back-up systems, with respect to service life, temperature fluctuations and back-up times. In addition, they are very ecological because they generate pure water as reaction product.
Air-Conditioning Risk
Modern blade server technologies boost productivity in data centers. Climate-control solutions are primarily concerned with transporting the heat emitted in the process. Data center planners must remember that every productivity boost also increases the demand for cooling capacity of the air-conditioning units in a data center. With maximum thermal loads of 800 W/m2 in a data center, cooling units that are suspended from the ceiling or mounted to walls can be used. With thermal loads above 800 W/m2, however, the data center requires floor-mounted air-conditioning units which blow the air downwards into the raised floor.
In general, air-conditioning units can be placed inside or outside a data center. Inside use is better for rack-based cooling or specific cooling of hot spots in a server room. Operational costs are lowered and the associated noise stays inside the room. Units are also protected from unauthorized access and the protective wall is not weakened by additional openings for ventilation slides. The advantages of outside use are that service technicians do not need to enter the room for maintenance work, the air-conditioning units require no extra space in the data center and the fire load is not increased by the air-conditioning system. Fresh air supply can generally be achieved without additional costs.
In most cases, redundancy in accordance with Tier III and IV (see section 2.2.1.) is only required for air-conditioning units on walls and ceilings that have a rather low cooling capacity. When higher capacities and thus stand-alone units are required, a "N+1 redundancy" in accordance with Tier II should be established, which leaves a number of units in continuous operation while an additional unit acts as redundancy (reserve).
In order to establish and maintain the recommended relative humidity range of 40% to 60%, the air-conditioning units should feature both air humidifiers and dehumidifiers. A safe bet is to opt for a system certified by Eurovent (the interest group of European manufacturers of ventilation and air-conditioning systems). For cooling hot spots in data centers, the use of liquid-cooling packages is also an option. These packages extract the emitted heat/hot air along the entire length of the cabinet by means of redundant, high-performance fans, discharging it via an air and water heat exchanger into a cold water network or a cooler.
Dust Risk
Dust is the natural enemy of sensitive IT systems and does not belong in secure data centers. Fine dust particles can reduce the life cycle of ventilators and other electronic units drastically. One main source of dust is maintenance work and staff – any intrusion into secured data centers must be avoided. An intelligent IT room security system is absolutely free. The free policy also applies to extension or upgrade work. The dust-tightness in place should comply with the specifications in EN 60529, and fulfill IP56 with characteristic 1 (see water risk) in order to avoid any unpleasant surprises at a later stage.
Unauthorized Access Risk
The data center is one of the most sensitive areas in a company. It is critical that only authorized persons have access, and that every data center access is documented. A study of the International Computer Security Association (ICSA) showed that internal attacks on IT systems are more frequent than external ones. Protection of the data center must therefore first meet all requirements in terms of protection against unauthorized access, sabotage and espionage, and also ensure that specifically authorized persons can only enter the rooms they need to perform their defined tasks. Burglary protection in accordance with EN 1627 with resistance class III (RCIII) is easily implemented. All processes are to be monitored and recorded in accordance with the relevant document-tation and logging regulations.
If possible, the air conditioning system and electrical equipment should be physically separated from the servers so that they can be serviced from the outside. For access control purposes, biometric or standard access control solutions can be installed or a combination of the two. Biometric systems in combination with magnetic card scanners enhance the security level considerably. Most of all, the access control solution installed should fulfill the specific requirements of the operator. The highest level of security can be guaranteed by the new vein recognition technology. Its high precision and thus critical advantage stands out in its false acceptance rate of less than 0.00008 percent and a false rejection rate of only 0.01 percent. Moreover, it ensures highly hygienic handling, since operating the device does not require direct contact.
By using video surveillance systems with image sensors in CCD and CMOS technology, up to 1000 cameras can be managed with the matching software, regardless of the manufacturer. Video surveillance systems provide transparency, monitoring and reliability in data centers. Advanced video management technology enables modern surveillance systems to manage and record alarm situations. For images to be analyzed and used as evidence, an intelligent system must provide the proper interfaces and processing possibilities.
Explosion Risk
The risk of terror attacks or other catastrophes which could trigger an explosion must be factored in right from the start when planning a highly available security room concept. Modern, certified server rooms need to undergo an explosion test in accordance with the SEAP standard. High-security modular server rooms are built with pressure-resilient wall panels to resist heavy explosions, protecting valuable IT systems from irreparable damage. IT systems need also be protected against debris and vandalism, to ensure real all-round protection.
