Preparing to Operate a New Data Center Facility. AFCOM October 3, 2012

Preparing to Operate a

New Data Center Facility

AFCOM

Background

 Managed 100k sq.ft. critical data center facility for GTE Data Services (now Verizon)

 Site Uptime Network Program Director  ComputerSite Engineering team leader  David Boston Consulting since 2006

• Critical Facilities Operations, including new facility start-up strategy and implementation assistance, procedures and training programs, assessments

Proper Planning

 An enormous amount of planning and

coordination goes into the justification, design, and construction of a new data center facility

 Commissioning

 Factory acceptance  Site acceptance

Proper Planning -

 It is equally important to thoroughly develop your strategy for staffing and operating the facility  Much of the strategy must be implemented while

Why is This Critical?

 Facilities related downtime for computer

operations is caused by human error 60-80% of

the time

Why is This Critical? -

 Companies that are serious about uptime

dedicate significant effort to Facilities staff plans, procedures, and training, in addition to

constructing systems designed for high reliability  A data center facility will only operate

successfully if the Facilities team is provided

management support, appropriate resources, site-specific systems experience, and the

Developing a Strategy

 Clarify your new facility’s reliability objective

 Match Facilities Operations strategy to systems availability level expected

 This drives staff size, shift coverage, operating budget  It will also dictate the level of training and procedures

Organizational Structure

 Determine optimal structure to maximize success

 Which department will Facilities Operations report to in order to receive the best

communication, funding, and resources?

(And achieve optimal shared processes for hardware master planning and power/network installation)

Annual Objectives

 Facilities staff objectives should include

collective goals for consistent facilities systems uptime and successful/safe completion of all

assigned PM tasks and customer requests

 Individual objectives should vary by position, allowing ownership of specific systems, tasks, and projects to be clear (accountability)

Hiring Timeline

 Hire the Facilities team early enough in the

design/construction process

 Involvement in construction monitoring will pay off  Several should participate in factory acceptance, site

What is the Right Timing?

 What do you think the appropriate timing is to

hire the Facilities Manager?  Shift supervisor?

 Shift engineers?

 Support services coordinator?  Project coordinator?

Almost Always Overlooked

 Unless 2-3 months are scheduled after the

construction and commissioning process for Facilities training, the Facilities team cannot develop the needed confidence level to

successfully respond to systems incidents

 Computer Operations and Network Operations

Procedures Recommendations

 A comprehensive program is needed (80-200 Facilities procedures, depending on the facility design)

 A number of the desired procedures should be

developed and tested prior to initial operation (tested during the 2-3 month window), if the

Procedures -

 Unless all are completed in advance, a priority

order for procedure development is important:

1. Emergency response procedures first

2. Planned “system transfer” procedures in the order that equipment PMs will be scheduled

Procedures -

 One of the Facilities team members should be

assigned the responsibility of Procedures and

Training Coordinator

 The Coordinator must be given resources to

accomplish this

• In-house staff assigned to help part time • Contracted system “experts” and writer

Procedures -

 Ensure that all procedures are tested

 What is clear to the author often leaves room for confusion for the reader

Procedures -

 It is critical that two individuals participate in each planned activity that utilizes a procedure

• One to read aloud, the other to confirm and act

• This greatly diminishes the risk of skipping a step or performing a step out of order

• It also encourages discussion and a potential decision to back out if there is any confusion

Sample Procedures Format

1. ____ Find breaker control switch “GTA HVEA GEN. TIE BREAKER” in the

bottom row of controls in Master Control Cubicle MC

2. ____ Turn this control switch

counterclockwise to the “TRIP” position and then release

Training Program

 Each equipment vendor will typically deliver training after equipment has been installed

 Scheduling should be managed to prevent overload

 Review each vendor’s materials in advance  Professionally video record the training

Training Program

 More critical than vendor training is the

implementation of monthly site-specific training

(train on each system at least annually)

January: Generators, controls, & fuel systems

February: Life safety and evacuation

March: Cooling towers, chillers, pumps, valves

April: UPS, UPS switchgear, batteries

May: CRAHs, VAVs, AHUs

Training Program -

 The Procedures and Training Coordinator should involve many of the identified system “experts” to conduct initial monthly sessions

 In-house staff should become trainers over a planned timeline

 Classroom (group) and in-front-of-equipment (individual) sessions each month

Training Program -

 Engineers of Record or Commissioning Agents

should be contracted to deliver Systems Design

Training Program -

 Identify system “experts” to help with initial in-house monthly training sessions

Training Program -

 Once your site specific program is established:

 Internal certification  Recognition

Budget Planning

 Experience provides the best resource for planning your new facility’s expense budget

 If your company has operated a data center

previously, that expense history should be utilized

when developing the new facility’s initial budget  History within same region most useful

Budget Planning -

 Your electrical engineer of record and your electric utility representative can help you estimate your initial year’s utility costs

 Utility expenses are consistently the largest operating expense in a data center facility

Budget Planning -

 The second highest cost category will be labor

and related expenses for Facilities Operations

Budget Planning -

 Collectively, maintenance contracts for

preventive maintenance (PM) on major systems will typically be the next highest expense

Budget Planning -

 Remaining categories include:

 Spare parts, materials, tools, uniforms, supplies

(First confirm what construction budget will cover)

 Janitorial (including sub-floor cleaning)  Grounds

Budget Planning -

 Expenses for individual data center facilities

vary widely due to:

 Cost of electric utility service (kWH)

 Levels of redundancy & concurrent maintainability

 Load variances: types of computer hardware installed  Salary & benefits for Facilities Operations team

 Union vs. non-union staff

 Single shift vs. continuous shift coverage  Number of employees per shift

Orientation Program

 Formal playbook to accelerate knowledge &

confidence acquisition for new team members  Include:

 Introduction (why site critical, cautionary philosophy)  Systems overviews and descriptions

 One line diagrams, building layout  Safety and Security policies

 Training session and major PM calendar, first year expectancies

Workspace Strategies

 Benchmark with peers or hire service providers

to help you maximize functionality for:

 Equipment rooms, parts storage areas

 Facilities shop, office, command center/library

Additional Strategies for Success

 Control processes

 Specific data center work rules

 Limited access – minimize those permitted badges  Shipping/receiving only on a planned basis

Additional Strategies -

 Team Development

 Clearly defined IT and Facilities relationship, mutual expectations (or SLAs), shared incentives  Defined Data Center Facilities and Office Facilities

Measuring Success

 Effectively deploying these strategies will

provide for a much higher reliability potential

in the first year of operation

 Continued application will dramatically increase continuous facilities systems operation potential over the life of the facility

Measuring Success -

 You may realistically aim for multiple years of

continuous facilities systems availability, a multi-million $ savings

Contact Information

E-mail: [email protected]

Web site: www.DavidBostonConsulting.com

References: upon request

Phone: 828-526-1675

Fax: 828-526-1673

Address: 215 Wildwood Dr.