Office of Information Technology
Core Services Resilience Plan
Version 6.5.6 March 2010
Table of Contents
Overview ... 3
Background ... 4
OIT Organizational Resilience Program ... 4
Data Centers ... 4
OIT Organizational Resilience Program Governance ... 6
Disaster Declaration Authorities ... 6
Guiding Principles ... 6
Incident Mitigation ... 7
Service Disruption ... 8
Data Center II Disruption ... 9
Data Center I Disruption ... 10
Both Data Center I and Data Center II Disruption ... 11
Review and Approval ... 12
Appendix ... 13
Core Services Recovery Timeline ... 13
By virtue of its design as a guiding blueprint, this plan is a “living document” that will change and evolve over time to adequately address the needs of NC State University’s information technology environment, scope and strategic direction.
Overview
The purpose of this document is to highlight the direction of resilience for North Carolina State University’s information technology environment. It serves as a “roadmap” for attaining a high state of readiness that allows the university’s central technical staff the flexibility and speed to react to unknown disruptions to the university’s infrastructure. The OIT Core Services Resilience Plan addresses OIT’s organization, its strategy and vision, processes, applications and data, technology and facilities. These areas are all interdependent, and their strength contributes to OIT’s overall state of readiness.
This document does not represent an attempt to anticipate every possible scenario that might affect the IT infrastructure of NC State University. Through the adoption of resiliency, however, it does represent an attempt to pro-actively create an environment that perpetually guards against and mitigates foreseen and unforeseen challenges. The adoption of this philosophy is supported by governing policies and procedures, governmental regulations and industry best standards. Examples of core services (but not limited to) that have been identified and focused on in this document include:
1. Enterprise Resource Planning Services (Student, Human Resources, Financials, and Portal)
2. E-mail Relays
3. Enterprise Resource Planning Internet Web Services 4. Authentication and Authorization
Background
OIT Organizational Resilience Program
Organizational Resilience represents a broader perspective than most business continuity planning. Most enterprises focus on short-term survival following a given operational failure – the emphasis being on operational recovery and tangible assets – and all too frequently on IT-related assets only. Organizational Resilience is a comprehensive approach intended to create a state of readiness that rapidly responds to unforeseen change (including but not limited to chaotic
disruption) with speed, precision and determination. Organizational Resilience emphasizes continuance (flexibility of daily operations), people, services, processes and external dependencies rather than operational recovery and tangible assets.
By adopting the philosophy of Organizational Resilience, we increase our capacity to:
1. Maintain (restore when necessary and as expediently as possible) the centrally hosted applications, desktop environment and IT infrastructure, including the university
telephone system and Web services.
2. Facilitate customer access – where and when necessary – to centrally hosted applications, systems and services using a variety of methods: the standard desktop delivery mechanisms as well as alternative mechanisms to include, but not limited to, remote access for employees’ homes or off-campus temporary locations, central data access/entry sites such as training labs, converted conference areas, and/or other temporarily provisioned spaces on campus.
Organizational Resilience is an additional consideration to the services provided by and the processes supported by OIT. It is an ongoing effort to mitigate risks and minimize the impact of disruptions to the services provided to the university community and other entities.
Data Centers
The university currently maintains two primary data centers with system equipment that supports the computing infrastructure: Data Center I (DC I) and Data Center II (DCII). The goal of the two data centers is to provide co-processing where applicable and to maintain fail-over capability for restoration of the central computing environment and data. Figure 1 below depicts the current data center configurations. The long-term goal of this plan is, in the event of a service disruption, to be able to redirect services to either of the data centers to lessen the impact to customers. Core services and supporting technologies that are provided within the data centers are so interdependent that separate processes cannot be restored individually without restoring entire systems. Mitigating risk and centralizing core application data delivery reduce the potential impact to affected users regardless of their locations or particular processes. The highly
integrated infrastructure facilitates easier management and access of services during an incident. Continued efforts are being made to reach the long-term goal.
Figure 1 Key:
Solid Dark Pink Fill: Assets that are fully redundant (i.e. exist in both data centers). Partial Fill: Assets that are not fully redundant (i.e. partly exist in both data centers). No Fill: Assets that are not at all redundant (i.e. exist in only one data center).
Notes:
Redundancy does not imply high availability. Although some services are deployed in a highly
available configuration, most are not. Technical teams will need to bring services that are redundant but not highly available back online in the alternate data center.
The Storage Area Network in Data Center II hosts some services that are not fully redundant. This is indicated by the DC II Storage Area Network graphic having a tan fill color. The Storage Area Network in Data Center I is redundant across both data centers.
Hosted Systems includes a variety of services - some of which are redundantly deployed, and some are not.
OIT Organizational Resilience Program Governance
Disaster Declaration Authorities
Marc Hoit – Vice Chancellor for Information Technology and Chief Information Officer Mardecia Bell – Director of OIT Security and Compliance
Guiding Principles Regulations o PCI o HIPPA o FERPA o Sarbanes-Oxley
o North Carolina State University Policy, Regulations and Rules REG04.00.7 Standards and Best Practices
o NIST: 800-30, 800-34, 800-52 o ISO: 20000, 17799, 9001
Page 7 of 13
Incident Mitigation
The computing environment of North Carolina State University is highly integrated and complex. Examples of core services that have been identified include:
• Enterprise Resource Planning Services (Student, Human Resources, Financials, and Portal)
• E-mail Relays
• Enterprise Resource Planning Internet Web Services • Authentication and Authorization
• Infrastructure Services
In order to achieve and maintain resiliency, upgrades and improvements to the above services, as well as their accompanying essential components and the constant mitigation of risks are necessary. Plans to enhance the resiliency of core services are recommended to:
1. Minimize risk of service loss
2. Expedite restoration of core services
3. Provide a stable environment for future growth
4. Align more IT objectives with industry standards, peers and the university’s mission There are four types of incidents that must receive constant assessment and mitigation to accomplish resiliency (See Figure 2). This section represents a high-level summarization of the four types of incidents. Scenarios are listed in order of severity (least impact to greatest impact). A. Service Disruption One or more core service(s) (i.e. payroll, accounts
payable) has become unstable, inoperable and/or
unavailable and thus, cannot be delivered from either data center. Examples: Equipment failure, malfunction or batch
job failure.
B. Data Center II Disruption All core services normally supported and delivered from Data Center II have become unstable, inoperable and/or unavailable via normal production responses and
processes. Examples: Localized power failure or flooding. C. Data Center I Disruption All core services normally supported and delivered from
Data Center I have become unstable, inoperable and/or unavailable via normal production responses and
processes. Examples: Localized power failure or flooding. D. DCI and DC II Disruption All core services delivered from both data centers have
become unstable, inoperable and/or unavailable via normal production responses and processes. Examples:
Regional disaster such as a hurricane or tornado.
Service Disruption
Service Disruption is defined as follows: One or more core service(s) (i.e. payroll, accounts payable) has become unstable, inoperable and/or unavailable and thus, can not be delivered from either Data Center. Examples: Equipment failure, malfunction or batch job failure. The existing plan of restoration of one or more core services includes the following high-level tasks and is estimated to typically take up to 50 man-hours. This time will vary depending on the disruption.
1. IT Organizational Resilience Services Incident Management 2. Risk Identification and Isolation
3. Solution, Restoration and Resource Identification 4. Communication of Restoration of Service
Page 9 of 13
Data Center II Disruption
Data Center II Disruption is defined as follows: All core services normally supported and delivered from Data Center II have become unstable, inoperable and/or unavailable via normal production responses and processes. Examples: Localized power failure or flooding.
In the event of a catastrophic impact to the infrastructure of DC II, OIT maintains contracts with third-party vendors to replace any damaged equipment of the site. Activation of this service requires a formal Declaration of Disaster (See Disaster Declaration Authorities, Page 6). The primary restoration location of DC II is DC I. Due to the lack of power, cooling and space capacity of DC I, alternate restoration locations may be needed to restore lost services. These alternate locations may include but are not limited to other campus facilities, third-party
contractors (e.g. portable trailer), UNC System Disaster Recovery Sites and partner universities. Assumptions:
• Enterprise Portal, HR, SIS and Financials will be available at a reduced capacity. • Batch jobs will be manually run.
• EDI payroll transactions will be transmitted via the normal processes. • Appropriate OIT staff are available.
Reduced Capacity State:
• Customers can log in to ERP environment (may need to be limited). • Only production services will be available.
• Reporting will not be available.
• Development environment will not be available. • Increased load will decrease performance.
• Transmissions to external vendors/institutions may be able to occur. • Check printing will be available.
The existing plan of restoration for the current identified core services should a Data Center II Disruption occur is estimated to take a maximum of 20 calendar days. Additional details along with the required high-level recovery steps are provided in the Core Services Recovery Timeline document listed in the Appendix, Page 13.
Data Center I Disruption
Data Center I Disruption is defined as follows: All core services normally supported and delivered from Data Center I have become unstable, inoperable and/or unavailable via normal production responses and processes. Examples: Localized power failure or flooding.
OIT maintains contracts with a third-party vendor to replace any damaged equipment of Data Center I in the event of a Data Center I disruption. Activation of this service requires a formal Declaration of Disaster (See Disaster Declaration Authorities, Page 6).
In the event of a catastrophic impact to the infrastructure of DC I, the primary restoration location is DC II. The current core services are not totally distributed between the two Data Centers. Specifically, the university’s ERP production environment, which encompasses the production, reporting, training, development databases and ancillary systems as well as accompanying hardware, will need to be manually configured in Data Center II.
Assumptions:
• Enterprise Portal, HR, SIS and Financials will be available at a reduced capacity. • Batch jobs will be manually run.
• Appropriate OIT staff are available.
• The scope of the disaster is campus-specific and does not hinder vendor responsiveness.
Reduced Capacity State:
• Customers can log in to ERP environment (may need to be limited). • Only production services will be available.
• Reporting will not be available.
• Development environment will not be available.
• Services on non-ERP databases are not available (Hercules, etc.). • No Sybase production environment available.
• Increased load will decrease performance.
• Transmissions to external vendors/institutions may be able to occur. • Check printing will be available via Paybase from central offices.
In the event of a Data Center I disruption, the existing plan of restoration for the currently identified core services is estimated to take approximately 30 calendar days. Additional details along with the required high-level recovery steps are provided in the Core Services Recovery Timeline document listed in the Appendix, Page 13. Data Center I has a core infrastructure environment that has not yet been migrated to DC II. In the event of a DC1 disruption, the environment will have to be rebuilt. As a result, the restoration time is longer when compared to the unavailability of Data Center II.
Page 11 of 13
Both Data Center I and Data Center II Disruption
The scenario of both Data Centers becoming inoperable is defined as follows: All core services delivered from both Data Centers (DC I and DC II) have become unstable, inoperable and/or unavailable via normal production responses and processes. Examples: Sabotage or natural
disasters.
In the event of a catastrophic impact to the IT infrastructure of both Data Center I and Data Center II, the current plan employs contracts with third-party vendors to temporarily replace damaged equipment at any specified location.
Resilience strategies that are currently being investigated and/or proposed for the specified location include:
• UNC GA Dual Data Centers
• State of North Carolina Disaster Recovery Site • UNC system peer institutions (i.e. UNC-Chapel Hill) • Vendor solutions
• Out-of-state peer institutions
•
MCNCBeing without both data centers simultaneously is the worst presented scenario. At the current state, restoration will probably take from 30 to 60 days, because a specified location will have to be identified and may require installation of core networking infrastructure. When the resilience plan for this scenario is finalized, a better estimated recovery time will be provided.
Review and Approval
This document is a “living document” that will change and evolve over time to adequately adapt to the needs of NC State University’s information technology environment, scope and strategic direction. The OIT Directors and the CIO/Vice Chancellor of Information Technology are required to review and approval this document on a periodic basis (at least annually).
Page 13 of 13
Appendix
Core Services Recovery Timeline (See Next Page)
Adobe Acrobat Document
ID Task Name Duration Predecessors
1 DC1 Outage 27 days?
2 Initial Notification of Customers 2 hrs
3 Declaration of Disaster 4 hrs
4 Reconfigure network environment 10 days
5 Client Server 9 days 4
6 Configure, restore, and boot Netscaler 4 hrs
7 Backup existing alternate DC configurations 4 hrs
8 Reconfigure servers for alternate usage as required 2 days 7
9 Retrieve/restore backup files from VTL/MCNC tape backup 3 days 8
10 Restore/Verify Oracle file space and database allocation 4 hrs 9
11 Reconfigure servers to utilize resources at the alternate DC 3 days 10
12 Mainframe 10 days?
13 Delivery of equipment from Mainline 5 days 3
14 Connect Mainframe to power and networking 3 days 13
15 Install OS and required software 1 day? 14
16 Restore configuration, applications and data 1 day? 15
17 Verify databases and applications 7 days 5,16
18 Customer verification/testing 1 day 17
19
20 DC2 Outage 19 days
21 Initial Notification of Customers 2 hrs
22 Declaration of Disaster 4 hrs
23 Reconfigure network environment 2 days
24 Configure, restore, and boot Netscaler 4 hrs 23
25 Backup existing alternate DC configurations 4 hrs 23
26 Reconfigure servers for alternate usage as required 2 days 25
27 Retrieve/restore backup files from VTL/MCNC tape backup 3 days 26
28 Restore/Verify Oracle file space and database allocation 4 hrs 27
29 Reconfigure servers to utilize resources at the alternate DC 3 days 28
30 Verify databases and applications 7 days 29
31 Customer verification/testing 1 day 30
-2 -1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
