Disaster Recovery Planning
Information Technology Services
December 11, 2012
Project Team:
Irene Larkin, Deputy City Auditor
Aaron Cook, Internal Auditor IV * IT Audit Stacey Linch, Internal Auditor
Project Number: 1130071
City Auditor Department
Bill Greene
City Auditor
City of Phoenix
City Auditor Department 17 S. 2nd Avenue, Suite 200 Phoenix, AZ 85003
This report can be made available in alternate format upon request.
Mission Statement
“To improve the quality of life in Phoenix
Recovery Point Objective (RPO) – the point in time to which work should be restored following a disruption or disaster (how much data is the business willing lose).
Recovery Time Objective (RTO) – the time by which a mission critical system must be restored (how long the business can operate without the system being available).
Disaster Recovery Plan
Executive Summary
PURPOSE
We reviewed the adequacy of disaster recovery services provided by Information Technology Services (ITS) for the most critical systems it supports. We also reviewed the overall framework for disaster recovery planning as provided by ITS.
BACKGROUND
Business continuity management (BCM) reduces an organization’s business risk that arises from an unexpected disruption to key services. As the City relies heavily on information systems, the City’s ability to recover those
systems is important in order to continue providing essential services. A disaster recovery plan (DRP) provides the basis for recovering essential IT services in the event of a disruption. A DRP addresses the strategy for the recovery of IT resources that meets the business units’ needs regarding recovery point
objective (RPO) and recovery time objective (RTO). Our review focused on the most critical systems that
ITS supports, including SAP, eCHRIS, Tax Mantra/eTax, Lotus Notes, KIVA, PensionGold, and phoenix.gov web pages.
RESULTS IN BRIEF
Overall, we found that the lack of current restoration plans, failure to conduct testing, and failure to provide training significantly increases the risk that if the City were to experience a disaster for systems supported by ITS, ITS would not be able to restore services in a timely manner.
Disaster recovery guidance, expectations, and requirements were not established for the City.
A.R. 1.94 Information Technology Governance Policy states that the Enterprise IT Architecture Team is “responsible for creating and maintaining the City of Phoenix enterprise IT Architecture, IT standards, and Citywide Standard Operating Procedures associated with a particular domain.” One of the domains is the Business Continuity & Disaster Recovery Domain. ITS has not established a DRP framework that provides departments guidance on how to design a DRP (business impact analysis, strategy development) as well as expectations regarding testing, maintenance, training, and communication of DRPs.
ITS did not establish Service Level Agreements for most critical business applications it supports.
A Service Level Agreement (SLA) outlines the responsibilities and roles of two parties – the IT service provider and the department responsible for the system. An SLA will identify the roles of ITS including system availability and performance, data storage and security, response times, and disaster recovery including RTO and RPO. The
responsibilities of the application owner could include user security and communication of issues. ITS and Finance have created an SLA for Tax Mantra, but SLAs regarding disaster recovery do not exist for the other systems supported by ITS.
ITS developed a Disaster Recovery Manual but has not centralized system restoration and requirement details that are necessary to ensure timely and successful restoration of systems in the event of a disruption.
ITS has developed a DR Manual that provides high level roles and responsibilities for restoring system services. The manual does not detail specific steps for restoring services or requirements for specific systems, but rather relies on these plans and other documentation to be developed and hosted by teams separately. We reviewed these documents with ITS staff to determine if the data was current. We found that in many instances, documentation does not exist and the vast majority of documentation that does exist is not current. This exposes applications to an increased risk that systems could not be restored in the event of a disaster.
Disaster recovery plans were not tested and staff were not trained on their roles and responsibilities.
The DR Manual states that disaster recovery plans should be tested at least annually. A portion of one system’s plan was tested in April 2011 and prior to that, the most recent test took place in 2008. Many of the mission critical applications have never been tested. ITS pays an alternate facility contractor for 10, 8-hour testing periods each year.
The DR Manual identifies eight different teams that are comprised of 41 employees with responsibilities for recovery/salvage, communications, backup site coordination, and other DR roles. These employees have not received training regarding their
responsibilities and duties and have not taken part in disaster recovery exercises. Lotus Notes was not included in the ITS disaster recovery contract for an
alternate restoration location and was not included in the ITS Disaster Recovery Manual.
Lotus Notes is a critical system for the City that ITS supports. The system does not have a documented recovery plan and there is no documented RTO and RPO. The City is moving towards replacing the current email system. As a new system is selected and implemented, the City must include a defined disaster recovery plan.
Department Responses to Recommendations
NOTE: This table will be completed after the responses are received by the department. The complete table will appear in the final audit report.
Rec. 1.1: Develop a disaster recovery policy or framework that provides guidance and
expectations for all City departments and systems. The framework should outline key requirements and steps such as business impact analysis, risk assessment,
documentation of RPO and RTO, strategy development, testing, maintenance, training, and communication of plans.
Response: Per A.R. 1.94, ITS will develop the documentation for the DR domain, including a framework to develop and implement disaster recovery best practices for departments.
Target Date: July 1, 2013
Rec. 1.2: Establish SLAs with the system owners of the applications in the DR Manual
as well as the email system owner. The Tax Mantra SLA can serve as a good template and should include agreed upon RTO and RPO.
Response: ITS will update the DR manual to identify and document appropriate applications which require DR plans. ITS will also
establish agreements and/or MOUs with system owners whose applications are in the DR manual.
Target Date: July 1, 2013
Rec. 2.1: Update the DR Manual including such information as flow charts and contact
information.
Response: ITS will review information such as flow charts and contact information and determine the most appropriate way in which to reference this information in the DR plan.
Target Date: July 1, 2013
Rec. 2.2: Communicate to all applicable staff the location of the DR Manual and
ensure that the manual is accessible by them. Additionally, ensure that all staff has a copy of the DR manual in hardcopy or portable electronic media. As staff store restoration plans on the SharePoint site (see Recommendation #6), ensure that responsible staff also maintain applicable restoration plans in hardcopy or portable electronic media format.
Response: ITS will grant access to applicable staff to the Living Disaster Recovery Planning System (LDRPS), which contains the updated DR plan. ITS will determine the best solution to ensure the plan is accessible to employees needing to respond to a disaster.
Target Date: Sept 30, 2013
Rec. 2.3: Update the system requirements and restoration plans for the following
applications: Active Directory, SAP, eChris, backup systems (e.g., CommVault)Tax Mantra, eTax, PensionGold, GIS, Cashiering for Windows, BRASS, KIVA, ePayments, phoenix.gov web pages, Intranet pages (Content Management System), and Remedy. Response: ITS will update appropriate documentation for
Rec. 2.4: Within the DR Manual establish restoration plan owners and describe how
those plans will be continuously maintained.
Response: ITS will develop a Standard Operating Procedure to define when and how DR manual documentation will be updated and maintained.
Target Date: Sept 30, 2013
Rec. 2.5: In addition to centrally storing the DR Manual on the SharePoint site,
centrally store all restoration plans for applications, databases, operating systems, and the network.
Response: The DR plan will be maintained in the Living Disaster Recovery Planning System (LDRPS), in the cloud by a third-party provider to ensure access at anywhere and anytime. ITS will also define a secondary, secure location to store the disaster recovery plan for access by appropriate staff.
Target Date: Sept 30, 2013
Rec. 2.6: Ensure a copy of the DR Manual as well as all restoration plans for
applications, databases, operating systems, and the network is kept at the offsite storage location at all times.
Response: See 2.5. Target Date:
Sept 30, 2013
Rec. 2.7: Perform regular testing of the disaster recovery plans as required by the DR
Manual. Ensure that all systems included in the DR Manual, as well as the email system, are tested.
Response: ITS will regularly test the disaster recovery plan as required by the DR manual.
Target Date: Sept 30, 2013
Rec. 2.8: Develop a training plan and ensure all staff involved in the DR process has
the appropriate training to fulfill their roles.
Response: ITS will develop a training plan and ensure appropriate staff are trained to fulfill their roles.
Target Date: Sept 30, 2013
Rec. 2.9: As the email system is replaced in the coming year, ensure the new system
has an appropriate disaster recovery plan to ensure that it can meet a defined RTO and RPO. If ITS continues to host the new email system, then it should be included in the ITS DR Manual; if hosting of the system is contracted out then the DR plan should be established with the vendor.
Response: ITS will ensure that the successful email replacement has an appropriate disaster recovery plan. The timing is contingent on the successful contract and Council approval for a replacement email system.
Target Date: July 1, 2014
Table of Contents
Executive Summary ... 1
Department Responses to Recommendations ... 4
Table of Contents ... 6
Scope, Methods & Standards ... 7
Detailed Observations by Major Scope Areas:
1 – Disaster Recovery Oversight and Management ... 8
Scope, Methods & Standards
Scope
Disaster recovery planning for enterprise systems supported by ITS: Active Directory, SAP, eChris, backup systems (e.g., CommVault), Tax Mantra/eTax, PensionGold, GIS, Cashiering for Windows, BRASS, KIVA, ePayments, phoenix.gov web pages, Intranet pages (Content Management System), Remedy, and Lotus Notes. ITS is responsible for providing disaster recovery services for many other applications that are not as critical, we did not review DRP practices for these other systems.
Methods
The following methods were used to complete this audit:
• Interviewed key staff and external vendors regarding DRP development, current
status, testing practices, training practices, and communication of the plan.
• Reviewed supporting documentation including the DR Manual, plans, restoration
procedures, system requirement documentation, and contracts.
• Visited the location where backup tapes are stored offsite to observe practices and
security.
• Identified applicable standards and regulations, including: o A.R. 1.94 – IT Governance
o Disaster Recovery Institute International (DRII) Professional Practices o Business Continuity Institute (BCI) Good Practice Guidelines
o Information Technology Infrastructure Library (ITIL) v3 Service Design o International Organization for Standardization (ISO) 27002 Chapter 14:
Business Continuity Management
We did not evaluate processes for backing up systems or managing backup tapes. We recently audited this function and our findings can be found in ITS Tape Management &
Control (1110084, issued June 2011).
Additionally, in June 2010, we issued the Disaster Recovery Planning Risk Assessment (1100076). While not making any recommendations, this risk assessment noted the general lack of guidance provided to City departments regarding disaster recovery planning and that the IT Governance Business Continuity / Disaster Recovery architect and working group provided an appropriate forum to address this weakness.
Standards
We conducted this audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
1 – Disaster Recovery Oversight and Management
BACKGROUND
Information Technology Services (ITS) provides support to city departments for various information systems. The most critical information services in terms of providing
essential services include the following:
Application Purpose
Active Directory
Provides file sharing and directory services for end users throughout the City except for Police and Municipal Court.
Backup Tape Management Systems
Manage the backup and recovery of databases, applications, operating system configurations, and network device configurations supported by ITS Static web pages
(phoenix.gov)
External Website for public access at phoenix.gov
TaxMantra / eTax Tax and licensing system
SAP City’s financial system
Cashiering for Windows (CFW)
Records payments by customers at pay stations for NOV (notice of violation), SAP payments, and Water/Solid waste payments
KIVA Development services permitting system
Geographic Information Systems (GIS)
The spatial data engine includes maps and locations of geographic data files
PensionGold Retirement system
BRASS Develop and process annual budgets
Content Management System (CMS)
Produce the Phoenix.gov Website and other internal Websites (Intranet)
eChris Payroll, benefits, employment application
ePayments Credit card payment administration for several City
functions most notably Water bill payments
Remedy Enterprise Help Desk problem management
application and ITS change management system
Lotus Notes Email system
These systems provide information to citizens, facilitate key City functions, and record the collection and disbursement of hundreds of millions of dollars annually. ITS is responsible for providing disaster recovery services for these systems. Industry
standards, such as the Disaster Recovery Institute International’s Professional Practices dictate common steps in the development of disaster recovery plans, including:
• Establish a policy to govern the development, maintenance, testing, and training of plans
• Determine business units’ RTO and RPO based on business impact analyses and risk assessments
• Design recovery strategies to meet RTO and RPO for each system, including documenting system requirements and restoration plans
• Exercise (or test) DRPs to identify weaknesses and areas of improvement • Provide training and awareness of the DRPs to key staff
We determined the presence and adequacy of the City’s IT Continuity Framework that provides guidance to departments regarding disaster recovery planning, including those systems not supported by ITS. Additionally, we evaluated the service level agreements ITS established with business units regarding disaster recovery services.
RESULTS
Disaster recovery guidance, expectations, and requirements were not established for the City.
A.R. 1.94 Information Technology Governance Policy states that the Enterprise IT Architecture Team is “responsible for creating and maintaining the City of Phoenix enterprise IT Architecture, IT standards, and Citywide Standard Operating Procedures associated with a particular domain.” One of the domains is the Business Continuity & Disaster Recovery Domain. We reviewed the City standards regarding BCM and DRP and found that ITS has not established standards that provide departments guidance on how to design a DRP (business impact analysis, strategy development) as well as expectations regarding testing, maintenance, training, and communication of DRPs. ITS has created standards for departments that choose to use the City's contract with the recovery site service provider; however, there is no guidance for those departments and applications not included in this contract. ITS does not provide disaster recovery services for many mission critical systems including those supported by the Police, Fire, Municipal Court, Law, Library, and Public Works departments. Without an IT Continuity Framework, City departments may be unaware of their responsibility to have disaster recovery plans in place for their business applications and the City cannot ensure consistent disaster recovery preparedness should disruptions occur.
ITS did not establish Service Level Agreements for most business applications in the Disaster Recovery Manual or Lotus Notes.
A Service Level Agreement (SLA) outlines the responsibilities and roles of two parties – the IT service provider (i.e., ITS) and the system owner (e.g., HR and Finance
departments are the system owners of eCHRIS). An SLA will identify the roles of ITS including system availability and performance, data storage and security, response times, and disaster recovery including RTO and RPO. The responsibilities of the application owner would be identified as user security and communication management of issues identified. We identified one SLA for the TaxMantra application that clearly identified the responsibilities for both ITS and the Finance and City Clerk departments (owners of the application). The agreement identified how issues would be reported, the expected response times, and the process for documenting the resolution. Without clearly identified SLA's ITS cannot ensure that its services align to the business needs of the City. This could lead system disruptions that negatively impact the City’s ability to continue providing services.
RECOMMENDATIONS
1.1 Develop a disaster recovery policy or framework that provides guidance and expectations for all City departments and systems. The framework should outline key requirements and steps such as business impact analysis, risk assessment, documentation of RPO and RTO, strategy development, testing, maintenance, training, and communication of plans.
1.2 Establish SLAs with the system owners of the applications in the DR Manual as well as the email system owner. The Tax Mantra SLA can serve as a good template and should include agreed upon RTO and RPO.
2 – Disaster Recovery Planning
BACKGROUND
A disaster recovery plan (DRP) provides guidance on how to restore a system in the event of disruption. The plan details roles, responsibilities, restoration plans, and
system requirements. The plan ensures that critical functions can be restored within the timeframe required by the business unit.
A key part of a DRP is having a place to re-locate and restore services in the event that the primary datacenter is unavailable. In July 2011, ITS contracted with a vendor to provide recovery processing services for all of the applications listed on Page 8 except Lotus Notes. The annual cost for this service is approximately $130,000 and includes allowing the City to use the vendor’s disaster recovery planning computer system for plan development and storage.
Disaster recovery plans must be kept current in order to be effective. As changes take place in systems, infrastructure, and staff, the plan must be updated and communicated to employees. Plans must be tested to ensure that systems can be restored to meet recovery point objectives (RPO) and recovery time objectives (RTO). Testing is what indicates the effectiveness of a plan, without testing there is no assurance that systems can be restored.
We reviewed the ITS DR Manual and supporting restoration plans for completeness and adequacy. We met with key staff to ensure that plans are documented, current, tested, and maintained. Additionally, we determined if staff were aware of DR plans and if they received appropriate training to fulfill their responsibilities.
RESULTS
ITS developed a Disaster Recovery Manual and generally kept it up-to-date. ITS has developed a DR Manual that provides high level roles and responsibilities for restoring system services. The manual does not detail specific steps for restoring services or requirements for specific systems, but rather relies on these plans and other documentation to be developed and hosted by teams separately. The manual is only applicable to systems with alternate recovery services provided by the third party service provider – those listed on Page 8 except for Lotus Notes. The manual contains the following items:
• DR goals and objectives
• Procedures for declaring a disaster
• Roles and responsibilities of recovery teams • Communication plans
• Application criticality and restoration priorities • Expectations regarding testing the DRP
The manual was last updated in June 2012. In general, we found the document
contained current information; however, we did note some outdated information such as references to the prior alternate recovery site provider and procedures related to travel and purchasing. Additionally, flow charts were not current due to recent changes in processes. This information was not updated in June 2012. Additionally, we noted that the DR Manual fails to include one significant enterprise system that ITS supports – the City email system.
ITS was not able to provide any documentation supporting how the disaster recovery strategies were developed and chosen. Typically, business impact analyses and risk assessments should be conducted with business units and IT staff to determine RTO and RPO and the appropriate cost versus benefit of achieving those objectives. The manual states that ITS anticipates being able to restore critical systems within 48 hours if the Information Technology Operations Center (ITOC) is still operational and 2 to 7 days if the ITOC is no longer available (blanket statement for all systems covered by the manual). The RPO (acceptable data loss) is not listed in the manual.
The ITS disaster recovery manual was not available to all disaster recovery team members.
The disaster recovery manual states that copies of the manual are distributed to team members as a hard copy and on a CD. ITS has changed this process and now has published the plan on the department’s SharePoint site. We interviewed various DR team members and found that not all members were aware the DR Manual was
available on-line. No team members we met with had a copy of the plan in hardcopy or on CD. Additionally, we found that not all members had access to the document via SharePoint. If team members do not have access to the document, they are not aware of their assignments and would not be able to fulfill them in a timely manner in the event of a disaster or disruption. Solely providing the manual via SharePoint is not sufficient to ensure that team members have the manual in the event of a disaster because a disaster may include network failure, meaning team members could not access SharePoint. The manual should be provided either in hardcopy or portable electronic media to each team member.
ITS sent the Disaster Recovery Manual to the offsite storage vendor; however, a copy of the manual and all restoration plans was not kept offsite at all times. The City uses a vendor to store all system backup tapes offsite. On a daily basis, a courier delivers a canister containing older backup tapes to the ITOC for City staff to swap the older tapes with the current tapes. The courier then takes the tapes back to the offsite storage vendor. At the site the canisters are kept locked and all authorized City staff that access the canisters on site are monitored. Access to the site is available 24 hours in the event of a disaster.
We noted that there are two copies of the DR Manual kept in the City of Phoenix canisters. However, the two copies are in the canisters that are used for daily
movement of backup tapes, meaning they are not at the offsite vendor for a significant portion of time each day. If a disaster were to occur during this time, City staff would not have ready access to the DR Manual. ITS should ensure a copy of the DR Manual is stored at the offsite storage vendor at all times. Additionally, only the high-level DR
Manual is stored offsite. This plan provides very little guidance about how to actually restore any particular mission critical system. The restoration plans and system
requirements for applications, databases, and operating systems also must be stored at the offsite location.
System requirements and restoration plans for ITS supported applications were not always available and were not always current.
Because the DR Manual is written at a high level, programmers (application level), database administrators (database level), operating system administrators (server level), and network administrators are expected to develop and maintain their own restoration plans. ITS currently does not store all plans in a central location. We obtained copies of the system requirements and the restoration plans from the ITS shared drive or from the appropriate administrator. We reviewed the documents with the application owners to determine if the data was current. We found that in many
instances, the documentation was not available, and that the vast majority of any existing documentation is not current. Not having current restoration plans and system requirements exposes applications to an increased risk that systems could not be restored in the event of a disaster:
Testing of the disaster recovery plan did not occur for some time and was not frequent enough to provide assurance that systems can be restored in a timely manner.
The DR Manual provides high level roles, responsibilities, and procedures including testing. It specifies the testing plan in four phases; technical, communication, business
System
Requirements on
file
On File Unknown or Not on FileSystem
Requirements
Current
Current Unknown or Not CurrentRestoration Plans
on file
On File Unknown or Not on FileRestoration Plans
Current
Current Unknown or Not Currentrecovery, and employee awareness. Additionally, the DR Manual states that plans should be tested at least annually. A portion of one system’s recovery plan was last tested in April 2011 (before the City changed recovery site vendors in July 2011). Prior to the April 2011 test, the most recent test took place in 2008. Many of the mission critical applications have never been tested. ITS pays an alternate facility contractor for 10, 8-hour testing periods each year. Not using this test time wastes the City’s expense of the contract. Additionally, without testing the plan there is no assurance plans would actually work and systems could be restored. Typically, plans are tested at least
annually and changes should be made to plans based on weaknesses or errors discovered during the testing.
Key employees did not receive training regarding disaster recovery duties and responsibilities.
The DR Manual identifies eight different teams with 41 employees that have responsibilities for recovery/salvage, communications, backup site coordination, restoration/relocation, business continuation, return from backup site, and
internal/external information. We interviewed ITS staff and found that training is not provided to team members. With the constant change in both business services and staff, it is essential that training be provided to ensure the team members have the necessary skills and information to implement the plan in the case of a disaster. Lotus Notes was not included in the ITS disaster recovery contract for an
alternate restoration location and was not included in the ITS Disaster Recovery Manual.
Lotus Notes is a critical system for the City that ITS supports. The system does not have a documented recovery plan and there is no documented RTO and RPO. If the ITOC were unavailable, while many other systems could be restored at an offsite location, Lotus Notes could not. Failure to take appropriate disaster recovery planning steps exposes the system to significant risks of not being able to come back online in the event of a disaster. The City is moving towards replacing the current email system. As a new system is selected and implemented, the City must include a defined disaster recovery plan.
RECOMMENDATIONS
2.1 Update the DR Manual including such information as flow charts and contact information.
2.2 Communicate to all applicable staff the location of the DR Manual and ensure that the manual is accessible by them. Additionally, ensure that all staff has a copy of the DR manual in hardcopy or portable electronic media. As staff store restoration plans on the SharePoint site (see Recommendation #6), ensure that responsible staff also maintain applicable restoration plans in hardcopy or portable electronic media format.
2.3 Update the system requirements and restoration plans for the following
Mantra, eTax, PensionGold, GIS, Cashiering for Windows, BRASS, KIVA, ePayments, phoenix.gov web pages, Intranet pages (Content Management System), and Remedy.
2.4 Within the DR Manual establish restoration plan owners and describe how those plans will be continuously maintained.
2.5 In addition to centrally storing the DR Manual on the SharePoint site, centrally store all restoration plans for applications, databases, operating systems, and the
network.
2.6 Ensure a copy of the DR Manual as well as all restoration plans for applications, databases, operating systems, and the network is kept at the offsite storage location at all times.
2.7 Perform regular testing of the disaster recovery plans as required by the DR Manual. Ensure that all systems included in the DR Manual, as well as the email system, are tested.
2.8 Develop a training plan and ensure all staff involved in the DR process has the appropriate training to fulfill their roles.
2.9 As the email system is replaced in the coming year, ensure the new system has an appropriate disaster recovery plan to ensure that it can meet a defined RTO and RPO. If ITS continues to host the new email system, then it should be included in the ITS DR Manual; if hosting of the system is contracted out then the DR plan should be established with the vendor.
