Report Number: 2014-MAS-04 Disaster Recovery Exercise
MANAGEMENT ADVISORY SERVICE REPORT
2014 Disaster Recovery Exercise
Date: September 8, 2014
Report Number: 2014-MAS-04 Disaster Recovery Exercise
Table of Contents:
Page
Executive Summary
Background
1
Objective and Scope
2
Results
2
Appendix
Distribution
4
Executive Summary
Report Number: 2014-MAS-04 Disaster Recovery Exercise 1
Background
The Office of the Internal Auditor (OIA) recently partnered with the Information Technology function (IT) to observe the Disaster Recovery Exercise which was conducted on May 17, 2014, and to review the IT Disaster Recovery Plan (DRP) and supporting documentation. The exercise was limited in scope and only included the Citizens Insurance Suite (f/k/a “CORE”), CDW, Cognos and supporting applications.
Several significant changes have occurred in the IT environment over the past year which supported OIA assisting IT by performing this engagement:
New Personnel - An IT staff member who was not previously involved with disaster
recovery has been assigned responsibility for managing the exercises, as well as for maintaining the DRP and supporting documentation.
New Applications - The exercise was focused on the Billing Center and Policy Center
modules of Citizens Insurance Suite (CORE) which have gone live since the last exercise.
New Technology - New hardware was installed in the DR Data Center in Tampa and new
software was implemented to automate the transfer of IT operations from the Production Data Center in Jacksonville and back.
The Enterprise Risk Management (ERM) Business Continuity group provides the framework and administers the Citizens Business Continuity Program (BCP) which is guided by the principles and standards of The Disaster Recovery Institute International (DRii), the Disaster Recovery Journal (DRJ), and the Business Continuity Institute (BCI). The BCP is a comprehensive and proactive program focused on maintaining time-sensitive business functions during an outage so that Citizens’ customers continue to receive quality products and services with minimal disruption. The BCP includes both the business unit(s) recovery capability (referred to as Business Continuity or BC) and the information technology (IT) recovery capability (referred to as Disaster Recovery or DR).
Citizens’ BCP is based upon Business Impact Analyses (BIAs) which were performed in 2012. A BIA evaluates and prioritizes business processes by assessing “the potential quantitative (financial) and qualitative (non-financial) impact that could occur if any business function was unable to operate for a period of time for any reason”. The BIAs help to reveal business process and supporting IT system interdependencies, and to determine the Recovery Time Objectives (RTOs) for the processes and systems. Using the BIAs, IT maps systems and operations to business processes and develops recovery strategies to meet the required RTO’s. The IT Operations Department is responsible for the DR program and associated test exercises.
The Citizens IT infrastructure has been built so that in response to a pending disaster or in the event of the loss of the Production Data Center in Jacksonville, the delivery of critical business applications and supportive technology services can be transferred to the DR Data Center in Tampa. It is important to note that the IT DRP does not include all systems and services and is approved as such. Business Units will be required to continue processes without the support of technology in some cases. Determining an “acceptable level” of business continuance is the
Report Number: 2014-MAS-04 Disaster Recovery Exercise 2 responsibility of the Executive Leadership Team (ELT) based on recommendations from the ERM BC team.
Citizens has also developed a Catastrophe (CAT) Plan to provide scalability for handling the increased volume of claims in the event of one or more storms or other weather events affecting Florida. The CAT Plan and testing exercise are not directly related to the BCP or DRP.
Objective and Scope
The objective of the review was to evaluate the adequacy and effectiveness of the processes and controls that comprise disaster recovery planning, documentation and test execution.
The focus of the review included:
Observation of the disaster recovery exercise and determination if it is executed in accordance with the DRP.
Determination if the DRP and disaster recovery exercise execution aligns with updated business impact analyses and recovery objectives and whether the DRP is adequate to successfully support the business in the event of a disaster.
Evaluation of the DRP and supporting documentation with consideration of relevant standards, best practices and the ERM BCP Manual.
Results
We noted that the planning meetings and preparations leading up to the disaster recovery exercise were comprehensive and that there was excellent communication and teamwork during the exercise. We also observed that the execution of DR exercises, the annual testing of all applications and systems, and the content of the DRP and supporting documentation should be ameliorated.
These observations include:
Disaster Recovery Exercise - The Emergency Response Team (ERT) members
responsible for overall Incident Management and Operations, and for Incident Management and Staff Coordination monitored the exercise and participated in issue resolution via the conference call bridge. It may be beneficial for one or both of them to be on-site during the exercise to provide a presence consistent with the importance of the exercise and to be able to observe activities first-hand. In light of recent and on-going staff departures, additional training, cross-training and knowledge transfer for remaining staff should be considered. To the greatest extent possible, disaster recovery exercises should be representative of circumstances that would exist during a real disaster with respect to availability of systems, means of communication, network resources and so forth.
Annual Testing of All Systems and Applications - An annual exercise of all DR capable
applications should be performed so that recovery personnel stay familiar with recovery procedures and that any changes in the IT environment over the past year are included.
Executive Summary
Report Number: 2014-MAS-04 Disaster Recovery Exercise 3
IT EDRP (Enterprise Disaster Recovery Plan) and EDRP Supplemental Information
- It would be beneficial to ensure that the EDRP and ERRP Supplemental Information are reviewed and updated on a regular basis and after major changes to the IT environment. The externally hosted Sustainable Planner application which is used by ERM could be leveraged to store IT disaster recovery documentation in an external location which would be accessible from anywhere via the Internet in the event of a real disaster.
In addition, we noted the following process improvement opportunities related to the storage of the DRP and supporting documentation, as well as a potential impact on the IT DR capability as a result of the upcoming relocation of IT personnel from Tallahassee to Jacksonville:
Organization of Disaster Recovery Documentation on the Network Shared Drive -
The EDRP and EDRP Supplemental Information which are stored on the network should be stored in a shared folder which is readily identifiable, well known, and not easily confused with any other network folder.
Relocation of IT Personnel to Jacksonville - The potential impact on the disaster
recovery capability resulting from the relocation of IT personnel from Tallahassee and Tampa to Jacksonville should be assessed. Consideration should be given to sending disaster recovery personnel to a distant location in advance of an approaching hurricane. Management has agreed with our observations and provided action plans.
We would like to thank management and staff for their cooperation and professional courtesy throughout the course of this review.
Report Number: 2014-MAS-04 Disaster Recovery Exercise 4
Distribution
Addressees: Robert Sellers, V.P. - IT Infrastructure and Operations
Copies: Juan Cocuy, Citizens Audit Committee Chairman
Bette Brown, Citizens Audit Committee Member Jim Henderson, Citizens Audit Committee Member Barry Gilway, President/CEO/Executive Director Kelly Booten, Chief - Systems and Operations Curt Overpeck, Chief Information Officer
Christine Turner Ashburn, V.P. - Communications, Legislative and External Affairs
Debby Kearney, Ethics and Compliance Officer Bruce Meeks, Inspector General
Carol Williams, Director, Enterprise Risk Management Johnson Lambert, LLP (External Auditors)
Following Audit Committee Distribution The Honorable Rick Scott, Governor
The Honorable Jeff Atwater, Chief Financial Officer The Honorable Pam Bondi, Attorney General
The Honorable Adam Putnam, Commissioner of Agriculture The Honorable Don Gaetz, President of the Senate
The Honorable Will Weatherford, Speaker of the House of Representatives
MAS Performed By
Auditor in Charge Gary Sharrock Audit Director Karen Wittlinger
Under the Direction of
Joe Martins
Chief of Internal Audit
