Business Continuity Planning
and
Execution
The Teachers’ Retirement System
of the City of New York
and
Hurricane Sandy
Presented by: Patricia Reilly
Deputy Executive Director Teachers’ Retirement System of the City of New York
Overview
•
What We Planned
•
What Happened
•
What Worked / What Didn’t Work
•
Lessons Learned
TRS’ Experience on 9/11
•
Temporary Office Space (cold site): Shared 50 Seats
•
TRS Declares Disaster
•
Cold Site Could Not Accommodate TRS
Lesson Learned: Need
Guaranteed
Disaster Recovery Site,
NOT
Shared Site
New Mandate from Executive Director:
Develop New Business Continuity Plan
Onsite
Production Data Center Mirrored Offsite
TRS Business Continuity Plan Components
Dedicated Mobile Office Secure Storage Facility for
Critical Equipment & Supplies
Documented & Rehearsed Continuity Plan
Prior to Hurricane
•
In the month before the Hurricane, TRS Lost 3 Senior Staff
•
2 Senior Staff on Vacation
•
Negotiations with Vendor for High-Volume Printing
•
Check Printer from Iron Mountain in TRS Office
•
TRS Email Functionality Not Redundant
•
City Evacuated Zone A and Executed Controlled
Citywide Transit Shutdown
Benefits Admin Deputy Director Member Services Director
CIO
•
3 ½ Feet of Water in Lobby
•
6 Levels Flooded Below Ground
•
Building Infrastructure Services Compromised
• Power, Telecom, Water, Sewage, HVAC• 71 Elevators in Building, Inoperable
•
Some Damage to Walk-In Center Entrance
•
No Damage to 16
thor 17
thFloor Administrative
Offices
•
We Have No Access to Building
55 Water Street
After the Hurricane
New York Metropolitan Area
After the Hurricane
– Tragic Loss of Life
– More than 1.5M Lose Power in Tri-state – Gas Shortages
– Damage to Homes, Loss of Landlines, Internet – Public Transit Severely Limited
– Cell Towers Powered by Battery Backup/Service
Disruption
•
TRS Staff
– At Home Dealing with Their Own Problems – Displaced
•
TRS Members
– May have Lost Their Home – Need Loans and
M 10/29 T 10/30 W 10/31 Th 11/1 F 11/2 Sa 11/3 Su 11/4
HURRICANE HITS
•Disaster Declared/Arranged for Delivery of Mobile Office
•Member Website & Phone Message Updated
•Daily Departmental Conference Calls Initiated
•NYC Employee Retirement System Provides Office Space in Queens
•Office of the Actuary Provides Space
•Board of Education Retirement System Provides Space for Walk-In Center
•Start Setup at Queens Facility for Call Center
•HR Accesses Employee Payroll via Space at Other City Agencies •TRS IT Performs Mobile Office Preparations -Workstations -Printers -Fax -Communications
• Still No Dial Tone at
Queens Location
•No Power and Gas
•Team Conference Calls Begin
•Call Tree Triggered – Outreach to Staff
•Employee Website & Emergency Phone Message Updated
•Mobile Office Arrives in
Manhasset, Long Island
•Phones Are Down at Queens Location
What Happened – First Week
NO COMMUNICATION WITH 55 WATER ST.
NO TRS EMAIL – Personal Email Addresses Shared CONTINUING SIGNIFICANT TRANSIT ISSUES
SNOWSTORM
M 11/5 T 11/6 W 11/7 Th 11/8 F 11/9 Sa 11/10 Su 11/11
What Happened – Second Week
•Mobile Office Site Up & Running •Chairman of the TRS Board, Mel
Aaronson, Personally Visits Staff in Trailer
•Only 4 Dedicated Call Center Phones
Available at Trailer
•Website Configured for Member
Inquiries via Email
•1st Executive Team Face-to-Face
Meeting
•Phones Finally Working at
Queens Location
•Setup for Call Center Complete
•Still no Access to Computer
System at Queens Location
•Additional Call Center Staff
Deployed to Queens Location
•Member Data Requests
Relayed to Mobile Office – Research Results Relayed Back
•Retirements Unit Starts
Working from Actuary’s Office
Lack of Check Printer
EFT Campaign
Non-Critical Staff Assignments
•Walk-In Center Opens
in BERS Offices
•Word on the Street:
LONG Absence from 55 Water
•Expansion Planning at
Queens Location: Equipment &
Business (Continuity) as Usual
Mobile Office • Retirement Payroll • A/R • General Accounting • Loans • Member Accounting •TDA •Death Benefits •Member Service •IT Operations/Support •Help DeskOffice of the Actuary Retirements/Withdrawals BERS Walk-In Center Work at Home Non-Critical Staff Queens Site •Call Center •All Other Units
November 2012
•
Access Granted to 55 Water to Retrieve
Essential Items
•
Board and Investment Meetings Held at
NYCERS Offices
•
IRS Relaxes Hardship Withdrawal Rules for
Hurricane Sandy Victims
•
Cold Weather Begins to Take Hold
December 2012
•
Business Continuity Fatigue Sets In
•
Non-critical Business Processes Become
Critical as Time Progresses
•
Backlogs Generate Increasing Call Volume
•
Building Sends Re-Occupy Notice
•
Business Resumption Planning Begins
•
Fire in the Building – Re-Occupation Delayed
•
It’s Getting Colder!
Business Resumption
Working to Ensure Adequate Building Services
Working to Assure Safety of Premises for Staff
What Worked
What Didn’t Work
• Documented & Rehearsed
Business Continuity Procedures • Communication • With Staff • With Members • Mobile Office • Retrieving Continuity
Materials from Iron Mountain
• Technical Setup for Working
from Business Continuity Sites
• Non-Critical Staff Not
Aware of Business Continuity Plan
• Communication
• With 55 Water St.
• With NYC Government • Mobile Office Alone
• Incomplete Set of
Continuity Materials at Iron Mountain
• Technical Setup for Working
What Worked
What Didn’t Work
• In-Process System Upgrades
(Email, Data Tools)
• Mobile Office Mail Room • Size
• 2 Months’ Mail Volume • Lack of Generic Business
Resumption Planning Checklist
• Commute was a Hardship • No Pre-Disaster Exit Plan
• System Redundancy • Redirecting Incoming
Mail to Mobile Office
• Reactive Business
Resumption Planning
• Staff Able to Get to
Business Continuity Locations
