Computer systems failure contingency planning

(1)

Computer systems failure

ContingenCy planning

second edition

Cihan Cobanoglu, ph.D., stephen Blasik,

and mark g. Haley, CHtp

88127NEI02ENGE

a technology primer Developed by the

american Hotel & lodging association’s technology and e-Business Committee

funded by the

american Hotel & lodging educational foundation

(2)

(3)

This guide was written by Mark Haley, revised and updated by Cihan Cobanoglu, Ph.D., Stephen Blasik, and Mark Haley and developed by the Technology and E-Business Committee of the American Hotel & Lodging Association (AH&LA), with a grant from the American Hotel & Lodging Educational Foundation (AH&LEF).

technology and e-Business Committee

Co-Chair Daniel Connolly, Ph.D., Associate Professor

University of Denver

Co-Chair Clay Dickinson, Executive Vice President

Jones Lang LaSalle Hotels Vice Chair Cihan Cobanoglu, Ph.D.

University of Delaware

AH&LA Officer Liaison David Kong, President & CEO

Best Western International

AH&LA Officer Liaison Pam Inman, EVP & COO

American Hotel & Lodging Association AH&LA Staff Liaison Laurence Barron, Vice President & CIO

American Hotel & Lodging Association

members

James O. Abrams ISHAE Representative President & CEO

California Hotel & Lodging Association

Laurence Barron Vice President & CIO

American Hotel & Lodging Educational Institute

Stephen Barth President & Founder

HospitalityLawyer.com

Kathleen Pearl Brewer, Ph.D. Professor/Director of Graduate Studies

University of Nevada at Las Vegas

Dennis Carpenter, MHS Director, Association Alliances

Heartland Payment Systems

Cihan Cobanoglu, Ph.D., CHTP Associate Professor

University of Delaware—HRIM

Patton Conner

Vice President, Information Resources

Marriott International

Daniel Connolly, Ph.D. Associate Professor

University of Denver

Clay B. Dickinson Executive Vice President, Strategic Advisory

Jones Lang LaSalle Hotels

Gregory A. Dugal ISHAE Representative Executive Director

Maine Innkeepers Association

Rich Ehlers Director

iBAHN

Bernard Ellis

Managing Director—Americas

IDeaS Revenue Optimization

William Folkerts Owner

Comfort Inn

Mark Haley, CHTP Partner

(4)

Pam Inman, IOM, CAE Executive Vice President & Chief Operating Officer

American Hotel & Lodging Association

David Kong President & CEO

Best Western International

Monika Nerger

Vice President of Technology, The Americas

Mandarin Oriental Hotel Group

Kirby D. Payne, CHA President

HVS/American Hospitality Management Company

Laura Powers, MHS

Director of Business Development

Teledex Corporation

Tina Reese

Regional Vice President

First Data Commercial Services— Revenue Sharing Alliances

Jeffrey Senior

Executive VP, Marketing & Sales

Fairmont Raffles Hotels International

Jules Sieburgh, CHTP

O’Neal Consultants

Kevin H. Smith EVP & General Manager

(5)

acknowledgements

American Hotel & Lodging Educational Fondation’s Research and Project Funding Committee

editorial reviewers

Pearl Brewer, Ph.D., UNLV Kutay Kalkan, University of Delaware

Disclaimer

This guide is intended only as a general guide concerning Hotel Computer Systems Contingency Planning and does not purport to be, nor should it be used as, a complete description of Hotel Computer Systems Infrastructure problems or solutions. Companies should not rely upon this guide for other than general information and should consult their employees and attorneys before implementing any suggestions or procedure or using ant forms contained in this guide. AH&LA does not warrant the accuracy of the guide, the accuracy of completeness of the procedure described in this guide, the effectiveness of such procedures or the effect of any forms contained herein.

(6)

(7)

general oVerVieW

purpose of this Document

The AH&LA Technology Commit-tee commissioned this document, an informative guide identifying essential topic areas assisting hotels in preparing and planning for brief, short and long-term failures of their vital computer systems, especially associated with the Property Man-agement System (PMS), Guestroom Locking System, Telephone System (also called Private Branch Exchange, PBX) and Point of Sale Systems (POS). Today, the majority of hotels are entirely dependent on these systems in order to deliver essential guest

services, and relatively few of today’s hotel employees have experience in working in a non-computerized hotel. Therefore, the Committee identified a need for a basic-level manual on how to prepare for and plan around a system failure.

Computer Failure Contingency Planning is actually a sub-set of a broader discipline, Business Continuity Planning. Should the AH&LA membership find value in this document, a future publication may address the broader scope of Business Continuity Planning, which would encompass maintain-ing and restormaintain-ing operations in the event of natural and manmade disasters (such as a hurricane or terrorist attack).

Why Computer failure Contingency planning is

important

The scope of this manual mainly focuses on computer system failure in relation to the lodging industry, but limited to providing basic operating functions. The Contingency Plan includes multiple strategies, actions and procedures to resume business operations and functions associated with computer failure. Having a well-developed plan in place is essential for brief, short and long-term system failures. Similar to other important deci-sions in the hotel industry, priorities are focused on guest services, revenue and costs.

If the hotel does not know which rooms are vacant and clean during a short-term failure of the PMS, then they will not be able to check guests in or service walk-ins. The lack of a plan thus decreases guest service levels and reduces revenue. Without a plan in place, if there is a long-term failure

“The survey addressed the level of corporate readiness for disaster recovery, con-cluding that only 12 per-cent of enterprises have an effective disaster recovery plan. Fully 83 percent of companies were ineffectively prepared for a technology disaster.”

(10)

of the POS, then one might have to close the restaurant. The lack of a plan decreases revenues and guest service, not to mention putting the F&B staff out of work, directly impacting employee satisfaction.

A long-term failure of the PMS, such as a complete disk drive failure without adequate data backups, can easily put a hotel out of business by losing all of the in-house guest receivables, all of the accounts receivable, future reservations and future availability. With a tested plan in place, the hotel should be able to survive these risks sufficiently if not gracefully. Without a plan, the hotel simply wouldn’t know who owed how much, who had a reservation at what rate and whether or not they could take another reservation.

This document discusses various elements and processes that provide guid-ance for IT systems restoration during a system disruptions or disasters, enabling the hotel to establish a coherent and effective contingency plan. These plans will serve the hotels as Users’ Manuals that effectively facilitate the recovery of IT systems, operations and data after a disruption, major or otherwise.

(11)

general strategies in Computer

failure ContingenCy planning

Every hotel and hotel company have unique circumstances according to their location, flag if any, range of services offered, systems installed and so on. Therefore, Contingency Planning must be done at the property level, specific to the needs of the property. While recognizing this fact, there are key strategies common to almost any Contingency Planning effort. This section reviews these general strategies, separated into the following key phases:

• Prevention and Minimization of failures • Preparation for possible failures

• Responding to actual failures • Recovery from failures

prevention & minimization

Time and effort spent on Prevention, Minimization and Preparation greatly mitigates the impact of a computer failure on guest services, hotel operations and hotel profitability, and accelerates the recovery process significantly. These conditions are true for almost any variety or duration of computer system failure.

Physically Secure Data Centers

Although in many hotels the “Data Center” is a closet behind the Front Desk, the prudent hotelier will recognize that it is still a Data Center and needs to be treated as such. Physically securing the room(s) containing critical systems can prevent and minimize a wide range of potential causes of failure. Data centers should be locked, accessible only from the back of the house and not have signage indicating the function of the room. Data centers should be windowless and not on exterior walls. Data Center should be located above grade and not have major plumbing directly above them to minimize water intrusion in case of a flood or plumbing failure. Data Centers should have independently controlled HVAC service. Larger Data Centers should have two sources of cooling, not just house HVAC. As discussed below, Data Centers and the equipment in them should have protected electrical supplies.

Protected Electrical Supplies

Many system outages revolve around transient or longer-term failures of clean, reliable electricity. Protecting computer systems with Uninterruptible Power Supplies (UPS), diesel generators and surge protectors will often go a long way to avoiding system failures altogether or make responding to them less problematic.

Recovery Prevention and Minimization

Responding Preparation

(12)

Management must verify that all key systems are on UPS-protected power, including at least one terminal and printer on the PMS. If the electricity fails, the purpose of the UPS is to prepare for an orderly shut down and then to shut the systems down. Preparing for an orderly shut down includes running your Contingency Reports and notifying affected departments. Computer equipment not on a UPS should be turned off before the power comes back up.

If the hotel has a diesel generator in addition to the UPS, it probably is not engineered to support the entire building’s load, so a shutdown is in order even then.

preparation

Contingency Plan & Testing

Clearly, creating a Contingency Plan as described in this document is an essential part of preparing for a pos-sible system failure of any severity. Likewise, testing the plan so that staff and management know how to imple-ment it becomes vital to ensuring that the team knows how to implement it and recover from the failure.

Hardware and Software Inventory

The Plan should have an inventory of all of key hardware and software used in the hotel. While serial numbers of desktop PCs are not important to a restore effort, detailed information

about all servers can be essential to have at hand. A key piece of inventory information often overlooked, record the make and model of all back-up devices and software. You will need this information if you need to send your back-up media to a remote site for recovery.

Alternative Locations for Contingent Operations

Many Contingency Plans call for moving operations to an alternate loca-tion if the use of the hotel or its Data Center is compromised due to an emergency condition such as fire, flood or terrorist attack. For many hotels, the alternate location will simply be a corporate office or sister property outside the affected area where a server can be installed and data tapes restored to it.

A more elaborate approach, outside the scope of this document, is to contract with a service provider for “hot site” services, where they promise to make

“When creating your backup plan, take into account the workflow of the organization, department, or group that you’re protecting. Ask your-self: am I protecting the right information at the right time? If at any point of failure can cause a workflow bottleneck, make sure that failure can be overcome quickly. If not, the costs of downtime will be multiplied by the number of people sitting around twid-dling their thumbs.”

(13)

available server capacity and workspace for you to move your operations to. Few hotels choose this costly type of alternate location solution.

react

Reliable Current Data

Although we cannot overemphasize the value of reliable back-ups, in all but the most-sophisticated operations the back-up becomes out-dated in hours or minutes after it is taken and is not immediately available. This time lag establishes the need for reliable and more recent current data to know the current state of the house in time to deliver effective guest service.

To use the PMS as an example, most PMSs have a group of reports that together can maintain operations. Some PMSs have a dedicated Contingency Reports menu; others allow the hotel to configure a job stream of selected reports that can be run as a batch.

(Figure 1, screenshot of Reports Menu with “Down-time Reports” selected from OnQ. Image courtesy of Hilton Hotels Corporation)

In reference to Figure 1, some PMSs generate Contingency Reports auto-matically according to a schedule defined by the hotel. These reports can be sent to print or saved as files on the PMS or another server. If the hotel is saving the files rather than printing them, then utilize another server to store them. If the reason you need the Contingency Reports is because the PMS server’s disk drive failed, then you can’t access them unless they are on another device. Of course, ensure that you have protected power to retrieve and then print the files in case the root problem is an electrical failure.

(14)

(Figure 2, screenshot of configurable schedule for Contingency Reports. Image courtesy of Northwind.)

These reports will contain, at a minimum:

• A list of all guests in-house, checked-out or expected today with balance, rate, group affiliation, special services and billing instruc-tions, sorted in room number order. This report is used to maintain balances as charges are incurred during the outage.

• A list of all guests in-house, checked-out or expected today with group affiliation, comments and typically part of the address field, sorted in alphabetical order. This report is used to find out what room a given guest is in.

• A list of every room in the hotel with the current occupancy status: vacant and clean, vacant and dirty, occupied, out of order, #days available. This report is used to assign rooms to new arrivals and create housekeeping assignments

In reference to Figures 1 and 2, the hotel may refer to these reports as “Contingency Reports”, “Back-Up Lists” or “Downtime Reports”. Most PMS vendors advise hotels to run current contingency reports hourly or as business activity dictates. This task is generally assigned to whoever is responsible for the Front Desk during each shift.

(15)

We also recommend that the hotel run on an at least daily basis the fol-lowing reports:

• All expected arrivals for next five days

• A room availability report for the next 90 days or longer, depend-ing on the property’s seasonality

• A blocked room report for the same period as above This task is generally assigned to the Reservations staff.

Increased Staffing Requirements

Manual operations always take longer and require more personnel to deliver acceptable service levels than when we have access to the automated tools we use every day to run hotels. Hotel managers should expect to call in additional staff during any extended period of downtime. Night Audit, Housekeeping room assignment, check-in, check-out and controlling room inventory all take longer and will require additional personnel during the system outage.

Many of these human resources can be drawn from other departments during the emergency, especially Accounting.

Processes for Contingency Operations

Each hotel must have well-considered processes in place for dealing with the failure of key systems. These processes will vary according to the nature and expected duration of the failure. Brief, short and long term failures call for somewhat different approaches. For these purposes, define a brief outage as four hours or less, short-term as less than two night audit cycles and long-term as lasting two night audit cycles or more.

A computer system failure affecting only a single system (a disk drive fail-ure, for example) requires a different response than one affecting the entire hotel (a fire, extended electricity outage or other disaster, for example). Most system failures are fortunately of shorter duration and affect a single system only. Many hotels will determine that some transactions are not worth capturing manually during a system failure. Telephone calls repre-sent a class of transactions that a hotel will sometimes choose to give away during a system failure. Note that for transactions involving revenue-share partners (such as In-Room Entertainment, typically), the partner will still expect to be paid even if your PMS was down, so the cost implications of ignoring those transactions is much greater than merely foregoing the hotel’s commission.

Another means of simplifying transactions and transaction accounting during a system failure is to simply alter the services offered. If the restau-rant POS fails completely and will be down for several days, then a logical management decision would be to switch to buffet-only service with even-amount pricing, tax and tip included.

(16)

Interface failures represent another class of situation to consider. Both the PMS and PBX may be functioning normally, but if the interface between the two is down, then contingency procedures must be in place to enable out-bound telephone calls from newly-checked in rooms. If the interface between tPMS and In-Room Entertainment system fails, then contingency procedures must call for enabling Pay-Per-View movie purchases in checked-in rooms and also for posting of movie purchases to guest folios.

Guest communications represent another set of processes. For a brief PMS outage, periodically send a hotel manager into the lobby to explain to guests that the Front Desk system is temporarily down, we are operating normally but in a manual mode and although check-ins may take a little longer, we will get everyone checked in. Longer outages will call for a let-ter from the General Manager delivered to each occupied guest room and posted at the Front Desk.

The specifics of these processes will be unique to each hotel’s business circumstances and systems installed, but must be documented and staff trained on in order to ensure that guest services are of the desired level. The contingency operations processes suggested below in this document are based on the concept of using the Contingency Reports from the PMS before the system failed and manually maintained control sheets for vari-ous functions to maintain services during the system failure. The control sheets are essential to recovering to normal operation after the system is restored.

recovery processes

Running the hotel and delivering essential guest services is not sufficient. Contingency Planning must also include how to use the Contingency Operations processes to bring the system up-to-date with activities that occurred while the affected system(s) were down. Typically, when system services are restored, the data will be current as of the moment when the system went down or as of the most recent data backup. In either event, the Contingency Reports from before the system failed and the various control logs created during the system failure must be entered into the system in order to return to normal operations.

System Backed

Up to Tape System Fails

Contingency Operations Using Contingency Reports and Logs

System Up; Back Up Tapes Restored

Data from Contingency Reports & Control Logs Input

(17)

applicable Case studies

Hemphill & Wendland1_{present two examples highlighting the importance}

of contingency planning. One example, the World Trade Center bombing of 1993 was a terrorist attack. The other, Hurricane Iniki in 1992, illustrates the need for contingency operations due to natural disaster. They also summarize a study of 320 computer failures conducted by Comdisco, categorizing the cause of each:

Hardware Problems 24% Power Outage 16% Hurricane 16% Flood 15% Miscellaneous 12% Fire/Explosion 7% Bomb 5% Earthquake 5%

World Trade Center Bombing 1993

“A case in point involved the bombing of New York’s World Trade Center in 1993. The terrorist act inflicted physical damage to a large area of downtown Manhattan, including an adjacent hotel. Although the hotel was evacuated in less than fifteen minutes with no fatali-ties, the telephone and computer system was damaged. The hotel could not account for guests and employees. Communications fail-ures prevented management from communicating with emergency services. The loss of heat could have caused further damage to existing systems from the bursting of frozen water pipes. The hotel’s experience points to one of the key elements in a technology disaster recovery plan—maintaining critical operational and financial data at an off-site storage facility. Following a disaster, off-site facilities that maintain reservation and accounting information would be available immediately for retrieval of in-house guest records, financial docu-ments, guest history and sales information, minimizing a disaster’s effect on a hotel’s data and information.”

1 Hempell, Christian E. and Wendland, Nicole R. (1999). Disaster Recovery in Hospitality—The Risks in Computerization and Information Management. Retrieved February 3, 2009, from Hotel Online Web site: http://www.hotel-online.com/Trends/Andersen/1999_Disaster-Recovery.html

(18)

Hurricane Iniki, 1992

“One of the most destructive events for hospitality companies occurred in September 1992 as Hurricane Iniki unleashed winds up to 200 miles per hour on the Island of Kauai, Hawaii. The damage amounted to $1.6 billion and resulted in 85 percent of the island’s hotel rooms shutting down for at least two months. With 30-foot waves and complete power outages, all hotel technology systems on the island were lost. Particularly in areas with potential for destructive weather, the need for disaster recovery and off-site back-up systems cannot be overstated. Hotel properties without technology disaster recovery plans may be closed for much longer periods of time than other companies.”

(19)

ContingenCy team ContaCts

identification of members

The Contingency Plan needs to identify the members of the Contingency Team. The Plan document should list them all by name, position and role in the response effort. Key roles include:

• Team Leader—Responsible for implementing the overall plan and recovery process. Typically the highest-ranking hotel executive available, such as the General Manager or Manager on Duty. • IT Leader—Responsible for restoring service from the affected

system(s). Typically an IT manager in a larger hotel; likely an out-sourced IT resource or key operator in a smaller property.

• Department Heads—Responsible for maintaining service levels in affected operating departments during the outage and restoring system data afterwards. Typically Front Office management in a PMS, PBX or locking system failure and Food & Beverage manage-ment in a POS failure. Accounting personnel have key functions in restoring the data maintained manually during the outage to their proper systems for revenue recognition.

The Contingency Team will be a lengthy list of managers in a large or com-plex hotel or resort. The list must be maintained as people come and go, and must include multiple forms of contact information, including home and cellular telephone numbers, pagers, etc. All members of the team must be aware of their roles in responding to a failure. Hotels operated by a manage-ment company or a brand will need to include corporate contacts on their Contingency Team list, especially the Central Reservations Office. Please see table 1 and table 2 in Appendix

Vendor support Contacts

The Contingency Plan should contain a list of key vendors and suppliers of technology products that will be of value in a failure. This list should include contact information for telephone support for all major vendors of critical systems, especially PMS, POS, PBX, etc.

See sample table 3 in Appendix.

support agreement terms

Most hotels have support agreements with the vendors of their mission-critical systems such as PMS, POS, Sales & Catering, etc. The standard agreements initially proposed by vendors typically impose little obligation on the vendors for response to emergency situations, such as a system out-age. The standard support agreement will invariably exclude any vendor

(20)

responsibility for contingent liabilities, protecting the vendor from paying for losses consequential to the system failure. These losses might include lost room revenue due to inability to provide guest services, lost revenue from a failed posting interface, loss of goodwill and so on.

Hotel companies should negotiate terms in their support agreement that require specific obligations to support the hotel in a system failure, for whatever reason. Typical terms will include:

• Definition of varying severity levels of system failures

• Maximum response times for telephone support for each severity level

• Defined Escalation Path for elevating the ownership of problem resolution within the vendor organization, going higher up as the problem severity or duration increases

• Minimum requirements for back-up devices and practices and perhaps power protection

Vendors will reasonably negotiate to a responsibility to help you with the repair of the system failure. Many support services, especially those required due to a root cause not related to the vendor’s software (flood or power prob-lem, for example) may be billable services. They will not normally assume any responsibility for the recovery process, such as inputting data.

escalation path

A Contingency Plan must have a clearly defined Escalation Path to resolve an issue. The Escalation Path should provide a method to promptly resolve a difficulty. The Team Leader role, responsible for overall resolution and service delivery during the outage is a pivotal figure in escalating decision-making. The figure2_{on the following page is a structured model outlining}

the responsibilities and tasks that are needed to constitute a final decision made by an individual.

(21)

Manager Overview Agreement? Agreement? Consult among Contingency Team Resolution Attained Executive Decision Identify Difficulty

(22)

program anD Data BaCK-up

Explaining the importance and need for reliable and tested back-ups of all programs and data are beyond the scope of this document. Suffice to say here that many system failures that call for the implementation of the Contingency Plan may require restoring data from back-up media. The General Manager of the hotel needs to assure herself that:

• Programs and data are backed up reliably according to a docu-mented practice accepted by the involved vendors, brands and management companies

• At least two recent generations of back-up (i.e., last night and last week) are stored in a fireproof cabinet or safe outside of the hotel • Rotate only the oldest tapes back into the hotel, so that there is

always one copy off-site

• Back-ups are periodically verified

• Someone in the hotel knows how to restore data from back-up media reliably, at least with vendor assistance

• Restores are tested periodically to ensure that everything works and staff actually knows how to conduct a restore

locations of Back-up tapes

Management has a primary responsibility to ensure that all data and pro-grams necessary to restore operations after a failure are properly backed-up. Once the data and programs are backed up, at least two generations of the back-up media should be stored outside the hotel, “off-site.”

Considerations in the selection of the off-site storage location include: • Security—Is your media secured from theft or loss in the off-site

location?

• Cost—How much will it cost to store and access your back-up media?

• Location—Is the off-site storage location close enough that you can readily access the data when required?

• Environment—Is the storage location temperature and humidity controlled

One tactic some hotels have used is a reciprocal off-site storage agreement with another hotel in the area, where each stores the other hotel’s media. Others store the tapes in a secure fire-proof located in the hotel, but in a separate wing or other location relatively remote from the Data Center. The recommended solution is to use an off-site records management service to store and transport tapes.

(23)

Another approach is to back up across the Internet to a hosted back-up service, making the location of the back-up media a non-issue. Most experts would recommend using this approach only for incremental data back-ups, with multiple generations of program back-ups available on physi-cally accessible media. There are a number of such service providers, with specialized software to facilitate the back-up and restore processes. These hosted service providers will typically utilize a Tier 1 co-location facility in a city remote from the hotel, which offers a major benefit to the hotel: If the cause of the outage is a major regional natural disaster (earthquake, for example), then having the data in another city is valuable.

Back-up software and Hardware

Tape drives remain the most common and effective approach to program and data back-up. Other options include optical media (CD/DVD) and Internet back-up services as described above. Regarding Internet back-up services, recognize that the speed of the download capability will govern the duration of a system restore: it could take several hours to restore the required back-up files.

Also note the use of RAID (“Redundant Array of Inexpensive Disks”) arrays as an intermediate approach to data protection. Applicable in vari-ous levels, RAID arrays distribute the data and programs across multiple disks such that if one fails, the server can simply keep going without loss of service. Some RAID implementations can also offer performance benefits. Although RAID arrays can provide a significant layer of fault-tolerance, the only rational recommendation is add them to a daily back-up strategy, not in place of.

Many software providers will specify the use of particular back-up software, especially PMS vendors. This practice enables the software vendor to assist in the restore process if needed. If you are using a back-up program that the vendor does not know, then you are on your own for a restore. Support agreements often specify the use of a specific back-up application.

(24)

DoCumentation

Contingency plan Documentation

The Plan should be recorded in written form, with hard-copy printed ver-sions readily available to hotel management. Soft-copy verver-sions should also be available stored on a computer. A sample outline for a Contingency Plan document is in the Appendix. The Plan must contain enough infor-mation in enough detail so that current hotel management can maintain services as needed and that third parties can assist in the replacement of failed equipment.

Ownership and maintenance of the Contingency Plan rests with the General Manager of the property. The Plan should be reviewed at least annually and updated with personnel and system changes.

essentials to Quality Documentation

Good documentation is essential to an effective Contingency Plan. This means access to vendor-supplied documentation on utilizing the various programs and devices (such as application programs like the PMS and utility programs like your back-up software, hardware and media) but also data unique to your hotel.

Include hard and soft copies of key parameter tables from major systems in the Plan. This would include the room number table, room type table and revenue account list from the PMS and menu table from the POS. In a worst-case scenario where no data backup is available at all, these records could at least provide a starting point for recovery.

Like the Plan, this documentation should be stored in hard copy as well as soft copy files. If there is no electricity, the soft copy files stored on a computer will not be very useful.

locations of Vendor Documentation on system

operation

Management should designate a particular location inside the hotel, to store reference copies of any necessary vendor documentation, particularly the back-up program and hardware, as well as any application software.

pms interfaces

A hotel may need to implement contingency operations because an inter-face between the PMS and another system is down. Both systems may be operating normally, but without the interface, the operation needs to adapt. Also, in the event of a PMS failure, the hotel will need to adopt contingency operations for the systems dependent on interfaces to the PMS.

(25)

point-of-sale system

Most POS systems support a number of settlement types (cash, Visa/ Mastercard, room charge, etc). Some or all of these settlement types will be interfaced to the PMS. Hotel POS screens should be configured with a settlement method that allows checks to be closed without going through the interface. The specifics will vary with the POS and its configuration. F&B personnel should use this settlement method when the interface or the PMS itself is down.

F&B personnel must physically carry room charges or phone them in to the Front Desk for manual posting during contingency operations. Doing this promptly is essential during breakfasts, when many guests rush to check out immediately after breakfast. Many POS systems have a non-interfaced settlement method to close checks out to when the PMS is down.

in-room entertainment interface

If the In-Room Entertainment (IRE) system interface is down, you will need to enable movie viewing for each room, much like the PBX status. Likewise, movie postings must be done manually, typically during night audit from a report generated by the IRE system.

Some IRE systems will buffer charges until they sense the PMS interface is up. Then they will attempt to post movie charges to the PMS. This will often create discrepancies if the guest that watched the movie is not the one that was checked into the room in the system as of the time the PMS came back up.

Call accounting system

If the Call Accounting System (CAS) interface fails, the hotel may choose to manually post all telephone charges, only high-value calls or to ignore them. Most CASs offer an option to print all calls as they are made or all calls since a given time. Using either method, designate someone to post the calls. Some PMS interfaces add taxes and surcharges to the amount reported by the CAS. If your systems are configured in that manner, then advise the posting clerk what markup to increase the amount posted by.

As with the IRE systems, many CASs will buffer call charges until it senses the PMS is back up.

pBX/Voice mail interface

If the interface(s) between the PBX and/or Voice Mail system and the PMS fail, then as guests check-in to the hotel, the PBX operators must be advised to enable outbound calling for all guests that have established credit with

(26)

the hotel. This process may need to be repeated for some Voice Mail systems depending on the specific configuration.

Most hotels will not bother to turn outbound calling off after a check out, but if you checked guests into the Voice Mail system, you must check them out in order to cancel their mailbox before the next guest checks in to that room.

When the PBX and PMS detect that the interface is back up, they will initi-ate a “handshake” to synchronize the current room status correctly. Most interface configurations assume that the PMS is the authoritative source of data and PMS status will overwrite PBX status.

Today’s PBX systems do not fail very often. They are very reliable and usually feature built-in redundancy. When they do fail, it is likely to be a partial failure that is readily resolved by replacing a board. If a PBX com-pletely fails or there is a fire in the telephone room, the only real recourse is to work with your vendor to get a new system installed as quickly as possible. A hotel that needs a new PBX on an emergency basis will likely be forced to close until telephone service is restored due to life safety and liability concerns, a good argument for maintaining PBXs under mainte-nance agreements.

other servers & peripherals

Guestroom Locking Systems

The basic approach to contingency operations for most guestroom locking systems is to maintain an inventory of contingency key cards, often called “fail-safe keys” by some lock system vendors. Make these keys in advance, label them clearly and store in a very secure place accessible only to the Manager on Duty, typically a safe deposit box at the Front Desk with the key on the MOD ring.

Most locking systems support multiple levels of keys at any one time. The fail-safe level keys should work without re-phasing the lock if used in the proper order. When normal locking system operations are restored, the nor-mal guest level keys should also function without re-phasing required. Making contingency keys is tedious and time-consuming. Most manufac-turers recommend maintaining three complete sets of contingency keys, each to be used in sequence. Therefore, contingency keys must be labeled with the room number and sequence. Color-coded stickers may be of value to establish the sequence.

(27)

ContingenCy operations

major system failures

The discussions below begin with managing through a PMS outages of varying durations, the goes on to discuss failures of several other critical systems.

The following section describes suggested contingency operations for brief, short-term and long-term system failures. Hoteliers often cannot identify that a given situation will turn out to be a brief problem (< four hours), a short-term failure or a long-term failure (two or more night audit cycles). What appears a minor problem creating a brief outage can often evolve into a longer-term issue. Other problems are obviously longer-term, such as a fire in the computer room.

Management’s first task is to assess the severity of the situation and estimate the duration of the problem. For an outage caused by some external factor (power failure, natural disaster, terrorist attack, for example), the obvious priorities are safety of guests and staff first, security of data back-ups second and salvaging hardware and physical assets third. Note that for many hotels during Hurricane Katrina the contingency operations consisted largely of clearing guests and staff from the hotel, securing the building and evacuat-ing with tapes and servers to a safe location.

The following sections discuss basic operational practices for managing through failures of interfaces and several major operational systems such as the PMS or POS. Hotels may find these practices useful to incorporate into their Contingency Plan documents and Plan tests.

pms Brief outage (less than four Hours estimated)

If the duration of the failure is estimated to be very brief, less than four hours:

• Alert the management team and affected departments

◦ Implement contingency operations for interface failures as discussed above, such as checking guests into PBX, settling POS charges to non-interfaced methods, etc.

• Secure all older copies of the contingency reports so they cannot be confused with the most recent reports

• Make photocopies of the most recent contingency reports if that is possible and distribute, especially to telephone operators and housekeeping

• Give a copy(ies) of the guest list by name to the telephone opera-tors

(28)

• Distribute radios to key personnel, especially if telephone service is affected.

◦ If not enough radios are available, designate individuals to hand-carry communications between the Rack Clerk, House-keeping and PBX

• Get the crash kit(s) in case the contents are needed.

• Centralize the room inventory function with a single individual located at or near the Front Desk, essentially a “rack clerk” from when we used wire-frame room racks to control room inventory. This person will be responsible for assigning rooms for arrivals and coordinating room availability with Housekeeping.

◦ Update the Contingency Reports with all arrivals, departures, etc that have occurred since they were last run

▪ Registration cards and cashier copies of folios will be the best sources of this data

◦ Coordinate with Housekeeping to maintain the current status of room availability as rooms are cleaned, checked-out or checked in

◦ If availability for the day is tight, immediately get an accurate House Count and maintain it during the course of the outage ▪ As arrivals and checkouts are processed manually, cross

them off the Contingency Reports so that the Contingency Reports reflect the current status of the house

▪ Record all transactions related to arrivals and departures chronologically on a separate control sheet (see Appendix for samples)

▪ This document, and similar ones, will greatly facilitate the recovery process

◦ As guests check-in, advise PBX to enable outbound calls • Centralize the revenue posting function with a single individual

located at or near the Front Desk, essentially a “posting clerk.” ◦ Utilize posting slips to record the charges and file them in folio

buckets

◦ Separately record all room charges on a posting control sheet chronologically to facilitate recovery

◦ Separately record all checkout transactions onto a control sheet to facilitate recovery

• If there is a Central Reservations Office, refer reservations inquiries to them

(29)

◦ If the hotel’s PMS is tightly integrated to the Central Reser-vations System, the CRS may be able to provide availability and expected arrivals reports to the hotel during the system failure

▪ This capability becomes very valuable in extended system outages

short-term pms outage (more than four Hours,

less than two night audits)

If it appears that the failure may extend beyond four hours but, not more than one night audit, then implement the above procedures and also:

• If there is electricity, write a letter from the General Manager to all guests and have it delivered to all occupied rooms and posted at the Front Desk

• Have staff come in early and plan to stay late, especially Night Audit, Front Office Supervisory and Housekeeping Supervisory personnel

• Transcribe beginning balances from Contingency Reports and all room charges to hard copy guest folios

◦ If you use pre-printed folio forms normally, use those for con-tingency operations. If not, create in advance and maintain a supply of folio forms with key fields (name, address, room, rate, dates, billing, group, clerk, etc) identified in Crash Kits ◦ Include Rate, Group, Billing Information and Address from

the Contingency Reports onto the folio to facilitate checkouts ◦ File by room number in a bucket

• Prepare hard copy Registration Cards for expected arrivals • Create a Rack Sheet from the Contingency Reports and control

sheets used during brief outage contingency operations

◦ If there is electricity, do it in an Excel worksheet to facilitate changing statuses as time goes on

▪ One row per room, with Status, Guest Name, Arrival/ Departure Dates

◦ Maintain constant communications with Housekeeping on room status

▪ Reconcile Room Availability with Housekeeping at least twice daily during the outage, morning and afternoon • Prepare an updated Telephone List with arrivals and departures

as necessary and give to operators • Prepare for a manual Night Audit

(30)

◦ In practice, most operators will forgo Night Audit reporting during a system failure and simply post room and tax manu-ally to the paper folios established for each room

◦ Manually create Housekeeping Assignment sheets during the Night Audit so they are ready when the housekeepers arrive in the morning

• As checkouts occur, settle the balances manually and record each transaction on a separate control sheet for each cashier that will become their cashier’s report

◦ Each cashier should advise the rack clerk of what rooms have checked out periodically

▪ The rack clerk should update Housekeeping and PBX as checkouts occur

◦ Advise guests that the system is down and that a copy of the folio will be sent to them after the system problem is resolved

▪ Be certain to capture addresses, email or other, in order to honor this commitment to the guest

▪ Expect that some guests will demand a copy of their folio and make photocopies if possible

◦ Each cashier needs to balance their cash and credit card receipts to their cashier’s report at the end of their shift

◦ Drop cash receipts as per the hotel’s established practices

long-term pms outage (two or more night audits)

If it appears that the outage will go on for more than one night audit cycle, then implement the above procedures and also:

• Establish a room revenue control sheet in Excel to capture room revenue as it is posted by hand to the hard copy folios

◦ Use this sheet, plus the room charge control sheets to generate a basic revenue report

• Secure current availability and expected arrivals reports from the CRO if integrated

• Store all control sheets (Arrival/Departure; Posting; Cashier Control; Room Revenue) in an accordion file by day to facilitate recovery processes

pms recovery processes and testing

The essence of recovering from an extended PMS outage is to bring the system current for each day, run a night audit in the system and to repeat

(31)

as necessary. This process will usually create discrepancies between what was manually posted and collected and what the system says later. Interfaces constitute a major source of these discrepancies: many interfaced systems, especially call accounting systems, will buffer charges until they sense the PMS is there. Then they will dump all of the charges into the PMS, even though the guest in the system on the PMS may have checked out several days earlier. Most hotels will elect to adjust off unverifiable revenue of this sort and move on. Substantial discrepancies need to be resolved, and may require contacting the guest to explain why the charge posted to the credit card is not the same as the amount cited at checkout.

If the system was restored from a back-up tape, then the first step in recov-ery is to bring the system current with the Contingency Reports used when contingency operations first began. Then utilize the chronological control sheets for Arrivals and Departures, Cashiering and Posting to bring the system current day by day. Maintaining those control sheets in an orderly manner is what makes recovery possible. Watch for duplicate or erroneous postings created by interface buffer postings, as these will not be shown on the Posting control sheets.

These processes represent an often-overlooked area in Contingency Plan-ning. If a system has been down for a period of time, then when it comes back up, the hotel needs defined processes for getting the data in the system current with the actual state of the house based on transactions performed manually while the system was down.

In a brief outage, the system will probably come back up with data current as of the moment it went down, so only manual transactions conducted during the failure need to be re-input. Because the contingency plan pro-cesses included steps for tracking all of these transactions, this activity is relatively straightforward.

In an extended outage, the system may be brought up with only the data as of the previous night’s data back-up or even older. This scenario means that all transactions that occurred between the back-up and the implementa-tion of the Contingency Plan need to be recovered from somewhere (typi-cally other systems such as the POS and IRE system and the Contingency Reports)

testing

If the hotel does not test and practice their Contingency Plan, they won’t know if it works or how to use it. A planned shutdown is an ideal time to exercise the Contingency Plan.

(32)

point-of-sale system

A failure of a hotel’s Point-of-Sale system requires reverting to manual pro-cesses and controls. Controlling Food & Beverage product and controlling cash are the key elements to maintaining operations without a POS. Restaurant checks with serial numbers and carbonless duplicates are essen-tial to controlling product. Keep a supply of these checks in crash kits and more in a storeroom.

When contingency operations are triggered:

• Issue a small supply of checks to servers, recording the serial num-bers given to each

• Remind kitchen personnel to make meals only from the soft copies of the duplicate checks

• Use cashiers rather than allowing servers to self-cashier under contingency operations in order to improve cash control

• Recognize that service will be slower without a POS and adjust both staffing and guest expectations accordingly

• Have cashiers maintain a manual control sheet (cashier’s report) for all closed checks by meal period or shift

◦ Also record on the hard copy of the checks the settlement method

• After each shift, reconcile the used checks, kitchen dupes and unused checks against those issued to each server to ensure that checks are not being “lost” instead of turned over to the cashier with the cash

• Beverage outlets should implement “red-lining”, where bartenders must draw a line under each drinks order as they make it

• Bartenders may record immediate sales over the bar onto a single control sheet rather than a check for each transaction

◦ Alternatively, instruct bartenders to do a “blind drop” of their cash. Not recommended from a control perspective

Cas—Call accounting system

Call Accounting Systems are no longer as important as they used to be to hotels. Most travelers prefer to use cell phones for their calls, leading to sharply decreased telephone revenue. During a brief or short-term failure, most hotels will simply forego the telephone revenue. In a longer-term failure, the hotel may wish to capture some of the revenue, particularly international calls.

(33)

Some ways this can be done include:

• Offer guests a flat-rated bundle of unlimited calling on a per-day basis until the system is replaced

• Ask the PBX maintenance vendor to attach a line printer to the serial SMDR port on the PBX

◦ As calls are made, the SMDR (Station Message Detail Record) data gets printed out showing the number called, originating extension, time and duration

◦ Create simple rules of thumb consistent with your standard telephone pricing and post manually

(34)

loCation anD Contents of

ContingenCy supplies

“Crash Kits”

Contingency operations frequently require paper forms and supplies that are often no longer generally used in many hotels. Examples might include registration card/folio “buckets”, mechanical credit card imprinters, 10-key adding machines, miscellaneous charge slips, duplicate copy restaurant guest checks with serial numbers, flashlights and glows sticks, etc. Such forms and supplies called for in a hotel’s Contingency Plan should be stored in a secured, consolidated location where the designated manager can readily access them when needed, without needing to hunt through a storage room. Many hotels maintain “crash kits” containing these supplies and hard copies of the Contingency Plan document in a single easy-to-find container. Please refer to the Appendix of this report to further develop an understanding of a Crash Kit.

importance of testing

Testing is essential to ensuring that the Contingency Plan is complete and that staff know how to utilize it.

Periodically testing the plan will identify problems in the plan and will deter-mine key areas that need additional attention during the recovery process.

“Almost all hospitality companies have one main database for res-ervations and guest history. Both are at risk if a disaster occurs. In addition, the industry’s reliance on technology and communication creates several less traditional risks. The global distribution system, the importance of customer data, productivity heavily reliant on terminals and information, and the long-term effect on loyalty of a single negative experience, are additional risks particularly present in the hospitality industry. In addition, companies are racing to build information about customer and spending patterns, investing mil-lions of dollars in systems to institutionalize this knowledge. Loss of these systems and databases is analogous to organizational memory’ loss. While rebuilding physical property may take several years, guest loyalty and behavioral history’ may often require more time and money.”

(35)

appenDiX

section a: Crash Kit Contents (Contingency supplies)

A.1 General Information

• Hard copy of Hotel Contingency Plan

• Bucket(s) for folios and registration cards (if not used at Front Desk)

◦ Alternatively, accordion files with a slot for each room or floor

A.2 Credit Card Supplies

• Mechanical Imprinter(s)

• Standalone authorization terminal

• Credit card vouchers, American Express, Visa/ MasterCard, Din-ers, others as required

A.3 Forms

• Pre-printed miscellaneous charge slips • Restaurant checks

◦ Serial numbered ◦ Duplicate soft copies • Guest folio stock

• Registration card stock • Blank control sheets

◦ Arrival/ Departure ◦ Posting

◦ Front Office Cashiers Report ◦ F&B Cashiers Report ◦ Room Revenue Report

A.4 Extra Office Supplies

• Erasers (ample amounts needed) • Pencils

• Accordion Files

◦ For storing control sheets by day

• 10-key adding machines (several; battery powered) • Staplers

(36)

• Paper clips • Tape

A.5 Miscellaneous Items

• Flashlights, battery-operated lanterns & glow sticks

◦ Whenever possible store batteries sealed and not installed in flashlights or other devices.

• Camera: capture pictures of damaged inventory and surrounding environment for legal purposes.

section B: sample Control sheet formats

Note: Create these forms in advance and maintain them in “Crash Kits” as well in soft copy.

B.1 Local Emergency Contacts

Type Name Phone Extension/

Notes Police

Fire Dept. EMS

Poison Control

B.2 Names, Titles, Roles, Contact Information

Name Title Role

during a failure

Office

phone # phone #Home Pager/cell phone #

(37)

B.3 Vendor Support Agreements

Type of Supply/ Equipment

Company

Name Contact Phone Notes

B.4 Arrival/Departure Control Sheet

Action (Arr/Dep/Move)

Room Name #

GSTS

Time Clerk Notify

HSK?

Notify PBx?

Arrival 1234 Henderson 2 14:20 CJJ  

Move 2333 James 2 14:40 CJJ

B.5 Posting Control Sheet

Room Name Outlet Tkt # Amt Time Clerk Post to

Folio?

2218 Kapioltas Dining Rm 123456 $44.87 19:50 MGH  810 Weadock Gift Shop 445566 $12.82 19:55 ABC Y; Pd out _{to GS} B.6 Room Revenue Control Sheet

Room Name # GSTS Group/ Pkg Room Rate Pkg Elements Tax 1 Tax 2 301 Shaw 01 Rack $0 $4.93 $2.96 301 Sorenson 01 Rack $0 $4.93 $2.96 302 303 Bollenbach 02 Golf $200.00 $19.95 $0 TOTALS 03 04 $200.00 $29.81 $5.92

(38)

B.7 Checkout Control Sheet—Front Office Cashier’s Report

Room Name Settle

Type AMT Comments Notify Rack?

1822 Nassetta AX $344.19 Not happy, but understood; _{Be sure to mail folio}  2113 Bainum VM $187.46 No movie on folio, should be _{one; reconcile later}

Summary

Total Cash Total Paid Out Net Cash Total Ax Total VM

B.8 F&B Cashier’s Report

Check

# Server Food $ Bev $ Tax $ Tip $ Total $ Settlement

123456 Mary $38.90 $15.00 $2.70 $11.00 $67.60 Room 1636, _Brian 123457 Julie $25.00 $0 $1.25 $5.00 $31.25 AX, Silverman B.9 Room Rack Sheet

Room Type Orig

Stat New Time New Time New Time

301 NK VC OC 14:40

(39)

WorKs CiteD

Hempell, Christian E. and Wendland, Nicole R. (1999).

Disaster Recovery in Hospitality- The Risks in Computerization and Information Management.

Retrieved February 3, 2009, from Hotel Online Web site:

www.hotel-online.com/Trends/Andersen/1999_DisasterRecovery.html Cougias, Dorian J. (2003).

The Backup Book Disaster Recovery from Desktop to Data Center. Lecanto, FL: Schaser-Vartan Books

Collins, Galen R. and Cobanoglu, Cihan. (2008).

Hospitality Information Technology Learning How to Use IT. Dubuque, Iowa: Kendall/Hun

Computer systems failure contingency planning