DATA SECURITY MANAGEMENT
O
PERATIONAL
F
ORENSICS
Michael Corby
I N S I D E
Definition; Related Business Requirements; Justification Options; Basics of Operational Forensics; Building the Operational Forensics Program; Linking Operational Forensics to Criminal Investigations;
Linking Operational Forensics to Business Continuity Planning
DEFINITION
The increased complexities of computer systems today make it difficult to determine what happened when a malfunction occurs or a system crashes. It is sometimes difficult to even make the basic identification of whether the cause was accidental or intentional. If the cause was inten-tional, legal action may be in order; if the cause was operainten-tional, the rea-son must be identified and corrected. Both require a planned and measured response.
Unfortunately, with today’s emphasis on immediate recovery in the networked environment, and with the obligation to get back online as quickly as possible, determining the cause may be impossible. The ten-dency to restart, or reboot, may remove information that could be valu-able in ascertaining cause or providing evidence of criminal wrongdoing.
Operational forensics is a two-phased approach to resolving this problem. The first phase is the proper collection of operational informa-tion such as data logs, system
moni-t o r i n g , a n d e v i d e n c e moni-t r a c k i n g methods. The appropriate attention to this phase makes it much easier to identify the problem in the second phase, the recovery.
At recovery time, the information at hand can be used to decide whether a formal intrusion investiga-tion needs to be initiated and evi-d e n c e c o l l e c t e evi-d n e e evi-d s t o b e preserved. By responding in pscribed ways, which can include re-p a i r o r r e re-p l a c e m e n t o f t h e
P A Y O F F I D E A
The increased complexities of today’s computer systems make it difficult to determine what hap-pened when a malfunction occurs or a system crashes. Sometimes, it is difficult to even make the basic identification of whether the cause was acci-dental or intentional. If the cause was intentional, legal action may be in order; if the cause was oper-ational, the reason must be identified and correct-ed. Both require a planned and measured re-sponse. Operational forensics is a two-phased approach to resolving this problem. This involves the proper collection of operational information such as data logs, system monitoring, and evi-dence tracking methods; and the recovery.
D A T A S E C U R I T Y M A N A G E M E N T
equipment, correction of a software weakness, or identifying the human-caused errors that resulted in the disruption, the system can be returned to operation with a much reduced probability of the same event occur-ring in the future.
RELATED BUSINESS REQUIREMENTS
Technology has been more than an efficiency enhancement to the orga-nization. It has become the lifeblood of the successful enterprise and the sole product of the networked application service provider. As such, the maximum availability of this essential resource is critical. When a failure occurs or the system is not operating at expected levels, proper proce-dures should be used to accurately identify and correct the situation. Fail-ing to do so will result in unpredictable operations, inefficiencies and possibly lost revenue, tarnished image, and failure to thrive. The busi-ness case for investing in the time, procedures, and the relatively small cost of computer hardware or software components seems clear.
Why, then, do companies not have operational forensics (or the same functions by other names) programs in place? Well, for two reasons: peo-ple have started with the assumption that computers are perfectly reli-able and therefore will only fail under rare circumstances if programs are well-written. Why waste resources in pointing the finger at something that should never occur. Second, the topic of methodical, procedural in-vestigations is new to other than law enforcement, and only recently has come into the foreground with the advent of computer crimes, cyberter-rorism, and the relationship of vengeance and violence linked to some computer chat rooms, e-mail, and personal private data intrusions.
The good news is that operational forensics is not an expensive op-tion. There is some additional cost needed to properly equip the systems and the process for secure log creation, but unless the need is deter-mined for a full-scale criminal investigation and trial preparation, the pro-cess is almost transparent to most operations.
The business objectives of implementing an operational forensics pro-gram are threefold:
1. Maintain maximum system availability (99.999 percent or five-nines “uptime”).
2. Quickly restore system operations without losing information related to the interruption.
3. Preserve all information that may be needed as evidence, in an ac-ceptable legal form, should court action be warranted.
The acceptable legal form is what calls for the operational forensics pro-cess to be rigorously controlled through standard methods and a coordi-nated effort by areas outside the traditional IT organization.
JUSTIFICATION OPTIONS
The frequent reaction to a request to start an operational forensics pro-gram is one of financial concerns. Many stories abound of how forensic investigations of computer crimes have required hundreds or thousands of hours of highly paid investigators pouring over disk drives with a fine-toothed comb. All of this while the business operation is at a standstill. These stories probably have indeed occurred, but the reason they were so disruptive, took so long, or cost so much is because the operational data or evidence had to be reconstructed. Often, this reconstruction pro-cess is difficult and may be effectively challenged in a legal case if not prepared perfectly.
Operational forensics programs can be justified using the age-old 80-20 rule: an investigation cost is 80 percent comprised of recreating lost data and 20 percent actually investigating. An effective operational foren-sics program nearly eliminates the 80 percent data recreation cost.
A second way in which operational forensics programs have been jus-tified is as a positive closed-loop feedback system for making sure the in-vestment in IT is effectively utilized. It is wise inin-vestment planning and prudent loss reduction. For example, an operational forensics program can quickly and easily determine that the cause of a server crashing fre-quently is due to an unstable power source, not an improperly config-ured operating system. A power problem can be resolved for a few hundred dollars, where the re-installation of a new operating system with all options can take several days of expensive staff time, and actually solve nothing.
No matter how the program is justified, organizations are beginning to think about the investment in technology and the huge emphasis on con-tinuous availability, and finding ways to convince management that a plan for identifying and investigating causes of system problems is a worthwhile endeavor.
BASICS OF OPERATIONAL FORENSICS
Operational forensics includes developing procedures and communicat-ing methods of response so that all flexibility to recover more data or make legal or strategic decisions is preserved. Briefly stated, all the pro-cedures in the world and all the smart investigators that can be found cannot reverse the course of events once they have been put into action. If the Ctrl-Alt-Delete sequence has been started, data lost in that action is difficult and expensive, if not impossible to recover. Operational foren-sics, therefore, starts with a state of mind. That state of mind prescribes a “think before reacting” mentality. The following are the basic compo-nents of the preparation process that accompany that mentality.
D A T A S E C U R I T Y M A N A G E M E N T
• definition of the process to prioritize the three key actions when an event occurs:
– evidence retention – system recovery – cause identification
• guidelines that provide assistance in identifying whether an intrusion has occurred and if it was intentional
• methods for developing cost-effective investigative methods and re-covery solutions
• maintenance of a secure, provable evidentiary chain of custody For situations where legal action is warranted:
• identification of or development of professionally trained forensic specialists, and interviewers or interrogators, as needed
• procedures for coordination and referral of unauthorized intrusions and activity to law enforcement and prosecution, as necessary • guidelines to assist in ongoing communication with legal
representa-tives, prosecutors, and law enforcement, as necessary • instructions for providing testimony, as needed
Notice that the evidence is collected and maintained in a form suitable for use in cases where legal action is possible, even if the event is purely an operational failure. That way, if after the research begins, it is deter-mined that what was thought initially to be operational, turns out to war-rant legal action, all the evidence is available.
Consider the following scenario. A Web server has stopped function-ing, and upon initial determination, evidence shows that the building had a power outage and when the server re-booted upon restoration, a diskette was left in the drive from a previous software installation. Initial response actions include purchasing a new UPS (uninterruptable power supply) capable of keeping the server functioning for a longer time, and changing the boot sequence so that a diskette in the drive will not pre-vent system recovery. All set? Everybody thinks so … until a few days af-ter the recovery, when someone discovered that new operating parameters have taken effect, allowing an intruder to install a “trap door” into the operating system. That change would take effect only after the system rebooted. Is the data still available to identify how the trap door was installed, whether it posed problems prior to this event, and who is responsible for this act of vandalism?
An operational forensics program is designed to identify the risk of changes to the system operation when it is rebooted and to conduct baseline quality control, but also to preserve the evidence in a suitable place and manner so that a future investigation can begin if new facts are uncovered.
BUILDING THE OPERATIONAL FORENSICS PROGRAM Policy
To start building an operational forensics program, the first key element, as in many other technical programs, includes defining a policy. Success in developing this process must be established at the top levels of the or-ganization. Therefore a policy endorsed by senior management must be written and distributed to the entire organization. This policy both in-forms and guides.
It informs everyone that the organization has corporate endorsement to use appropriate methods to ensure long-term operational stability, and thus ensure that the means to accurately identify and correct problems will be used. It should also inform the organization that methods will be used to take legal action against those who attempt to corrupt, invade, or misuse the technology put in place to accomplish the organization’s mis-sion. There is a subtle hint here meant to discourage employees who may be tempted to use the system for questionable purposes (harassing, threatening, or illegal correspondence and actions): that the organization has the means and intent to prosecute violators.
It guides in that it describes what to do, under what circumstances, and how to evaluate the results. With this policy, the staff responsible for operating the system components, including mainframes, servers, and even workstations, as well as all other peripherals, will have a definition of the process to prioritize the three key actions when an event occurs:
• evidence retention • system recovery • cause identification
In general, this policy defines a priority used for establishing irrefut-able data that identifies the cause of an interruption. That priority is to first ensure that the evidence is retained; then recover the system opera-tion; and finally, as time and talent permit, identify the cause.
Guidelines
As a supplement to these policies, guidelines can be developed that pro-vide assistance in identifying whether an intrusion has occurred and if it was intentional. As with all guidelines, this is not a specific set of deftive rules, but is a checklist of things to consider when conducting an ini-tial response. More detailed guidelines are also provided in the form of a reminder checklist of the process used to secure a site for proper evi-dence retention. The suggested method for publishing this guideline is to post it on the wall near a server, firewall, or other critical component. Items on this reminder checklist can be constructed to fit the specific in-stallation, but typical entries can include:
D A T A S E C U R I T Y M A N A G E M E N T
Before rebooting this server:
1. Take a photograph of the screen (call Ext xxxx for camera). 2. Verify that the keyboard/monitor switches are set correctly. 3. Record the condition of any lights/indicators.
4. Use the procedure entitled “Disabling the disk mirror.”
Accompanying these posted instructions is a series of checklists designed to help record and control the information that can be collected through-out the data collection process.
Log Procedures
Policies and guidelines can help provide people with the motivation and method to act thoughtfully and properly when responding to an event, but they are insufficient by themselves to provide all that is needed. Most operating system components and access software (modem drivers, LAN traffic, Internet access software, etc.) provide for log files to be created when the connection is used, changed, or when errors occur. The catch is that these logs are usually not enabled when the component is in-stalled. Furthermore, the log file may be configured to reside on a system device that gets reset when the system restarts. To properly enable these logs, they must be:
• activated when the service is installed
• maintained on a safe device, protected from unauthorized viewing or alteration
• set to record continuously despite system reboots
Additional third-party access management and control logs can and should be implemented to completely record and report system in a manner acceptable for use as legal evidence. This includes data that can be independently corroborated, non-repudiated, and chain of custody maintained. These requirements are discussed further in the next section, Linking Operating Forensics to Criminal Investigations.
Configuration Planning
The operational forensics program also includes defining methods for maximizing the data and evidence collection abilities while providing for fast and effective system recovery. That often can be accomplished by planning for operational forensics when system components are config-ured. One technique often used is to provide a form of disk mirroring on all devices where log files are stored. The intent is to capture data as it exists as close as possible to the event. By maintaining mirrored disks, the “mirror” can be disabled and removed for evidence preservation
while the system is restarted. This accomplishes the preservation of evi-dence and quick recovery required in a critical system.
The process for maintaining and preserving this data is then to create a minimum of three copies of the mirrored data:
1. a copy to be signed and sealed in an evidence locker pending legal action (if warranted)
2. a copy to be used as a control copy for evidence and data testing and analysis
3. a copy to be provided to opposing attorney in the discovery phase, if a criminal investigation proceeds
LINKING OPERATIONAL FORENSICS TO CRIMINAL INVESTIGATIONS
The value of a well-designed operational forensics program is in its abil-ity to have all the evidence necessary to effectively develop a criminal in-vestigation. By far, the most intensive activity in preparing for a legal opportunity is in the preparation of data that is validated and provable in legal proceedings. Three concepts are important in understanding this capacity:
1. evidence corroboration 2. non-repudiation
3. preservation of the chain of custody
Evidence Corroboration
If at all familiar with any type of legal proceeding, from the high-profile trials of the 1990s to the courtroom-based movies, television programs, or “pseudo-legal” entertainment of judicial civil cases, then one knows that evidence that is not validated through some independent means may by inadmissible. Therefore, to provide the maximum potential for critical evidence to be admitted into the record, it should be corroborated through some other means. Therefore, based on the potential for legal action, several log creation utilities can be employed to record the same type of information. When two sources are compared, the accuracy of the data being reported can be assured. For example, access to a system from the outside reported only by a modem log may be questioned that the data was erroneous. However, if the same information is validated by access to the system from a system login attempt or from an application use log, the data is more likely to be admitted as accurate.
Non-repudiation
A second crucial element necessary for a smooth legal process is estab-lishing evidence in a way that actions cannot be denied by the suspect.
D A T A S E C U R I T Y M A N A G E M E N T
This is called non-repudiation. In many recent cases of attempted system intrusion, a likely suspect has been exonerated by testifying that it could not have been his actions that caused the violation. Perhaps someone masqueraded as him, or perhaps his password was compromised, etc. There is no way to definitely make all transactions pass the non-repudi-ation test; but in establishing the secure procedures for authenticating all who access the system, non-repudiation should be included as a high-priority requirement.
Preservation of the Chain of Custody
Finally, the last and perhaps most important legal objective of operation-al forensics is to preserve the chain of custody. In simple terms, this means that the data/evidence was always under the control of an inde-pendent source and that it could not have been altered to support one side of the case. This is perhaps the most easily established legal criteri-on, but the least frequently followed. To establish a proper chain of cus-tody, all data must be properly signed-in and signed-out using approved procedures and any chance of its alteration must be eliminated — to a legal certainty. Technology has come to the rescue with devices such as read-only CDs, but there are also some low-technology solutions (e.g., evidence lockers, instant photography, and voice recorders) to track ac-tivity related to obtaining, storing, and preserving data.
For all legal issues, it is wise and highly recommended that the orga-nization’s legal counsel be included on the forensic team, and if possible, a representative from the local law enforcement agency’s (Attorney Gen-eral, prosecutor or FBI, state or local police unit) high-tech crime unit. In the case of properly collecting evidence when and if a situation arises, prior planning and preparation are always a good investments.
LINKING OPERATIONAL FORENSICS TO BUSINESS CONTINUITY PLANNING
What makes operational forensics an entity of its own is the ability to use the time and effort spent in planning for benefits other than prosecuting criminals. The key benefit is in an organization’s ability to learn some-thing from every operational miscue. Countless times, systems stop run-ning because intruders who only partially succeed at gairun-ning access have corrupted the network connections. In most instances, all the informa-tion that could have been used to close access vulnerabilities goes away with the Ctrl-Alt-Delete keys. Systems do not crash without cause. If each cause were evaluated, many of them could be eliminated or their proba-bility of reoccurring significantly reduced.
In the current age of continuous availability, maximum network up-time is directly linked to profit or effectiveness. Implementing an opera-tional forensics program can help establish an effective link to business
continuity planning risk reduction and can raise the bar of attainable ser-vice levels.
Although evidence collected for improving availability does not need to pass all legal hurdles, an effective method of cause identification can help focus the cost of prevention on real vulnerabilities — not on the whole universe of possibilities — no matter how remote. Cost justification of new availability features is more readily available, and IT can begin to function more like a well-defined business function than a “black art.”
SUMMARY AND CONCLUSION
When a system interruption occurs, operational forensics is a key com-ponent of the recovery process and should be utilized to identify the na-ture and cause of the interruption as well as collecting, preserving, and evaluating the evidence. This special investigation function is essential because it is often difficult to conclusively determine the nature, source, and responsibility for the system interruption. As such, to improve the likelihood of successfully recovering from a system interruption, certain related integral services, such as establishing the data/activity logs, mon-itoring system, evidence collection mechanisms, intrusion management, and investigative management, should be established prior to a system interruption’s occurrence. This is the primary benefit of operational fo-rensics. And one will see much more of this in the near future.
