Computer Science Formal Reasoning about Intrusion Detection Systems
Abstract
Intrusion detection is an appealing approach to improving the security of systems. The goal of intrusion detection is to detect attacks with low false positive rate and low false negative rate. New approach is needed to evaluate the effectiveness of the intrusion detection systems (IDSs).
In this dissertation, we present a formal framework for the analysis of in-trusion detection systems that employ declarative rules for attack recognition. Our approach allows reasoning about the effectiveness of an IDS by formalizing and prov-ing security properties of the IDS. Detection rules of IDSs and security requirements of the system are formalized and properties about the IDSs are proved within the framework. To illustrate the validation of our approach, we formalize and proved properties about three typical IDSs.
SHIM (System Health and Intrusion Monitoring) is used as an exemplary host-based IDS to validate our approach. We formalized all specifications of SHIM which together with a trusted file policy enabled us to reason about the soundness and completeness of the specifications by proving that the specifications satisfy the policy under various assumptions.
DRCP (Dynamic Registration and Configuration Protocol) is an auto con-figuration protocol in mobile Ad Hoc networks. With respect to this protocol, our approach defines a global security requirement for a network that characterizes the ”good” behavior of individual nodes to assure the global property. We formally prove that the local detection rules (identifying activity that is monitored) imply the global
security requirement.
OLSR (Optimized Link State Routing) protocol is a proactive, table-driven routing protocol in MANETs. We analyze a specification-based intrusion detection mechanism to detect insider attacks in the OLSR protocol. We proved that the intrusion detection approach, which focuses on monitoring of local behavior, achieves a global integrity of network routing information.
Our approach, novel to the field of intrusion detection, can, in principle, yield an intrusion detection system that detects any attack, even unknown attacks, that can imperil the security requirements of the system. The originality of our formal analysis is that it is completely based on analytical methods and does not rely on simulation or experimental evaluation.
By
TAO SONG
B.S. (Peking University) 1997 M.S. (Peking University) 2000
DISSERTATION
Submitted in partial satisfaction of the requirements for the degree of
DOCTOR OF PHILOSOPHY
in
Computer Science
in the
OFFICE OF GRADUATE STUDIES
of the UNIVERSITY OF CALIFORNIA DAVIS Approved: Committee in charge 2007 –i–
Copyright 2007 by Tao Song
I would like to express my sincere gratitude and appreciation to my advisor, Professor Karl Levitt, for his guidance, understanding, patience and support at all levels. In every sense, none of this work would have been possible without him. I would also like to thank Pro-fessor Devanbu and ProPro-fessor Bishop for reading this dissertation and offering constructive comment.
Dr. Poornima Balasubramanyam, Dr. Jeff Rowe, Dr. Calvin Co and Dr. Marcus Tylutki offered much-appreciated advice, support and thought-provoking ideas throughout my research in computer security. Many specifications and data I used in the formal veri-fication were provided by many people inside and outside UC Davis. In particular, Calvin Co provided the spec of SHIM and Chinyang Henry Tseng simulated the OLSR detection mechanism on the GloMoSim simulation platform. I thank them and other friends at UC Davis for their encouragement and support in my research.
In addition, I am grateful to my mother and father who have always been sup-portive of my academic pursuit in the United States. And I thank my brother and sister for supporting my study and taking care of our parents when I am far away from home.
My final, and most heartfelt, acknowledgment goes to my wife, Xiaohan Gu. Her support, encouragement, and companionship has turned my journey through grad-uate school into a pleasure. When I was preparing this dissertation, our lovely daughter, Sophia Song, was brought to the world. She brings endless surprise and joy to my life. I dedicate this work and my love to them.
List of Figures viii
List of Tables ix
1 Introduction 1
1.1 Overview of Intrusion Detection . . . 1
1.2 Limitations of Earlier Work . . . 3
1.3 Contribution of this Thesis . . . 4
1.4 Introducing Terminology . . . 4
1.5 Dissertation Outline . . . 5
2 Motivating examples 7 2.1 Examples of Intrusion Scenarios on Host-based Systems . . . 7
2.1.1 Ftp Vulnerability . . . 7
2.1.2 Lpr Vulnerability . . . 8
2.2 Detecting Attack Scenarios . . . 10
2.2.1 Overview of Intrusion Detection Systems . . . 10
2.2.2 Developing Detection Rules . . . 12
2.3 Enforcement of Security Policies . . . 13
2.3.1 Trusted File Policies . . . 14
2.3.2 Policies Regarding Race Condition Attacks . . . 15
2.4 Need For a New Approach . . . 15
3 The Framework 17 3.1 Introduction . . . 17
3.2 Overview . . . 18
3.3 Hierarchical Model of the Framework . . . 19
3.3.1 High level Security Requirements (Policy and Attacks) . . . 20
3.3.2 Modeling the Behavior of Systems . . . 21
3.3.3 Specifications of Intrusion Detection Rules in the Model . . . 23
3.4 Verification . . . 23
3.4.1 Specification Completeness . . . 24
3.4.2 Environment Assumptions . . . 24
4.2 Analysis of Intrusion-Detection Rules . . . 28
4.3 Framework . . . 31
4.3.1 Hierarchical framework of Verification . . . 31
4.3.2 Formalization of the Model . . . 32
4.3.3 Mechanization of the model . . . 34
4.4 Specification and Verification of SHIM . . . 38
4.4.1 Introduction to SHIM . . . 38
4.4.2 Formalization of Specifications . . . 39
4.4.3 Verification of SHIM rules . . . 41
4.4.4 Performance . . . 46
4.5 Discussion . . . 46
4.6 Summary . . . 47
5 Formal Reasoning about a Specification-based Intrusion Detection for Dynamic Auto-configuration Protocols in Ad hoc Networks 49 5.1 Introduction . . . 49
5.2 Formal network model and hierarchical framework . . . 52
5.2.1 A Hierarchical Framework for Formal Reasoning . . . 52
5.2.2 A Formal network model . . . 53
5.2.3 Automated verification with ACL2 . . . 54
5.3 Overview of DRCP . . . 55
5.3.1 DRCP Vulnerabilities and Attacks . . . 57
5.3.2 Example DRCP attacks . . . 58
5.4 A Specification-based Intrusion Detection applied to DRCP . . . 59
5.4.1 Global Security Requirement . . . 60
5.4.2 Motivation for local behavioral specifications . . . 62
5.4.3 Generation of local behavioral specification . . . 63
5.5 Formalization and Verification . . . 65
5.5.1 Formalization of security requirements . . . 65
5.5.2 Formalization of Specifications of DRCP . . . 67
5.5.3 Verification about enforcement of security requirements . . . 67
5.6 Discussions . . . 69
5.7 Summary . . . 70
6 Formal Reasoning about a Specification-based Intrusion Detection for OLSR protocol 71 6.1 Introduction . . . 71
6.2 Security Analysis of OLSR . . . 73
6.2.1 Overview of OLSR . . . 73
6.2.2 Attacks Against OLSR . . . 75
6.3 A Specification-based Intrusion Detection for OLSR . . . 77
6.3.1 Overview . . . 77
6.3.2 A Correct Behavior Model of OLSR . . . 78
6.3.3 Simulation Results . . . 79
6.4 The Framework and Formal Network Model . . . 79
6.4.1 The Framework . . . 79
6.5.1 Formalization of OLSR . . . 82
6.5.2 Validation of the OLSR Model . . . 84
6.5.3 Verification of the Intrusion Detection Mechanism for OLSR . . . . 87
6.6 Discussion . . . 88
6.7 Summary . . . 89
7 Conclusions and Future Work 91 7.1 Conclusions . . . 91
7.1.1 On the Hierarchical Model of Formal Analysis . . . 92
7.1.2 On Formal Reasoning about Host-based IDSs . . . 92
7.1.3 On Formal Reasoning about a Specification-based Intrusion Detection for Dynamic Auto-configuration Protocols in Ad hoc Networks . . . 93
7.1.4 Formal Reasoning about a Specification-based Intrusion Detection for OLSR Protocol . . . 93
7.2 Future Work . . . 94
7.2.1 Compositional Verification . . . 94
7.2.2 Improving Specifications for Privileged Programs . . . 95
7.2.3 Distributed Monitoring . . . 95
7.2.4 Detection of Temporary Inconsistency . . . 96
7.2.5 Verification on other Systems . . . 96
Bibliography 98
A Specifications for Privileged Programs in UNIX systems 103 B Functions and Theorem of the Verification 126
3.1 Framework of Our Approach . . . 18
4.1 Verification Hierachy . . . 31
4.2 Relationship among security policy, specifications and attacks . . . 33
4.3 Mechanism of SHIM to filter concurrent execution audit log . . . 44
5.1 Hierarchical Framework for Verification . . . 52
5.2 Example Operation of DRCP . . . 56
5.3 Example DRCP Attack . . . 58
5.4 Global Requirements and Local Specification for DRCP protocol . . . 61
5.5 An EFSM Model of DRCP’ Server Part . . . 64
6.1 Generation of a route from Topology Table . . . 74
6.2 OLSR Routing Finite State Automata (FSA) . . . 78
6.3 Hierarchical Framework for Verification . . . 80
2.1 The ftp buffer overflow attack . . . 9
2.2 The lpr race condition attack . . . 10
4.1 Important functions of ACL2 . . . 34
5.1 System Requirement for DRCP . . . 61
6.1 Attack techniques in MANETs . . . 75
6.2 OLSR Routing Table Establishment . . . 82
Intrusion detection is an appealing approach to improving the security of systems. It in-volves the runtime gathering of data from system operations, and the subsequent analysis of the data. There are three different kinds of detection models: anomaly detection, misuse detection and specification-based intrusion detection. Anomaly detection compares charac-teristics of the system with normal behavior profiles and an alert is reported if they do not match. Misuse detection compares current behavior to signatures of known attacks and re-ports an alert if there is a match. Specification-based intrusion detection monitors behavior of systems according to specifications, which describe desired functionality for security-critical entities. The goal of intrusion detection is to detect attacks with low false positive rate and low false negative rate. New approach is needed to evaluate the effectiveness of the intrusion detection systems (IDSs).
In this dissertation, we present a formal framework for the analysis of intrusion detection systems that employ declarative rules for attack recognition, e.g. specification-based intrusion detection. Our approach allows reasoning about the effectiveness of an IDS by formalizing and proving security properties of the IDS. Detection rules of IDSs and security requirements of the system are formalized and properties about the IDSs, like security requirements are satisfied by deploying detection rules, are proved within the framework. The formal framework and analysis methodology will provide a scientific basis for one to argue that the IDS detects all attacks that would violate security requirements of the system.
To illustrate the validation of our approach, we formalize and proved properties about three typical IDSs including a host-based IDS (SHIM), an IDS for an auto configu-ration protocol (DRCP) and an IDS for a routing protocol (OlSR). For each IDS, detection rules are formalized and security requirements of the system are proved to be satisfied with the enforcement of these detection rules.
SHIM (System Health and Intrusion Monitoring) is used as an exemplary host-based IDS to validate our approach. We formalized all specifications of SHIM which together with a trusted file policy enabled us to reason about the soundness and completeness of the
DRCP (Dynamic Registration and Configuration Protocol) is an auto-configuration protocol in mobile Ad Hoc networks. With respect to this protocol, our approach defines a global security requirement for a network that characterizes the ”good” behavior of indi-vidual nodes to assure the global property. We formally prove that the local detection rules (identifying activity that is monitored) imply the global security requirement.
OLSR (Optimized Link State Routing) protocol is a proactive, table-driven routing protocol in MANETs. We analyze a specification-based intrusion detection mechanism to detect insider attacks in the OLSR protocol. The specifications, also called local constraints, describe valid behavior of local network. We proved that the intrusion detection approach, which focuses on monitoring of local behavior, achieves a global integrity of network routing information.
Our approach, novel to the field of intrusion detection, can, in principle, yield an intrusion detection system that detects any attack, even unknown attacks, that can imperil the security requirements of the system. The originality of our formal analysis is that it is completely based on analytical methods and does not rely on simulation or experimental evaluation.
Chapter 1
Introduction
1.1
Overview of Intrusion Detection
Critical computer systems and networks, for economic, functionality, and com-patibility reasons, will likely be comprised of commercial-off-the-shelf (COTS) products. These COTS systems and software are not designed and built in a way that provides users with confidence in their ability to meet security requirements; nor are these systems secure. It seems new attacks that exploit vulnerabilities in existing systems occur daily. There-fore, there is an imminent need for new techniques that can increase the trustworthiness of COTS systems. Complementing preventive strategies, intrusion detection has been a prevalent technology that is becoming accepted by the community.
Intrusion detection is an appealing approach to improving the security of existing systems, as it can be retrofitted onto security-critical programs. Briefly, intrusion detection involves the runtime gathering of data from system operation, and the subsequent analysis of the data; the data can be audit logs generated by an operating system, packets sniffed from a network, or reports from instrumented programs, which could be applications such as a database system.
We can categorize IDSs with many different criteria. IDSs can be divided into network-based IDS and host-based IDS by the systems they monitor.. IDSs that monitor network backbones and look for attack scenarios are called network-based IDSs, whereas those that operate on hosts defend and monitor the operating and file systems for signs of
intrusion are called host-based IDSs.
Normal behaviors of a computer system have the characteristics of predictable statistics, the absence of a known attack scenario and a conformation to specifications [13]. Intrusion detection systems are developed on the assumption that attacks will break at least one of the characteristics. There are three different kinds of intrusion detection models: anomaly detection, misuse detection and specification-based intrusion detection. Anomaly detection, first proposed by Anderson [22], is based on the assumption that the characteristics of attacks are significantly different from normal behavior. The characteris-tics of the system are compared with the expected values and an alert is reported if they do not match. Anomaly detection is capable of detecting unknown attacks or variants of known attacks if such attacks significantly change the monitored characteristics of the sys-tem. But the trade-off is that detection often comes with high false-positive rate, which means tat normal system behaviors will be reported as potential attacks.
Misuse detection compares current behavior to signatures of known attacks and reports an alert if there is a match [49, 41]. Misuse systems have difficulty detecting unknown attacks as well as variants of known attacks. Whether generalized attack signatures are capable of detecting unknown attacks still remains an open question.
Specification-based intrusion detection monitors current behavior of systems ac-cording to specifications that describe desired functionality for security-critical entities [26, 30, 56, 29, 67]. A mismatch between current behavior and the specifications will be reported as an attack. Privileged programs, also called root-setuid programs, are often ana-lyzed by a specification-based intrusion detection system because of their significant impact on system security. The ”effective user” of a privileged program has root privileges and attacks against a privileged program often abuse the privilege to access security critical objects that are not supposed to be accessed by the victim programs. During program operation, the system accesses associated with the operation of a program are recorded in audit logs and matched against the specification by the IDS. Mismatches are reported and almost always indicate an attack. Theoretically, specification-based intrusion detection is capable of detecting unknown attacks or variant of known attacks and a report is issued as soon as a specification violation occurs.
While intrusion detection systems can improve the security of a system, it is hard to evaluate the effectiveness of the IDS with respect to the primary objective users have for the deployment of such systems: the ability to detect large classes of attacks (including variations of known attacks and unknown attacks) with a low false alarm rate. In addition, it is hard to assess, in a scientific manner, the security posture of a system with an IDS deployed. Even more important to users than detecting attacks is to know that IDS will generate alerts if there is any violation of a security policy.
1.2
Limitations of Earlier Work
So far, experimental evaluation and testing have been the only approaches that have been attempted to evaluate the effective of an intrusion detection system. First, the intrusion detection system is subjected to test data, which are normal (e.g. attack-free), to develop a profile of normal behavior. Then it will be challenged by test data that contain attacks, like DARPA Off-line Intrusion Detection Evaluation [50]. Results, including the detection rate for real attacks and false alarm rate, are used to assess the effectiveness of the intrusion detection system. The usefulness of these evaluations depends on the coverage of the test data and the sensitivity of the test results with respect to changes in the environments. However, it is hard to obtain test data from real environment that are known to be free of attacks or obtain test data that cover attacks representing most attack categories. Attacks experienced by a system quite often change with time as variations of known attacks surface or new attack emerge. Thus it is difficult to develop a set of test data that is comprehensive and that has strong prediction power with respect to changes in operating environments.
There is a critical need to establish a scientific foundation for evaluating and analyzing the effectiveness of intrusion detection systems. This research employs formal reasoning to analyze intrusion detection systems, attempting to provide a scientific founda-tion for assessing the security of the overall system. We consider host-based systems and network-based systems, reasoning about intrusion detection systems in isolation and their composition.
1.3
Contribution of this Thesis
This thesis makes two contributions in evaluation of the effectiveness of intrusion detection systems.
The key contribution of this research is a mechanized methodology to formally reason about the effectiveness of intrusion detection systems in satisfying specific security requirements. This is one of the first approaches to analyzing detection rules of an IDS with formal methods. A hierarchical verification model is proposed, in which we formalize an abstract system model, detection rules and security requirements. Theorems are created and proved to reason about the soundness and completeness of detection rules.
Second, we implement the framework in our analysis of a host-based intrusion detection system, SHIM and network intrusion detection systems. The specifications in SHIM are formalized and reasoned about with respect of simulated attacks and access control policies. In the analysis of network intrusion detection systems in ad hoc networks, we develop global requirements to describe the security properties of ad hoc networks and local specifications that restrict behavior of nodes. We formally prove that the global requirements are enforce by the local specifications together with some assumptions. Parts of the research are represented in my papers[54, 59, 64, 60].
Taken together, this research puts formal evaluation of intrusion detection systems on much further footing, and provides direction for the continued development of the area.
1.4
Introducing Terminology
This section explains some terms that are used throughout this proposal. Some of the terms have well-accepted definitions among security professional while others have specific use in this dissertation.
Attack: An attack is any set of actions whose purpose is to compromise the in-tegrity, confidentiality, or availability of a resource. The set of actions may be performed by a single attacker or by a group of cooperating attackers. An attacker exploits vulnerabilities in a system to gain necessary privileges to achieve his/her goal.
the system in a clear and understandable manner [34].
Exploitation: An exploitation is a set of actions that result in a violation of the security policy of a computer system. Intruders exploit system vulnerabilities or flaws to gain unauthorized access to the system.
Logging: Logging is the recording of events or statistics to provide information about system use and performance [34].
Security policy: A security policy is a statement that partitions the state of the system into a set of authorized, or secure states and a set of unauthorized or nonsecure states [34].
Specification: A specification is a description of characteristics of a computer system or program.
Specification-based intrusion detection: An intrusion detection method which mon-itors current behavior of systems according to specifications that describe desired function-ality for security-critical entities and raises an alert if there is a violation.
Privileged program: A privileged program is a program that is executed with spe-cial privileges, enabling it to bypass the system security mechanism in order to accomplish its task. In Unix, examples are those programs that are owned by root and have the setuid bit on.
Vulnerability: A vulnerability is a weakness in automated system security proce-dures, administrative controls, or internal controls that could be exploited by a thread to gain unauthorized access to information or to disrupt critical processing.
1.5
Dissertation Outline
The rest of this dissertation is organized as follows. Chapter 2 presents a framework of our approach. Chapter 3 describes our analysis of a host-based IDS, SHIM. In chapter 4 we formalize security requirements for DRCP protocol and formally address the enforcement of these security requirements by a specification-based IDS. Chapter 5 presents a distributed IDS for OLSR protocol and analyze this IDS in fulfilling security requirements for OLSR. Chapter 6 discusses various issues of our work. Chapter 7 concludes our work and gives
Chapter 2
Motivating examples
This chapter presents several examples aimed at illustrating the subtleties involved in developing detecting rules and security policies to detect intrusions and the associated pitfalls. The examples motivate the need for a formal framework for reasoning about detec-tion rules in intrusion detecdetec-tion systems. The overall approach is to prove properties about intrusion detection systems. Our proof method depends on detecting rules and a security policy, where these rules are reasoned about with respect to the policy.
2.1
Examples of Intrusion Scenarios on Host-based Systems
We present two well-known intrusion scenarios, each exploiting a different class of vulnerabilities (buffer overflow, race condition). Detection rules and security policies, which detect or prohibit these attacks, will be introduced in the following sections.
2.1.1 Ftp Vulnerability
Ftp daemon is a Unix utility for providing file transfer service to remote clients. Ftp daemon is a setuid root program, or previleged program, because root privileges are required to monitor theftpport, which is a system port in UNIX. The fact that ftp daemon runs with root privilege enables it to perform operations inaccessible by normal users. Therefore if ftp daemon is comprised, the attacker may acquire root privilege. This is the basis for the exploitation. There is a buffer overflow vulnerability for an early version
of wuftp daemon(2.4.2-beta-18). The vulnerability can be exploited by an attack which enables the attacker to invoke any arbitrary system call. Specifically, wuftp daemon invoke a strcat() C library call without restrict the size of the input. As a result, when wuftp daemon is given a very long request, the data could overwrite the memory location beyond the buffer, causing unexpected behavior. An elaborated organized request may even invoke arbitrary system calls. Due to the fact that wuftp daemon is a setuid program, the attacker can invoke a shell with root privilege and use it to access any files of the system [8].
Table 2.1 depicts a trace of the exploit. When strcat() is called, the run-time stack consists of the activation record ofstrcat(), which include the parameters, the return address, and local variables. When an attacker sends a long request message consisting of code to the ftp daemon, strcat() fills the read buffer and beyond with the message so that the buffer now contains the injected code and the return address is overwritten with an address pointing to the buffer. When the subroutine returns, it branches into the buffer and executes the attacker’s code. Obviously, we can detect aattacks that exploit this vulnerability by checking the parameters ofstrcat(). However, most auditing systems do not record the content ofstrcat() operations because of the huge volume of data associated with these operations. Therefore, it is not possible to tell whether from the audit trails, wuftp daemon is under a buffer overflow attack. However, if we look at the operations performed by wuftp daemon, we can detect the intrusion by comparing the operations performed by wuftp daemon in a normal execution and under an attack.
Another idea is to detect the buffer overflow attacks by examining audit data for ”attack signatures”, like sequences of operations or strings which are used to exploit the vulnerabilities. However, it is not difficult to develop variants of known attacks which have different signatures but still exploit the same vulnerabilities. Therefore, the signature-based approach may have high false alert rate in detecting attacks exploit specific vulnerabilities. 2.1.2 Lpr Vulnerability
This section describes a race-condition vulnerability in an early version oflpd(line printer Daemon) utility and related intrusions that exploit this and similar vulnerabilities. Lpd is a UNIX utility which provides printing services. After a printing job is submitted
ftp daemon attack comments 1 ...
2 Len = ... Set the length of the buffer
3 ...
4 t=xcalloc(len,...) Malloc memory
5 ...
6 Strcat(t, ...) Buffer overflow attack
7 ... Following attack behavior
8 Open(O RW, ”/etc/passwd”...) Change the passwd file 9 ...
Table 2.1: The ftp buffer overflow attack
by a client, lpd will check whether the client has the privilege to access the file. After that lpd will read the file and send it to the printer. Lpd is a setuid root program because root privileges are required in order to send files to printing devices. Therefore, lpd run with root privileges and can access any arbitrary file in a UNIX system.
Lpd has a race condition flaw, which enables an attacker to read files which is inaccessible to him. The flaw relates to the way lpd check the permission of a file before printing it as well as its semantics in dealing with symbolic links. Specifically, the flaw enables a nonprivileged user to print out any file that he does not have privileges to read.
The lpr attack exploits the race condition vulnerability. Please refer to table 2.2 for a simple exploit. First, the attacker requests the printing of some benign file (for example, his own file doc.txt). His rights are checked by the lpr command and, if access is granted, the request is put into the printer daemon queue. Then, before the printing actually starts, the attacker removes the printed file and replaces it by a link to some file he is not allowed to access (for example, /etc/shadow). As a result, the latter file will eventually be printed on his behalf.
We can detect this intrusion in two ways. First, we can detect it by just looking at the operations performed bylpd. Although the sequence of calls and parameters performed by lpd are the same when it is under the race condition attack, the actual physical files printed by the calls are different. In Unix, each physical file is identified by a unique inode number. In the normal sequence, lpd prints the file corresponding to the inode of the
lpd daemon attack 1 lpr doc.txt 2 read /var/spool/lpd/job-**** 3 rm doc.txt 4 ln -s /etc/shadow doc.txt 5 read doc.txt
6 send document to printer
Table 2.2: The lpr race condition attack
file whose permissions have been checked. However, in the sequence of audit records of lpd under the exploitation, the file being printed and the file being checked have different inodes. Thus, by checking the inode of the files being printed, which are usually available in the audit trail, we can detect occurrences of this intrusion.
Second, we can detect the intrusion by checking whether operations that change the meaning of the file are performed by other processes between the timelpd checks the file and prints files. Some ways of specifying the valid order among these operations are needed in order to detect the intrusion in that way. In section 3, policies regarding noninterference will be introduced and they will provide general solutions for race condition attacks.
2.2
Detecting Attack Scenarios
In this section, we give an overview of intrusion detection systems first. Then a few examples are presented to indicate the subtleties involved in developing detect rules. 2.2.1 Overview of Intrusion Detection Systems
We can categorize IDSs according to different criteria. IDSs can be divided into network-based IDS and host-based IDS by the systems they monitor.. IDSs that monitor network backbones and look for attack scenarios are called network-based IDSs, whereas those that operate on hosts defend and monitor the operating and file systems for signs of intrusion are called host-based IDSs.
Normal behaviors of a computer system have the characteristics of predictable statistics, the absence of a known attack scenario and a conformation to specifications
[13]. Intrusion detection systems are developed on the assumption that attacks will break at least one of the characteristics. There are three different kinds of intrusion detection models: anomaly detection, misuse detection and specification-based intrusion detection. Anomaly detection, first proposed by Anderson [22], is based on the assumption that the characteristics of attacks are significantly different from normal behavior. The characteris-tics of the system are compared with the expected values and an alert is reported if they do not match. Anomaly detection is capable of detecting unknown attacks or variants of known attacks if such attacks significantly change the monitored characteristics of the sys-tem. But the trade-off is that detection often comes with high false-positive rate, which means tat normal system behaviors will be reported as potential attacks.
Misuse detection compares current behavior to signatures of known attacks and reports an alert if there is a match [49, 41]. Misuse systems have difficulty detecting unknown attacks as well as variants of known attacks. Whether generalized attack signatures are capable of detecting unknown attacks still remains an open question.
Specification-based intrusion detection monitors current behavior of systems ac-cording to specifications that describe desired functionality for security-critical entities [26, 30, 56, 29, 67]. A mismatch between current behavior and the specifications will be reported as an attack. Privileged programs, also called root-setuid programs, are often ana-lyzed by a specification-based intrusion detection system because of their significant impact on system security. The ”effective user” of a privileged program has root privileges and attacks against a privileged program often abuse the privilege to access security critical objects that are not supposed to be accessed by the victim programs. During program operation, the system accesses associated with the operation of a program are recorded in audit logs and matched against the specification by the IDS. Mismatches are reported and almost always indicate an attack. Theoretically, specification-based intrusion detection is capable of detecting unknown attacks or variant of known attacks and a report is issued as soon as a specification violation occurs.
2.2.2 Developing Detection Rules
In designing an intrusion detection system, one must select which kind of detec-tion rules will be developed to detect intrusions. These detecdetec-tion rules include specificadetec-tions of a specification-based intrusion detection system and attack signatures of misuse detec-tion systems. Anomaly detecdetec-tion systems will not be analyzed because most of them are statistics-based and difficult to analyze with logic while specifications and signatures are declarable.
Development of correct intrusion-detection rules is a very difficult and error-prone task: it involves extensive knowledge engineering on attacks and most components of the system; it requires a deep and correct understanding of most of the components in a system and how they work together; it requires the rule developers to be cautious and careful to avoid mistakes and gaps in coverage. Often, it is very difficult to assess whether a given set of intrusion-detection rules is correct (they detect the attacks or enforce security policies). We discuss the subtleties involved in writing valid behavior specifications for a program. Traditionally, in specification-based IDSs, a valid behavior specification for a program declares what operations and system calls are allowed for the program. Whether an operation is allowed or not depends on the attributes of the process and the reference, and attributes of the system calls. In SHIM, a specification for a program is a list of rules describing all the operations valid for the program. For example, the following rule in the ftp daemon (ftpd) specification allows the program to open any file in the /var/log/wtmp directory to write.
(OP ENWR,$F.path== ”/var/log/wtmp”)
With specifications that list valid operations of security-critical programs, an IDS can monitor the execution of the programs and raise an alert if the specifications are violated. If the specifications are correct and complete, the IDS is capable of detecting all, including known and unknown, attacks that change the behavior of the programs. However, the draft specifications, when used to monitor the program execution, will produce false positives (i.e., valid operations performed by the program reported as erroneous because they are
not included in the specification). Then, one augments the specification to include rules to cover these operations.
However, one needs to be very careful in writing the specification for a program to avoid some attacks going undetected. For example, given the above rule, if /var/log/wtmp somehow is writable by attackers, they can create a link from /var/log/wtmp/file to the /etc/passwd file. A specification-based IDS with this rule in the specification of ftpd will permit this operation and the attack will go undetected.
For the intrusion scenarios that we introduced in the previous section, the ftp buffer overflow attack can be detected by a specification-based IDS if the attacker intend to invoke a shell or open some security-critical objects, like the passwd file; the lpr race condition attack, however, may not be detected because the attack does not change the observed behavior of the lpd program. We can augment the specification of the lpd to check the number of links to the target object and to generate a warning if the number of links to the object is greater than one, thus preventing this kind of attacks. Based on our experience, writing specifications for a program is subtle and tricky, thus demanding an approach to rule validation.
2.3
Enforcement of Security Policies
In the current view, rules are evaluated against known attacks and hypothesis un-known attacks. How good are these rules? This thesis addresses this question by proposing that the rules can be reasoned about with respect to the security goal, e.g. a security policy. One interesting question about intrusion detection is, given the detection rules, whether the high-level security policies, like access control policies, can be satisfied. In this context, we mean that actions that violate the security policies are either prohibited or detected.
Most computer systems employ access controls policies as the first line of defense to protect resources and information in the systems. The access control policies limit access to files and other operating system objects based on their ownership and the settings of the owner, group and others permission bits. To ensure the security of a system, the access
control mechanism should work properly. And, an intrusion detection system should be able to monitor violations of the standard access control policy and raise an alarm.
In addition, we identify a suite of policies at different levels of abstraction to specify the full breadth and depth of intrusion detection mechanisms and detect different attacks. This hierarchy will be addressed in the formal specification and verification described in this dissertation.
2.3.1 Trusted File Policies
Initially, we start with a simple and easily understood integrity policy, which states that certain files cannot be modified. For example, a UNIX system relies on a number of privileged programs and devices drivers. These programs often read data from configuration files, trusting the integrity of these files. These trusted files must be specified as special files by the policy and then protected as such. Trusted file access policies are security policies that we developed to keep trusted files from unauthorized access. In UNIX systems, a discretionary access control (DAC) mechanism defines whether a subject can access an object or not depending on the privilege of the subject and the access permission of the object. Some files are intended to be accessed by specific users or using specific programs. Thus, trusted file access policies are defined in our format as: (trusted file, authorized user, program, access) where trusted file is the file to be protected, authorized user defines the user that can aaccess the file with any programs and program defines the program that can be used by other users to access the file. As an example, the passwd file access policy is defined as: (/etc/passwd, root , passwd, (open-wr,create, chmod, chown, rename)). It indicates thepasswd file of a UNIX system should be editable by root using any program or by an ordinary user using the passwd program. Therefore, if an attacker compromises a ftp daemon and try to access thepasswd file, thepasswd file access policy will be violated. In fact, this policy prohibited any unauthorized access to thepasswd file. Therefore, if this policy is enforced by an IDS, it means any unauthorized access to the passwd file will be detected by the IDS.
2.3.2 Policies Regarding Race Condition Attacks
A few policies are proposed to detect race condition attacks. Zimmermann et al. proposed a reference flow control model to policy-based intrusion detection [70, 71]. The authors claim that their approach is capable of detecting unknown noninterference behaviors which violate security policy specifications of a system [71]. In the reference flow control model, a security policy is enforced by defining domains that specify sets of method calls, each combination of these calls producing authorized information flows. Any cross-domain information flow operation will be considered as an intrusion. Race-condition attacks can be detected by observing information flows from an object created by an attacker to objects that he is not authorized to access.
Ko et al. proposed another approach to detect race-condition attacks by checking valid interleaving between privileged and unprivileged processes [28]. They developed a non-interference model which ensures that concurrent behavior of two programs will have no impact on behavior of each programs. Their approach only bases on the knowledge of system call execution interleaving legality which is defined once and for all by the implementation and is considered to be more simple and efficient in detecting race-condition attacks.
The lpr race condition attack, which we introduced in the previous section, can be detected by both approaches.
Formal [42, 53, 3] techniques for assuring the security of a system have been in-vestigated. It is our view that a security policy should support formal reasoning.
2.4
Need For a New Approach
As illustrated by the examples, writing intrusion-detection rules for specification-based and misuse detection is tricky. Existing techniques for evaluating the intrusion-detection rules, primarily based on testing, have limitations. A major problem is the de-velopment of a comprehensive suite of test cases, which are typically manually generated by intrusion detection experts, based on observed attack instances and analysis of captured attack tools. The process of developing the test suite is thus labor-intensive and error-prone. Moreover, the trend toward increasing sophistication of attacks—such as multi-step
attacks, distributed coordinating attacks, and the use of code obfuscation and IDS evasion techniques—makes IDS testing more and more difficult.
An automated tool that can assist IDS developers to generate, analyze, and verify the correctness of the rules, that they are both necessary and sufficient, will be of benefit to the process of evaluating the effectiveness of intrusion-detection rules. In the next section, we will present a complementary approach, based on a formal framework, for reasoning about intrusion-detection rules.
Chapter 3
The Framework
3.1
Introduction
We develop a formal framework that permits analysis of the detection rules in intrusion detection systems. This analysis can be used to identify errors and improve the quality of the rules. In particular, we focus our effort on misuse detection and specification-based detection, both employing declarative rules to detect intrusive activity. Based on our experience, developing good misuse signatures and good specifications for programs is tricky and error prone. It often requires insights into the attacks and critical security aspects of a system. In addition, one needs to be very careful when writing them to avoid gaps in coverage. In particular, it is hard to judge whether changes to the rules actually improve or degrade the ability to detect new attacks. This research explores the applica-tion of techniques from formal method research to assist the development and analysis of detection rules. We employ verification techniques to prove that a given rule set, together with the operating system, can enforce a given security policy. In addition, we enumerate assumptions of specific security policies to identify possible ways the policies can be violated without being detected by the detection rules. Thus we abstract any special attacks and strict to prove that our IDS will detect any attack that threatening the security policy. The assumptions cover the kind of attacks that cannot or very unlikely to occur.
In particular, we model the expert rules in host-based IDSs and network IDSs in this research. We envision that the analysis allows one to provide a solid argument that the
given intrusion detection rules can detect certain attacks and violations of certain security policies, thus providing assurance of the security of the overall system beyond the level of assurance achieved by experiments and testing. We build specifications of network protocols and use them to analyze packet traffic sniffed from the network. We verify security invariants that can be enforced by these specifications. In addition, to increasing the coverage of intrusion detection systems and our confidence in that coverage, we provide a methodology to make future enhancements and to allow others to continue this work in other areas, with other intrusion detection systems, and on other platforms.
3.2
Overview
!"#$ %& ' ( )*+, -. * / 0 12 30 -4 5 6 (4)+ 7 , 87898 :()+:, -. * / * 6 (4)+ , -** )+, 8 : ; <) / ( ' ()+,Figure 3.1: Framework of Our Approach
Our approach to formal analysis of intrusion-detection rules is inspired by the formal methods research on designing and building trusted computer systems (Figure 3.1:
Left side). Here, a model of a system is shown to satisfy a security policy. Formal methods involve the use of mathematical logic to model a system that enables one to reason about the properties of the system. Formal methods have been used to discover errors that occur in the process of designing and building a computer system. Formal methods are feasible for reasoning specifications with respect to a security policy, e.g. A1 certification as in the Orange Book. Our reasoning, although new and novel, offer a similar complexity. We reason about the detection rules with respect to security requirements of a system (Figure 3.1: Right side).
The process of designing and building a trusted system involves the development of a security model. A security model usually consists of a specification of a security policy (which defines the security requirements or what is meant by secure) and an abstract behavioral model of the system. Usually, the security policy can be stated as a mapping of system states into authorized (secure) and unauthorized (insecure) states; or can be a property of the system (e.g., noninterference). The model is an abstraction of the actual system which provides a high level description of the major entities of the system and operations on those entities. There may be many layers of abstraction within the model, each a refinement of the higher level abstraction. Given the security policy and model, one should be able to prove that the model satisfies the security policy assuming some restriction on the state transition functions. The behavioral model itself can be used to guide the design of the system. In addition, one can then formally specify the design and prove that the design actually adheres to the model. Further, one can also mathematically prove that the implementation (software and hardware) correctly implements the abstract system model, or rigorously test the code. Usually, the verification is done in a hierarchical manner in which a hierarchy of abstract machines, or refinements of the system, is defined.
3.3
Hierarchical Model of the Framework
We develop a security model for IDS. This IDS security model will consist of an abstract behavioral model of an existing system, specifications of high-level security prop-erties, and specifications of intrusion-detection rules. Rather than relying on a system with
software that has been verified, an IDS runs on a COTS system, monitoring activity (au-dit logs, network packets) in the system to identify security compromises. Therefore, the abstract behavioral model captures the real behavior of the targeted system. In addition to common abstractions such as access control, processes, and files, the model captures the auditing capabilities of the targeted system. (i.e., given an operation, it will be decidable whether or not an audit event will be generated and what the information about the oper-ation will be visible). Source code, documentoper-ation, execution and packet traces, and other information are used to form the abstract system model, which will be tested to verify correctness. The specifications of intrusion-detection rules describe formally when the rules will produce IDS alarms given the sequence of audit events generated. Intrusion detection rules can be viewed as constraints on the audit trace (e.g., the sequence of observable state changes) of the system.
We develop analysis methodologies for reasoning about whether a given set of detection rules can detect the specified attacks or a violation of specified security properties. In particular, we model the rules in SHIM IDS and network IDSs in this research. We envision that the analysis will allow one to provide a solid argument that the given intrusion detection rules can detect certain attacks and violation of certain security policies, thus providing assurance on the security of the overall system besides experimental testing. 3.3.1 High level Security Requirements (Policy and Attacks)
One interesting question about the model is, given the abstract system model and the intrusion-detection rules, whether the high-level security requirements can be sat-isfied. In this context, we mean that actions that violate the security requirements are either prohibited or detected. Two different types of security requirements of interests are enforcement of a security policies and detection of abstract attacks.
Policy
At a very high level, a policy is concerned with users and data. For example, a policy statement could be user Bob can only modify data belonging to him. At the next level, we are concerned with processes (controlled by a real user through devices) and objects
(e.g., files), which contain data.
Initially, we plan to start with an integrity policy, which states that certain files cannot be modified. For example, a UNIX system relies on a number of privileged programs and devices drivers. These programs often read data from configuration files, trusting the integrity of these files. These trusted files must be specified as special files by the policy and then protected as such.
As we have been using UNIX as a sample system in host-based IDSs, we must include the standard UNIX discretionary access control policy in our security policy. The UNIX discretionary access control policy limits access to files and other operating system objects based on their ownership and the settings of the owner, group and others permission bits. An intrusion detection system should be able to monitor violations of the standard access control policy and raise an alarm.
From these beginnings we identify a suite of policies at different levels of abstrac-tion to specify the full breadth and depth of intrusion detecabstrac-tion mechanisms. This hierarchy will be addressed in the formal specification and verification described in following chapters. Attacks
An attack can be modeled as a sequence of actions that achieves the goal of the attackers. An abstract attack is a set of goal states, or actions that are enablers for a particular security violation. For example, an attack that takes advantage of a buffer overflow vulnerability to execute an arbitrary program as the root user violates the UNIX security model which restricts the effective user id of any executing process. This attack is a member of the class of abstract attacks that execute arbitrary programs with root privileges.
3.3.2 Modeling the Behavior of Systems
The behavior of systems is modeled at two levels: system-call level and program level. We develop model of existing operating systems capturing the semantics of system calls. The model is based on a simplified model [28] and includes the concepts of processes, user identifies, files and directories, access controls, etc. In addition, we develop model
of critical system programs, applications, and other components. As discussed previously, we include the concept of auditing in our model. As we are interested in system with an intrusion detection system installed, the system must have some auditing mechanisms that produce audit records of the significant events. At an abstract level, auditing can be viewed as a function that takes a trace of a system to produce an audit trace of the system. The auditing system is an integral part of our formal system models.
The analysis of the intrusion-detection rules relies heavily on the models of the systems we want to protect. We have identified and propose to investigate the following approaches for modeling the behavior of existing systems: Based on system call traces and packets observed, one can construct a model to reflect the observed behavior of a system component. This approach is used in [14] for developing models for privileged Unix programs. An important advantage of this approach is that the process of developing these models is automatic. However, models developed this way are based on activities observed during normal runs and may be incomplete for the corresponding programs. For instance, a randomly generated test case may produce a system call trace not captured by a model generated this way, resulting in an increased false positive rate.
Based on a specification of a component (e.g., IETF RFC for servers that imple-ment certain standardized network protocols), one can develop a ”weakened” model for the component. In this weakened model, the model does not include certain functions to enforce a security policy. This model may reflect known or potential vulnerabilities of the component (with respect to that security policy). Note that intrusion-detection rules that are effective for protecting a weakened model are also effective for a stronger version of the model. In other words, these rules are applicable for implementations that may have (some) mechanisms, which may or may not be correct, for enforcing the security policy. Based on the source code of the system, one can reverse engineer a ”specification” that the code really implements. This reverse-engineered specification may not be the same as the specification that the component is expected to follow. An example is the spurious TCP state transitions UC Davis discovered in an implementation of TCP [15], which do not match the TCP RFC specification. An advantage of this approach is that the models constructed this way are more accurate. However, this approach for developing models is labor intensive and time
consuming. More research is needed for automating this process.
3.3.3 Specifications of Intrusion Detection Rules in the Model
Different intrusion detection systems have different ways to represent their rules and monitor a system against the rules. For example, in SHIM, behavior specifications are described using a formal language called Parallel Environment Grammar [24].
On one hand, we would like our representation of intrusion-detection rules to be broad enough to capture all the rules in all IDS. On the other hand, we need tractable analysis of the rules. We proposed to model detection rules as a finite state machine. In current IDS, intrusion-detection rules are encoded using languages and mechanism much richer than a finite state machine For example, in SHIM, the Parallel Environment Grammar is an extended context free grammar.
Nevertheless, we believe that most intrusion-detection rules can be specified as a finite state automaton.
3.4
Verification
We propose an analytic approach based on hierarchical verification, where one of the levels contains the specifications that form the basis for a specification-based intru-sion detected system. Other levels include an overall security model for the system whose security is enhanced by the intrusion detection system. We also have a level we call as-sumptions, which declare components of the system we are not monitoring. Either we are quite certain that these components, although security critical, are not subject to attack (e.g., the operating system kernel) or because we do not necessarily have the capability of efficiently monitoring for violation of these assumptions (e.g., violation of authentication that is entirely based on IP address). In some cases, once the assumptions are declared as required by the verification approach, an intrusion detection system or a suitable static analyzer if deemed suspect can monitor them.
This approach to verification provides the (ultimate) promise of verification of the intrusion detection system specifications with respect to an abstract security model. As we
discuss, this kind of verification is motivated by the work on verification of operating system specifications to satisfy the Orange Book A1 [12] assurance requirements. In this approach, there is no attempt to verify program code: the operating system, protocols, or intrusion detection system. There have been several successful efforts with specification verification, leading us to believe in its feasibility. Our work develops the overall methodology and carries out sample proofs. We begin mechanization of the proof methodology using a theorem prover, ACL2. As discussed below, We prefer ACL2 as it supports executable specifications, permitting us to ran and test the specifications we produce for operating systems (kernel and privileged programs) and network protocols.
3.4.1 Specification Completeness
All system specifications, at least implicitly, include a set of assumptions about the environment in which the system will execute and about the underlying abstract machine on which the system runs. Our specification-based constraints can be viewed, in part, as an attempt to make explicit all assumptions about the environments in which each level of the system – programs, protocols, hosts, system services, network services, as well as the whole enclave - function, and to delineate the attributes of the underlying abstract machines that must be correct in order for the complete system to be secure. All assumptions must be identified and specified before a system specification can be considered complete.
3.4.2 Environment Assumptions
Attackers frequently break into a system by violating assumptions about the sys-tem’s environment. A buffer overflow attack, for example, violates the (bad) assumption on the part of the programmer that the input data will never be larger than the space allocated for it. The assumptions about the environment must hold or else the system implementation cannot be guaranteed to satisfy its specification. This can be represented mathematically as E ⇒ M, where E is the specification for the environment and M is the specification for the system. The specification is true in all cases where both the environment and system satisfy their specifications, or where the environment does not.
can then be addressed. The buffer overflow problem, for example, can be prevented by reading data only in blocks of a size less than or equal to the buffer size. The improved program no longer makes any assumption about the length of the input, which eliminates the weakness. In some cases it may be impossible to eliminate an assumption in this manner, so sensors must be used to detect when the assumptions have been violated. For example, if it is difficult or impossible to rewrite a program to eliminate buffer overflows, techniques such as StackGuard [10] may be used to detect a buffer overflow. Also, specification-based intrusion detection can, in principle, detect any attack (including buffer overflow and race condition exploitation) that causes the exploited program to access objects not in its least privilege set. Security cannot be assured if all assumption violations cannot be detected or, with some assurance, be expected not to occur. One of the goals of our work is the explicit specification of these assumptions with the goal of better understanding the operating environment of the programs under attack.
The specifications characterize the behavior enforced by the intrusion detection system. These specifications, together with assumptions (such as the kernel is not vulnerable to attacks), can be verified with respect to more abstract security models of the system. For example, a Unix access control model can be used to verify the specifications for the Unix privileged programs. Other access models can be used to verify specification-based intrusion detection systems for protocols such as the Domain Name System (DNS) [38] and the Network File System (NFS) [62].Models about routing can be used to verify a specification-based intrusion detection system for the Open Shortest Path First (OSPF) routing protocol [39]. In all cases, the reasoning is about specifications and not program code, thus the technique is practical for automated reasoning.
In this chapter, we have presented the framework of formal reasoning about in-trusion detection systems. In the following chapters, we will discuss the implementation of the framework on host-based intrusion detection systems and network intrusion detection systems.
Chapter 4
Formal Reasoning about
Host-based IDSs
4.1
Introduction
Intrusion detection is an effective technology to supplement traditional security mechanisms, such as access control, to improve the security of computer systems. To date, over 100 commercial and research products have been developed and deployed on operational computer systems and networks. While IDS can improve the security of a system, it is difficult to evaluate and predict the effectiveness of an IDS with respect to the primary objective users have for the deployment of such a system: the ability to detect large classes of attacks (including variants of known attacks and unknown attacks) with a low false alarm rate. In addition, it is difficult to assess, in a scientific manner, the security posture of a system with an IDS deployed. So far, experimental evaluation and testing have been the only approaches that have been attempted. There is a critical need to establish a scientific foundation for evaluating and analyzing the effectiveness of IDSs.
This chapter presents an approach to formal analysis of IDSs. Our approach is primarily applicable to IDSs that employ declarative rules for intrusion detection, including signature-based detection and specification-based detection[41][49][24][30][25][29][65]. The former matches the current system or network activities against a set of predefined attack
signatures that represent known attacks and potential intrusive activities. The latter rec-ognizes attacks as activities of critical objects that violate their specifications. Testing is currently being used to evaluate the effectiveness of the rules. Nevertheless, testing is usu-ally performed according to the tester’s understanding of known attacks. It is difficult to verify the effectiveness of an IDS in detecting unknown attacks. Our approach is inspired by the significant body of formal methods research in designing and building trusted com-puter systems. Briefly, the process of designing and building a trusted system involves the development of a security model, which consists of a specification of a security policy (the security requirements or what is meant by security) and an abstract behavioral model of the system. Usually, the security policy can be stated as a mapping from system states to authorized (secure) and unauthorized (insecure) states[34]or as a property (often stated as an invariant) of the system (e.g., noninterference). The model is an abstraction of the actual system that provides a high level description of the major entities of the system and operations on those entities. There may be many layers of abstractions within the model, each a refinement of the higher level abstraction. Given the security policy and model, one should be able to prove that the model satisfies the security policy, assuming some restrictions on the state transition functions (e.g., the classical Bell and LaPadula model). Our framework consists of an abstract behavioral model, specifications of high-level security properties, and specifications of intrusion-detection rules. The abstract behavioral model captures the real behavior of the targeted system. In addition to common abstractions such as access control lists, processes, and files, the abstract behavioral model will capture the auditing capabilities of the targeted system(i.e., given an operation, it will be decided whether or not an audit event will be generated and what information about the operation will be visible). The specifications of intrusion-detection rules describe formally when the rules will produce IDS alarms given the sequence of audit events generated. Intrusion detection rules can be viewed as constraints on the audit trace of the system (e.g., the sequence of observable state changes).
We employ the formal framework to analyze the properties of SHIM [29], a specification-based IDS that focuses on the behaviors of privileged programs. In SHIM, specifications are developed to constrain the behaviors of privileged programs to the least privilege that
is necessary for the functionality of the program.
ACL2 [40] is employed to describe an abstract system model that can be used as the basis for different IDSs. A hierarchical model is built to generalize the verification of specifications. As an example, we formalize specifications of SHIM and a security policy (e.g, a trusted file access policy). And we prove that these specifications can satisfy the policy with various assumptions. Again, the assumptions represent activities that the SHIM IDS does not monitor, although it could if the IDS designer believes an attacker could cause the assumptions to be violated.
The rest of the chapter is structured as follows: Section 4.2 introduces and ana-lyzes intrusion detection rules, primarily used in a specification-based IDS such as SHIM. Section 4.3 describes a hierarchical framework of verification. Section 4.4 shows an example of our verification approach. We formalize specifications of SHIM and prove that these specifications together with assumptions satisfy trusted file access policies. In Section 4.5 we discuss our results and the limitations of the verification method we developed. We conclude and provide recommendations for future work in Section 4.6.
4.2
Analysis of Intrusion-Detection Rules
Development of correct intrusion-detection rules is a very difficult and error-prone task: it involves extensive knowledge engineering on attacks and most components of the system; it requires a deep and correct understanding of most of the components in a system and how they work together; it requires the rule developers to be cautious and careful to avoid mistakes and gaps in coverage. Often, crafting of intrusion-detection rules is performed by human security experts based on their knowledge and insights on attacks and security aspects of a system. Therefore, it is very difficult to assess whether a given set of intrusion-detection rules is correct (they detect the attacks). Furthermore, the complexity and subtlety of systems and attacks make it a challenging task to judge whether changes to the rules actually improve or degrade their efficacy with respect to their ability to detect new attacks.
program. Traditionally, in specification-based IDSs, a valid behavior specification for a program declares what operations and system calls are allowed for the program. Whether an operation is allowed or not depends on the attributes of the process and the reference, and attributes of the system calls. In SHIM, a specification for a program is a list of rules describing all the operations valid for the program. For example, the following rule in the line printer daemon (lpd) specification allows the program to open any file in the /var/spool/hp directory to read and write.
(open,$f lag==O RDW R&&InDir($F.path,”/var/spool/hp”))
The expression formally describes a set of valid operations: any open() system call with the flag argument equal to O RDWR (open the file for read and write) with an absolute path name subordinate to the/var/spool/hp directory.
One way to develop a specification for a program is to first identify what operations and accesses the program needs to support its functionality. Based on an examination of the code or its behaviors, one writes rules in the specification to cover the valid operations of the program. The ”draft” specification will be tested against the actual execution of the program. Often, the draft specification, when used to monitor the program execution, will produce false positives (i.e., valid operations performed by the program reported as erroneous because they are not included in the specification). Then, one augments the specification to include rules to cover these operations. In general, one needs to be very careful in writing the specification for a program to avoid errors.
For example, given the above rule, if/var/spool/hp somehow is writable by attack-ers, they can create a link from/var/spool/hp/file to the/etc/passwd file. A specification-based IDS with this rule in the specification oflpd will permit this operation and the attack will go undetected. Therefore, we augment the rule to check the number of links to the file and to generate a warning if the number of links to the file is greater than one,thus prevent-ing this attack from usprevent-ing hard links. This also works for soft links in our system because the audit record for anopen()operation will provide the absolute pathname of the file being opened, if the path is a symbolic link. Based on our experience, writing specifications for a
program is subtle and tricky, thus demanding an approach to rule validation.
Little research has been done on analyzing intrusion-detection rules. Different approaches have been taken to specify and analyze the intrusion signatures and detection rules[32][51][47], primarily for signature-based IDSs. A declarative language, MuSigs, is proposed in [32] to describe the known attacks. Temporal logic formulas with variables are used to express specifications of attack scenarios [51]. Pouzol and Ducasse formally specified attack signatures and proved the soundness and completeness of their detection rules. In addition, data mining techniques and other AI techniques such as neutral network are used to refine and improve intrusion signatures[16][33][58].
Our approach is different from these approaches in various ways. First of all, we developed a framework to evaluate detection rules of different IDSs. We formalized security-relevant entities of an UNIX-like system as well as access logs. Detection rules including intrusion signatures and specifications can be formalized and reasoned about in the framework.
Second, we proposed a method to verify security properties of IDSs together with assumptions, with respect to security policies. Security polices are always satisfied with sufficiently strong assumptions. So the key is to identify assumptions that are strong enough but not too strong. An attack can violate a security policy by breaking its assumptions. So it is possible to verify the improvement of security by proving the weakening of assumptions. For example, assuming a policyP is satisfied with assumptionA and with the deployment of the mechanism m , and P is satisfied with assumption B where A implies B, then we can say m improves the security because attacks violating assumption B will also violate A, but attacks that violate assumption Amay not violate B .
As our preliminary results, we have verified a significant property of specification-based IDSs: the capability to detect unknown attacks. In our verification, the specifications of SHIM satisfy apasswd file access policy with some assumptions. This meansany attacks, including known attacks and unknown attacks, that violate the policy can be detected by SHIM.
4.3
Framework
We present a framework for analyzing detection rules in IDSs. Our goal is to answer the question of whether a given set of intrusion detection rules can satisfy the security requirements of the system. Security polices and properties of attacks are used to describe the security requirements of the system. The satisfaction of the security requirements determines whether violations of security policies or instances of attacks can be detected by the detection rules.
4.3.1 Hierarchical framework of Verification
V erif ication S∪L∪H∪R→SR Seur
