two, we base our work on prior work on Private Information Retrieval, and for trust, on a protocol for developing a trust model.
In terms of scalability, we propose a divide-and-conquer approach to global scale or-ganization. We see a number of distinctive motivations all leading to our region-based approach, including simple reduction in scale, often leading to exponential decrease in complexity, and the efficiencies of operating in a homogeneous environment, leading to performance gained by physical or topological partitioning, etc. These sorts of motivations lead us to propose an underlying region-based capability to achieve any or all of these.
This chapter is organized as follows. Section 7.2 describes the framework for com-posite intrusion detection, then addresses the security issues including protocols of privacy protection and trust, and discusses the region-based organization for agent discovery and knowledge aggregation. Section 7.3 presents a case study on distributed intrusion detection on the DETER testbed based on the framework. Section 7.4 discusses a selection of related work. Section 7.5 concludes the chapter and highlights the future work.
User
P.E. P.E. P.E. P.E.
Secure Knowledge Sharing Detection Engine
Dependency Knowledge IDS
Existing Traffic
Monitor Local Context
Figure 7-1: Knowledge-based intrusion detection framework. P.E. refers to “policy en-forcer” described in Section 7.2.2.
engine analyzes the request, and collects necessary knowledge from knowledge agents and other detection engines. The knowledge agents provide processed knowledge, such as net-work traffic, local observations, according to their local policies. After collecting enough knowledge, each detection engine builds a dependency graph of the collected knowledge, and then runs inference algorithms on it and reports the result(s) to the user, as demonstrated in an example at the end of this section. All the parties use the same ontology language to describe their requests and capabilities, similar to [69]. We describe each component in detail below. The secure knowledge sharing between parties (dealing with privacy and trust) is discussed in Section 7.2.2, so here we will just mention it at the relevant places.
User
A user can issue a request to a detection engine directly or through a local knowledge agent.
A request consists of three parts: goal, constraints, and prior knowledge. The goal defines the task to accomplish, such as determining whether a network is intruded by a worm, etc.
The constraints define the conditions that detection engines must satisfy. Prior knowledge provides some existing knowledge that may be useful to the detection engines.
Detection Engine
The task of a detection engine is to detect intrusions. Its role is similar to the regional leader in the previous discussion in the sense that it coordinates the operations of the knowledge agents to resolve a request. A detection engine uses all available knowledge from knowl-edge agents to resolve a request in the following way. When a request is received, the detection engine analyzes it and figures out which agents are likely to meet the goal and constraints. During this process, the detection engine needs to coordinate the knowledge sharing among agents. Some agents may have relevant techniques, some may have the data, and others may have the dependency knowledge between the techniques. Those with the relevant techniques may not have access to the data due to privacy constraints. The detection engine needs to find proper agents that can run the technique over the data. Then it collects the responses from them, makes a final analysis, returns the result to the user, and caches it for future use. Detection engines are similar to the regional leader discussed in previous chapters, as they coordinate the operations between agents within a network, and communicate with other detection engines to exchange information, but unlike regional leaders, there may be multiple detection engines within a region, as discussed later.
We focus here on the analysis of the detection engine. Once the detection engine has collected some knowledge, it can start to build a dependency graph between the knowledge that it has received so far. During this process, new knowledge from additional knowledge agents may be added, and the detection engine dynamically adjusts the dependency graph.
With the addition of new knowledge, there are three possible actions that can result. First, this addition might produce knowledge that the current dependency graph (in this detection engine) does not know anything about. In this case, the most natural way to add this into the current graph is to treat it as new and independent information. Second, this addition might give knowledge relating several pieces of knowledge that are already in the current graph. In this case, the probabilistic structure of the graph will be changed to reflect this new knowledge of the dependency. Third, this addition might give knowledge that does not really change the internal structure of the graph, but instead wraps around it and thus affects the results of the detection engine.
Knowledge Agent
Knowledge agents correspond to the agent members in the previous discussion. Knowledge agents provide various kinds of knowledge, and act as a proxy of users. A knowledge agent can be an existing intrusion detection system that uses specific techniques on an end host or on a network, one that contributes a new detection technique, or one that simply provides any valuable knowledge. We identify four kinds of useful knowledge, as demonstrated in Figure 7-1.
The first and most important category is existing intrusion detection systems. Here we view a detection system as a database of results generated by applying a detection technique on an end host or on the part of the network it has access to. It includes detection techniques based on incoming and outgoing traffic such as [63, 70], and signature-based approaches [25] together with the worm signature databases.
The second is traffic monitor. Such agents monitor the traffic data for the analysis.
They often reside on vantage points in the network, such as gateways. Note that in our framework the detection techniques and the data are separate.
The third is local context. This includes network configuration, operating system types, running services, results of local virus scans, etc. Such knowledge exposes the potential vulnerability of the network and hosts.
The fourth is dependency knowledge. This describes the dependency between multiple pieces of knowledge, such as the conditional probability of incoming and outgoing traffic.
When a knowledge agent receives a query from a detection engine, it first checks its local policies on exchanging knowledge with this detection engine. Then it engages in a secure knowledge exchange with the detection engine to provide the knowledge without disclosing sensitive information.
Example
We use a simple example to demonstrate how the components interact with each other in the framework. The example request is to detect whether Code Red intruded the network 1.2.3.4/24 in the past seven days, under the scope constraint that the knowledge agents to
ask must be within the local ISP. A piece of prior knowledge is that the operating system running on most hosts within the network is Windows. The request is shown below. Note that this is just an example to demonstrate how a detection engine may deal with a request, and we do not focus on how a request should be precisely defined in this spec-KP.
<Request id=11239045>
<Goal>
<Worm> Code Red </Worm>
<Network> 1.2.3.4/24 </Network>
<TimeRange> 7 days </TimeRange>
</Goal>
<Constraint>
<Scope> Local ISP </Scope>
</Constraint>
<PriorKnowledge>
<OSType> Windows </OSType>
</PriorKnowledge>
</Request>
Figure 7-2 demonstrates the process. The request is sent by a user to a detection engine.
The detection engine parses the request and does the following. We stress again that when-ever there is a knowledge exchange, the parties involved use the secure knowledge sharing protocol as described in Section 7.2.2.
1. As the request is about a specific worm, the detection engine checks whether any knowledge agent knows the signature or some properties of Code Red. If not, it has no way to resolve the request, and will return a failure to the user, together with the reason. If the request does not specify any worm, then this step is skipped.
2. If the signature and some traffic pattern are available, the detection engine collects such knowledge, and chooses a number of knowledge agents based on trust, pri-vacy, and the scope constraint specified in the request, through the agent discovery mechanism in Section 7.2.3. Suppose that at this point in time, two agents happen to be chosen, one using signature-based technique, and the other using a traffic pat-tern based technique. Then the detection engine hands over the knowledge about the worm to the knowledge agents, respectively.
3. The two trustful agents analyze some hosts and the recent traffic in the network using their own techniques, respectively, and return the results. Note that the data analyzed may come from a third traffic-monitor agent.
4. After receiving the results from the knowledge agents, the detection engine builds a dependency graph of the results, and runs some inference algorithm, for instance, Bayesian inference.
As a concrete example, suppose the detection engine employs the following rule to integrate the results from the knowledge agents: the detection engine will report to the user the probability that both results are “No intrusion”. Since there is no dependency knowledge about the results (yet), the detection engine will assume in-dependence between them. Therefore, the final result will be calculated as:
P(Intrusion) = 1−P(R1=No & R2=No)
= 1−P(R1=No)·P(R2=No)
5. At this point in time, another relevant agent happens to join the system, giving de-pendency knowledge relating the signature and the traffic pattern. For instance, this might be knowledge about P(R2=No|R1=No). In this case the result will be revised as:
P(Intrusion) = 1−P(R1=No & R2=No)
= 1−P(R1=No)·P(R2=No|R1=No)
6. The detection engine reports a final result to the user.
This example demonstrates how a detection engine detects an intrusion with the collab-oration of multiple knowledge agents, while following the privacy and other constraints, and how a new piece of knowledge helps the detection engine obtain a better result. Note that a technique itself is just a piece of knowledge in this framework, and this is especially useful when a detection engine could not let the agent with that technique analyze the data directly due to privacy constraints.
User
Request
ͲGoal ͲConstraints ͲPrior Knowledge
Send
Receive
DetectionEngine
Query
Collect results Builddependencygraph Inference
KnowledgeAgent
Enforcelocalpolicy
Runlocalalgorithms Analyze
request
Figure 7-2: The resolution process of a request.