Knowledge Utilisation

The knowledge utilisation phase is based on three steps: first, generate the problem instance; second, produce a plan solution; and third, convert the plan into a user-friendly

Figure 5.3: Screenshot of the same application demonstrating the generation of PDDL domain action model from the existing rules stored in database. This domain model represents the set of all those rules, which were previously acquired from one or more

event log datasets.

format. The knowledge utilisation mechanism is implemented through a graphical user- interface application that guides the user to appropriately perform all steps in an ordered manner.

The first part of the application takes event log entries along with the domain action model that contains the security expert knowledge, and produces a problem instance as shown in Figure5.4. It is possible that the objects extracted from the event log entries and used in the problem instance do not align with the input format of planner, and hence will require alteration. For example, objects starting with a number or fully numeric objects are not acceptable, so we added a ‘O <numeric object>’ prefix to denote them as alpha-numeric objects. Similarly, some objects might contain special characters, such as round brackets, colons, commas and so on, which are not permitted. Such characters raise syntax errors during the planning process and will be completely removed if found in any object.

The second part of the same application is to use the domain action model and problem instance as input to LPG planner and produce a plan solution. The LPG produces multiple plans as it is an incremental planner, where each increment or iteration has increased quality. Therefore, only the last plan is displayed as an output as it would be

Figure 5.4: Screenshot of application that takes the domain action model and event log entries of a vulnerable machine to generate a problem instance, representing the

current security state of the machine.

of the highest quality. As the source code of LPG is available, we increased the value of constants MAX RELEVANT FACTS to 40,000 and MAX TYPE INTERSECTIONS to 10,000 and recompiled the code to accommodate all problem instances, especially the ones with higher number of objects. The developed application limits the maximum amount of plans to 15 and time to 30 minutes. The reason behind enforcing 15 plans limit is to find as many plans as possible; however, this limit is flexible and can be any number from 2 and onward. The 30 minute execution time of the planner is programmed

Figure 5.5: Screenshot of the same application that takes the domain model and newly generated problem instance to produce an action plan for expert security assessment

and configuration actions.

by-default and can be changed as well. Due to these limits, the execution of planner will automatically terminate if it either finds 15 plans or exceeds 30 minutes of planning process. The user can also manually signal the planner to exit. An example of plan is shown in Figure 5.5, which is the 12th plan solution. This plan was found in 95.47 seconds, which means that the planner executed for 30 minutes and could not find more than 12 plans. It should be noticed here that there is a repetition of actions in the plan. This is due to the planner providing a collective solution for multiple objects of the same types.

The final part of the same application is to convert the planner output into a more usable and explanatory format, as shown in Figure 5.6. This is achieved by implementing a simple plan parser that splits every action name into separate cause and effect events, along with adding their event descriptions. The parser processes the entire plan to describe the security issue and suggested solution. The order of parsed plan is kept same

Figure 5.6: Screenshot of the same application that contains a simple plan parser to display the LPG planner output in an easy-to-understand format.

as the original plan. Also, every object-value in the action parameter list is assigned a corresponding object-name to make it more descriptive. The object-names are extracted from the given event log entries. The plan parser also includes a searching mechanism that uses a simple string matching as well as regular expressions to find keywords in event types, descriptions and object-value pairs.

5.2

Evaluation

This section presents an evaluation framework to explore the accuracy and performance of the implemented solution. This involves empirical analysis of the automated learning behaviour of the developed solution using test data. The implementation is a prototype

of this research and has been developed with the intention to assess the proposed tech- nique from multiple aspects. The aims of the evaluation framework are stated in the following:

1. Test the ability of the developed solution to automatically discover the patterns of event relationships, which accurately depict the expert user actions for security improvements; and

2. Determine the efficiency and performance of using automated over manual plan solution in identifying security issues of the underlying machine and proposing mitigation actions.

5.2.1 Challenges

According to a recent study [190], there is a critical deficiency of security event log datasets that are operational and publicly available, especially from Microsoft Windows- based operating systems. This is due to the security and privacy related concerns of sharing data. The available datasets have limited access and other constraints regard- ing their usage. Most of the datasets are not suitable for research purposes as they do not provide a complete understanding of the context/environment and information about the user activities that triggered the events. This can lead researchers to forge assumptions about the fidelity and utility of event log datasets. So, one of the main challenge in evaluating the developed solution is to find or rather produce benchmark datasets that contain events depicting mitigation action(s) for predefined security vul- nerabilities. Generating such datatsets would need a thorough understanding of event logging mechanism and security auditing and policies, along with the expert security knowledge. Without understanding the knowledge inside the datasets, it would be quite challenging to evaluate the extracted sequences of event relationships.

The PDDL domain action model, which represents Temporal-Association-Causal (TAC) rules acquired from the event log datasets, acts as a knowledge base for generating plan solutions against specified problems. Therefore, the correctness of the domain knowledge and modelling process is a critical factor to determine the overall quality of problem solving process. According to multiple research studies [191, 192], the validation and verification (V&V) of planning systems are one of the biggest challenges. In our context,

verification is the process of finding whether the actions of a plan are correct with respect to the expert security knowledge, whilst, validation is the process of finding whether the actions of a plan are correct with respect to the domain model.

As mentioned in Section 4.2, the V&V process is performed based on the ability of operating in real-world settings, completeness and accuracy measures. Some tools are available that can validate a plan solution against the respective domain model, such as VAL [193]; however, there is no standard and dedicated process, automated or otherwise, which can be used for verification. The knowledge acquisition and representation are not mathematical procedures, thus a domain model cannot be measured on any accuracy scale. To encourage automated planning researchers, the International Competition on Knowledge Engineering for Planning and Scheduling (ICKEPS) has devoted a section to present novel domain debugging, validation and verification mechanisms. It is con- cluded in recent ICKEPS that the verification process is still relied on human expertise, and there is a lack of cooperation and coordination between the experts and domain modelling process [194].

The developed solution is the first system of its kind in terms of integrating automated rule mining, domain modelling and knowledge utilisation mechanisms. The previously proposed solutions employ human experts to define the event relationships, create and encode rules, and then perform automated or semi-automated deliberation process. As the aim of our solution is to replicate human expert actions for security assessment and configuration of a machine, it would require a human expert to decide whether the proposed sequence of actions for the problem instance is correct in terms of security resolution. Therefore, accuracy analysis of the solution would require human assistance.

5.2.2 Methodology

The evaluation methodology adopted in this research is inspired from the testing of several automated knowledge acquisition techniques [195, 196]. An overview of the methodology is shown in Figure5.7, and described in the following:

1. Obtain the event log datasets, where they are already known to contain events describing expert mitigation and configuration actions for specific vulnerabilities;

Figure 5.7: Overview of the evaluation methodology

2. Using the datasets, generate automated domain models from the developed solu- tion and manual domain models with the help of human experts;

3. Extract a problem instance from a machine that is known to be vulnerable; 4. Find plan solutions from both automated and manual domain models against the

same problem instance; and

5. Conduct a comparison between manual and automated plan solutions with re- spect to sequences of actions and time to determine accuracy and performance, respectively.

5.2.3 Process Of Evaluation

The developed solution has been evaluated using a set of security event logs acquired from multiple, live computing machines. Here the term ‘live’ is used to express that the machines are regularly used in the real-world operations and not solely for research purposes. The reason behind including multiple machines in the evaluation process is to verify that the developed solution is sufficiently adaptable and applicable under different circumstances. The accuracy of the developed solution is based on whether it was successfully able to extract knowledge from the configured machines, and then use the knowledge to propose human-like security actions for unseen vulnerable machines. This evaluation also determines the effectiveness and performance against a manual log analysis approach that requires significant security knowledge and human resources in terms of time and effort. It should be noticed here that the measured time displayed in the evaluation results is subjected to change. It depends on the hardware specification of a machine, and availability and amount of free computing resources. The remaining section provides a detailed explanation of the evaluation process.

5.2.3.1 Data Collection

To acquire data for evaluating the solution, 21 machines are chosen that are part of a university network and used on a daily basis by different kinds of users. All ma- chines are configured by security administrators according to the university’s policies. In addition, a tool called, Microsoft Security Compliance Toolkit4, was used to confirm the implemented policies. This tool performs the comparison of actual and expected machine configuration based on the given security policies, hence verifying the ground truth. These machines have different operational roles/settings (e.g. student, staff, administrator etc.) and configurations. Each event log dataset has a distinct set of characteristics, such as amount of routine events, security-related activity, number of entries and so on. As the specifications of security policies are already known, their in- terpretation provided the ground truth and made it possible to determine the accuracy domain model generated by the proposed solution.

4