BASIC CONCEPTS

The characterization of the systems involved in a benchmark execution is a particularly relevant topic, as benchmark users need to properly distinguish the systems designed to support the benchmark from those under evaluation. In fact, our methodology is built around systems derived from classical works on performance and dependability benchmarks (SPEC 1988; TPC 1988; DBench 2004) and, henceforth, we adopted the same terminology used in these previous works. The definition of the main benchmark systems are provided next and their relationship in a benchmarking scenario is illustrated in Figure 3-1.

 Benchmark Management System (BMS) refers to the components that manage the benchmark experiments. These components are installed and deployed during the preparation of the analytical and experimental setup of the static and dynamic part and take part of the benchmark instrumentation. These components are properly described in the next sections.

 Benchmark Target (BT) refers to the system or component that is characterized during a benchmark run. Considering, as an example, the context of web serving systems, the benchmark target can be the web server, the web application, the database, or the operating system (or even subsets of these components).

 System Under Benchmark (SUB) refers to the system that provides the operational environment for the execution of the benchmark target. This may include components such as operating system, libraries, and interface components, to name a few examples. It also includes benchmark components directly related to the execution of the benchmark, such as the workload. As our security benchmark methodology is generic there are no specific restrictions concerning classes of software-based system for the system under benchmark role. As an example, in the methodology implementation we provide in Chapter 5, a web serving system is used as the system under benchmark, with the web server component as the benchmark target, and the remaining components of the SUB include the operating system, the database, and a web application.

The goal of measuring security for comparative purposes makes security the most important concept to be defined. According to (Avizienis et al. 2001), security is the concurrent existence of the attributes confidentiality, integrity, and availability. Confidentiality refers to the protection of functionality and data against unauthorized access (Bishop 2003). Integrity refers to the trustworthiness

of data or resources, assuring that the actions and data are correct (Bishop 2003). Availability refers to the readiness of the system to provide the expected service, i.e., to the ability to use the information or resource desired (Avizienis et al. 2001). It is important to make clear that a system is not secure if attackers are able to obtain restricted content (confidentiality), or to modify it (integrity), or make it unavailable (availability).

The security of a system is compromised when one or more vulnerabilities are successfully exploited by attacks. A software vulnerability - as already described in Chapter 2 - is an instance of a mistake in the specification, development, or configuration of software such that its execution can violate the explicit or implicit security policies (Krsul 1998). An attack (also termed in this thesis as vulnerability exploitation) is any action aimed at compromising the security of a system (adapted from (Stallings 1999)). A successful attack is the one that exploits a vulnerability and compromises, at least in part, one of the security attributes already described.

The need of quantifying the loss caused by attacks and taking into account the probability of their occurrence led us to use the notion of risk. The definition of risk used in this thesis was adapted from (Lowrance 1976) and refers to the measure of the impact and probability of adverse effects. This notion brings two important characteristics (Kirkpatrick, Walker, and Firth 1992): loss (a Figure 3-1. Relationship between Benchmark Management System (BMS) and

vulnerability exploitation that has unwanted consequences or losses) and uncertainty (a vulnerability that may or may not be exploited). In the context of our work, the loss factor refers to the impact of an attack to the security attributes of a system. To measure the extent of such impact, we take advantage of the criteria defined by the Common Vulnerability Scoring System - CVSS (Mell, Scarfone, and Romanosky 2007). The impact assessment over system confidentiality is done by evaluating the system ability to keep confidential restricted areas, among other verifications. The verification if system responses are as expected is necessary to assess the impact over system integrity (i.e., attacks were unable to alter system content). The impact evaluation over system availability is performed by checking if the system becomes unresponsive for any period of time during the execution of attacks. The highest impact level is the one that cause a complete compromise of system confidentiality, integrity, and availability.

The probability of an attack occurrence is given by the level of easiness to exploit a vulnerability. This easiness is termed by the Common Vulnerability Scoring System 2.0 as exploitability and is given considering three independent factors: the network location from where the attack is executed (local network, adjacent network, or remote network); the need of system authentication (none, single, multiple); and the complexity to mount the attack over the target vulnerability (low, medium, high). In other words, an attack with the highest probability level is the one that can be executed remotely, with little skill needed to mount and execute, and with no need of authenticating into the system.