THESIS OBJECTIVE

This thesis proposes a novel methodology to support the development of functional and effective security benchmarks that can be applied over any class of software-based system. This methodology uses security risk as the benchmark metric, with a single metric (SBench) that enables users to compare system security. To the best of our knowledge, the notion of security risk has not been used for the definition of security benchmark metrics (although the notion of risk has been widely used in security evaluation, outside the benchmarking scope) and our decision to use it is based on the fact that this metric is able to translate into a single number the risk of vulnerabilities present in a software system. The purpose of this number (metric) is to indicate the security level of the system under benchmark, helping users to identify which system to use when faced with the need to select one system among functionally equivalent ones.

This thesis exemplifies the proposed security benchmark methodology by providing a security benchmark for web serving systems, also describing the tools implemented to speed up the benchmark execution. Web serving systems form the basis of many services, such as e-commerce and banking systems. These systems are heterogeneous and complex, based on several discrete components. This internal complexity potentiates the existence of vulnerabilities that might be exploited by attackers. Because these systems are naturally connected to the Internet, and thus exposed to many users and attackers, any internal vulnerability becomes a real threat to security (e.g., (OWASP 2013; B. Martin et al. 2010)). Therefore, the web-serving scenario as case study of our benchmark methodology is relevant. In fact, in this thesis we have the purpose of demonstrating the applicability of the benchmark prototype by conducting case studies to measure and compare the security of real web serving systems. Additionally, this thesis

and the research work supporting it provide to the community results, tools and an increase in knowledge concerning security at general, and in web serving systems in particular.

As mentioned, our benchmark metric (SBench) is estimated based on the security risk of vulnerabilities present in the system under benchmark. This is done by computing the risk related to vulnerabilities that are already discovered (known vulnerabilities) and by estimating the effect of not-yet discovered vulnerabilities (unknown vulnerabilities) on the system. This in fact corresponds to the two parts of our security benchmark methodology: static and dynamic parts. The assessment of the risk of known vulnerabilities (static part) corresponds to a static analysis of the target system and uses the knowledge about the impact and exploitability of vulnerabilities discovered in the field for that system to measure the security risk. These known vulnerabilities are obtained from two sources: (i) public repositories such as vulnerabilities databases and specialized web sites (e.g., (NVD 2014; OSVDB 2014; US-CERT 2014)); and (ii) results from security tests usually proposed by security experts. One important aspect is that vulnerability impact and exploitability are estimated considering the criteria defined by the Common Vulnerability Scoring System (CVSS) (Mell, Scarfone, and Romanosky 2007): this vulnerability framework has been widely used by large enterprises to characterize the risk of software vulnerabilities.

The assessment of the effects of unknown vulnerabilities (dynamic part) corresponds to an experimental approach where robustness attacks are conducted to observe the behavior of the system. This approach is properly detailed in Chapter 3, but it is worth pointing out that we do not propose a way to identify unknown vulnerabilities. Our experimental approach (already applied in dependability benchmarks to test the tolerance of system to software faults) stress the system with attacks, observe the impact of these attacks, and then estimate the security risk in case if these attacks were successful (i.e., the attack compromised at least one of the security attributes of confidentiality, integrity, and availability). In practice, this is done using two complementary steps: (1) stressing the system with malicious input parameters and multiples attacks (e.g., Denial of Services attacks, Buffer Overflow) directed against components that interact with end- users; and (2) mounting attacks against representative vulnerabilities that are injected in a component that is not included in the benchmark target (but interacts with it). By representative we mean the injection of vulnerabilities that are usually found (and consequently more exploited) in the target system.

The purpose of injecting vulnerabilities and attacking them is to anticipate if a security breach (a successful attack that leads to a security compromise) in a

component may affect the security of the whole system under benchmark. For example, by injecting vulnerabilities in the web application (it is plausible to assume that applications may have vulnerabilities) we can assess if the attacks launched over such vulnerabilities can compromise the system. One important aspect here is that the vulnerability and attack injection approach is actually the technique that allow us to assess the effects of unknown vulnerabilities to the system. As attacks exploit vulnerabilities injected in a component that is different from the benchmark target (i.e., the component is outside the perimeter of the benchmark target), any security compromise of the benchmark target during the execution of such attacks was caused by the presence of one or more unknown vulnerabilities (weak points). This idea was already applied in fault injection field to assess the behavior of fault tolerant system in the presence of erroneous software components and the goal here is to apply this concept to the security field, using the attack injection technique proposed in (J. Fonseca, Vieira, and Madeira 2009) as detailed later on.