Theory

Figure 5.1: Research method for the ISSRM metrics elicitation

in an analysis table. This table recalls the measured concepts of the source and the associated metrics. They are aligned with the concepts of the ISSRM domain model and the metrics obtained during the GQM study. If a metric not identied with the GQM framework is found, it is necessary to evaluate its relevance. Sometimes, it can highlight a deciency in the GQM study, and thus the GQM models are reviewed and improved considering this new issue, or a justication for the exclusion of the metric shall be given. Conclusions about the metrics of the source and their dierence with regards to our metrics are nally provided. The tasks composing Step 2 of the research method are performed iteratively for every selected source of the literature.

Once the literature is completely surveyed, leading to the GQM models in their last version, the nal set of metrics is introduced in the ISSRM domain model as at- tributes of the classes. Some complementary explanations are also provided, proposing denitions for each metric, and an example of their use is proposed.

5.2 Theory

In this section, we introduce some theoretical concepts and methods used in this chapter. First, risk estimation is presented, showing the dierent categories of risk

estimation in ISSRM. Second, the GQM approach, used in Step 1 of the research method, is introduced. Finally, the ROSI notion is explained, in the aim of identifying the underlying objectives to be used for the application of GQM on the ISSRM domain. 5.2.1 Introduction to risk estimation

As discussed in Section 2.1, risk analysis consists in identifying and estimating the dif- ferent risk components. Regarding risk estimation, various approaches exist [ISO08, AS/04]. The same approaches are adopted for measuring the asset- and risk treatment- related concepts. They can be classied in the following categories: qualitative, quan- titative, or a combination of both, commonly called semi-quantitative estimation:

• Qualitative risk estimation

Qualitative risk estimation approaches are currently the most widespread in the industry [ENI06, DCS04b, CLU07b, AD01b]. They propose a scale of levels for qualitatively describing the concepts to measure. These scales are ordinal scales [FP97]. An advantage of qualitative estimation is its ease of understanding by the sta involved in the estimation task, while a disadvantage is the dependence on the subjective choice of the scale [ISO08]. Examples of qualitative scales, for a nancial cost or an unavailability length, are shown in Table 5.1.

Table 5.1: Examples of qualitative scales

Level Financial cost Unavailability length

1 Low Short

2 Moderate Moderate 3 Important Long 4 Very important Very long

• Quantitative risk estimation

Quantitative risk management approaches propose to `precisely' measure each concept of ISSRM. By `precisely', we mean through ratio or absolute scales2 _in

terms of the scales provided in [FP97]. The quality of estimation depends on the accuracy and completeness of the numerical measures and the validity of the used models [ISO08]. For example, a nancial cost will be estimated in terms of  [Ins03] or an unavailability length will be reported in hours, as depicted in Table 5.2. Most often, historical incident data of an organisation or of a sector, like provided by the CERT3_{, is used to provide quantitative indications. Naturally,}

an advantage of such an approach is its accuracy, but its cost and the lack of useful data are the main disadvantages [ISO08].

• Semi-quantitative risk estimation

In semi-quantitative estimations, ordinal scales are also given to estimate con- cepts, but based this time on quantitative values. In other words, a quantitative

2_{No examples of interval scales have been found for the ISSRM domain, but they may theoretically be}

relevant quantitative scales.

5.2 Theory 119 Table 5.2: Examples of quantitative scales

Financial cost Unavailability length Amount of money in € Length in hours

scale is reduced to a discrete scale, to become an ordinal scale [FP97]. The objective is naturally to produce in a cost-eective manner more precise results than those obtained by qualitative approaches. However, the estimation remains naturally less accurate than quantitative estimation. A particular care should be given to the denition of the scales, to keep relevance in the equivalent levels and to obtain useful information about the relative criticality of the studied concepts. Examples of semi-quantitative scales are proposed in Table 5.3.

Table 5.3: Examples of semi-quantitative scales

Level Financial cost Unavailability length 1 Loss < 1000$ Unavailability > 1 week 2 1000$ < Loss < 5000$ 1 day < Unavailability < 1 week 3 5000$ < Loss < 10000$ 1 day < Unavailability < 1 hour 4 Loss > 10000$ Unavailability < 1 hour

In a given method, existing or dened by an user, the dierent approaches can be mixed, depending on the concepts analysed and the objective to be reached. For example, qualitative estimation could be rst used to obtain a coarse grained estima- tion of risks, and later, a quantitative estimation could provide further information on major identied risks. Finally, it is interesting to note that some approaches focus on concept identication, whereas they provide very few guidelines for risk estimation all along their process [Bun05c, AD01b]. These approaches are more directing towards reaching a `reasonable' security level and a complete identication, understanding, and coverage of risks.

Regarding the preceding approaches, it is also important to distinguish the scope of the metrics we identify. In our context, a metric indicates the magnitude of a con- cept, according to a given dimension (security, cost, etc.), as appears in the denition proposed in [ISO05a]. A metric should thus be dierentiated from what we call in- dicators, which in our context, are variables that can be set to a prescribed state based on the results of a process or the occurrence of a specied condition [ISO05a]. The main dierence between a metric and an indicator is that the rst one is ordered and the second one not. Indicators are based on nominal scales [FP97], proposing a non-ordered classication scheme. An indicator is something that gives information about a particular situation, but does not estimate the magnitude of a concept (e.g., Is the motivation of an attacker based on the nancial interest or on the challenging aspect? Is the attack method accidental or intentional?). In our context, indicators generally help to estimate the dierent metrics.

5.2.2 The GQM approach

GQM's basic idea is deriving metrics from measurement questions and goals. The GQM method was originated by V.Basili and D.Weis, as a result of both practical experience and academic research [SB99]. By now, it is widely used in a number of contexts, like in the aeronautics or telephony industry [BCR94, Kil01].

In the GQM approach, measurement is dened in a top-down fashion [BCR94]. GQM is based upon the assumption that, for an organisation to measure in an ecient way, it must specify the goals for itself and its projects rst, then it must trace those goals to the data intended to operationalise them. Finally, it must provide a framework for interpreting the data with respect to the stated goals [BCR94]. The result of the application of the GQM approach is the denition of the measurement system targeting a particular set of issues. The outcome is a GQM model that has three levels:

1. Conceptual level, called GOAL level: A goal is dened for an object (like a product, a process or a resource).

2. Operational level, called QUESTION level: A set of questions is used to char- acterize the way the assessment/achievement of a specic goal is going to be performed.

3. Quantitative level, called METRIC level: A set of data is associated with every question in order to answer it in a quantitative way.

Therefore, a GQM model is a hierarchical structure (Figure 5.2) starting with a goal. The goal is rened into several questions. Each question is then rened into metrics. The same metric can be used in order to answer dierent questions. More information about GQM can be found in [BCR94, GC87].

Figure 5.2: Example of GQM model (extracted from [BCR94])

5.2.3 The ROSI concept

The main outcome of the ISSRM process, and one of the main motivations, is to obtain the best ROSI [ISA06, SGF02]. CLUSIF proposes a state of the art around the notion of ROSI [CLU04b]. There is no clear consensus about the denition of ROSI, but two proposals are the most used. The rst one relates security costs to the expected