What Went Wrong

Defining a plausible, quantitative evaluation of the actual behaviour of security systems that one expects remains an unavoidable challenge within the security community [7], [9], [10], especially in today’s computing where cloud and ubiquitous computing, mobility, and constrained resources are becoming normal features for computing devices and platforms. Most current evaluation methods suffer from one major limitation: the inability to capture the actual strength of a security system. To do so, we need to arrive at a clear abstraction of a computing system and definition of a failure model, before any system-level representations and analysis methods can be established. Then, we need to develop models that can sensibly capture and bound the effect of security controls and impact of associated failures using a common model foundation, facilitating dual, confronting views of security systems. Doing so will allow consistent, structural semantics to be defined, so quantifiable performance measures of security controls, at any stage, from goal setting through design to operational lifetime, can be established.

We argue that two flaws have led to the above limitation, which is the inability to capture the security element itself. First, the inheritance of some irrelevant notions from other modeling approaches into security models renders the latter unable to capture confronting parties (i.e., a game- theoretic paradigm), dynamic progression, and the human factor. Second, the center of focus of model development in security studies is extremely misaligned. That is, it is either too specific or too general.

Inheritance issues: In our view, the aspect of inherited issues has appeared as a result of extending

some modeling semantics from other engineering fields that do not necessary lead to a successful application when adopted in security modelling [22], [23]. We present three features that differentiate security models from other models. First, capturing two competing parties, i.e., game-theoretic paradigm, onto the same model foundation is intrinsic to security modelling to reflect the adversarial chase between both the protection system (i.e., security system) from one side and the failure system (i.e., failure sources, whether malicious such as attackers or nonmalicious such as accidents) on the other side, somehow as a battlefield. In contrast, capturing this paradigm is not common in other modelling endeavors. That is, it is either not applicable or simplified and modeled in isolation when applicable.

Second, the progression in a security system is dynamic and multi-directionally evolving in nature once the system is operational. This setting makes security start first with continuity and resilience in

mind against failures. To deliver security, comparable security strength must be implemented across system controls and security boundaries, and security principles must be jointly realised in the face of failure as a chain while in operation. Thus, a security system might, for instance, require the redesign of certain controls due to a recent technology, auditing, or enhancement, and a recovery of others due to a recent attack, all in the same time and in a mixed order, dictated by the current state of the security system. In contrast, the progression in other systems usually proceeds in steps or increments with a clear start and an end. This setting is much less volatile and more structured than in security.

Third, the human factor in the security context is the most important element that affects security behaviour, representing an intrinsic element for both protecting and breaking security. Thus, its inclusion when building models becomes a necessary condition. In contrast, other contexts do not normally reflect the human factor as an identifiable element among model components.

Several examples exist that support these observations. For instance, the noise and interference in communication networks, representing the confronting party per se, are commonly modeled in isolation by known distributions. That is, the behaviour of the noise and interference does not intentionally change to break the flow of packets in an adversarial manner. The delivery of packets is also governed by a sequenced layered architecture, such as the TCP/IP protocol stack, without actively involving human behavior. Likewise, the semantics used in the software engineering capability maturity model, such as defined in [85], particularly maturity sequencing qualitatively, not directly capturing confronting parties, and not directly capturing human involvement, are not too successful when extended to security maturity models.

Coverage issues: The second aspect of the limitation above centers around models being too extreme

in their application scope. To a large extent, security evaluation models we have today are often either too specific, especially for quantitative models, or too general, especially for qualitative ones.

Being too specific is basically due to the limited coverage in the case of the quantitative models, as they cover partial areas of the system, and therefore, cannot capture the behaviour of the overall system [7]. This limitation applies to many examples, such as the approaches presented in [3], [9], [54], [86] to evaluate security controls and model some attack behaviours, and approaches in [24], [25], [26], [27] to address the security dilemma from a game-theoretic perspective. Hence, the usefulness is limited to the specific cases where assumptions made may hold.

On the other side, being too general is basically due to the qualitative nature in the case of the comprehensive models, such as [12], [39], [42], [43], [44], [45], [46]. These models can only reflect the existence and adherence to a predefined set of controls. Therefore, they reflect a high-level view of the system, measuring how well a security process is implemented [7], [9], [23], addressing organizational acceptance issues [10].

So, the limitation exists mostly with respect to the breadth level for the quantitative models and to the depth level for the qualitative models. Consequently, either way, minimal inference can be drawn about system-level situational security states.