In the previous section, several evaluation frameworks for ISO 27002 are described. A comparison is done between the evaluation frameworks based on some aspects including the requirements for the new evaluation framework. The following six aspects are used:
• ISO 27002 compliance
This aspect describes if the measurement in the evaluation framework contains a check for the ISO 27002 security controls so that it is compliant with the ISO standard. This means that all aspects of the ISO standard have to be covered.
• Flexible
This aspect describes if the measurement in the evaluation framework has the possibility to be used in different situations (e.g. small/large organizations) and different type of checks (e.g. partly and fully checkable).
• Measurable
This aspect describes whether the measurement in the evaluation framework can objectively be measured by an auditor.
This aspect describes if the measurement in the evaluation framework is easy to answer (e.g. unambiguous) and the evaluation framework is small (in short time doable).
• Market conformance
This aspect describes whether the results of the measurement are
represented into a market conform end mark. In other words: this means the result tells if the organization implemented the security controls better or worse than other organizations.
• Method
This aspect describes the method that is used in the evaluation framework.
The new evaluation framework aims to satisfy the first five aspects.
All the mentioned research papers had some differences with the master thesis research as clarified in the table below.
Table 14 Comparison approaches
ISO 2700 2 co mp lia nc e Fl ex ib le Me as ur able Lig ht w eig ht M ar ke t c on fo rma nc e Me th od
Approach of Karabacak [41] √ √ / X √ √ X Questionnaire Approach of Wright [34] √ √ √ X X Measurement
of facts * Approach of Bandopadhyay [37] √ √ / X √ X X Questionnaire Approach of Praxiom [43] √ X X X X Questionnaire
* This whitepaper describes evidence-‐based measurements to check the effectiveness of the current implementation of ISO controls. This result of the measurement only indicates whether the
organization improved in comparison with the last measurement. The method measures some specific facts inside the organization to show the ISO compliance.
The second aspect was flexibility. The most flexible approach is from the
approach of Wright where each ISO control has a specific measurement and does not depend on other measurements. As an organization you can select only the measurements that are applicable. The approach of Karabacak and the approach of Bandopadhyay also have flexibility, but are both more restricted. These two research papers [37][41] could measure a part of ISO (by headings), but it leaves no free space for extra ISO compliance actions and exceptions. The approach of Praxiom is not flexible, because an organization has to check the ISO standard fully for getting the end result.
Furthermore the evaluations that the frameworks produce have to be
measurable. The approaches of Karabacak and Bandopadhyay have questions and predefined answers, which all are objectively measurable. The approach of
Wright is different, because it is based on measurements on facts. An example of fact-‐based measurement is: ‘in the last year, how much time is system x down?’. However, the approach of Praxiom consists of several questions that are difficult to answer, because the aspects (e.g. performance, suitability) that were asked are not specific measurable. An example is ‘Do you improve the performance of your ISMS?’. The possible answers are: yes and no. For an employee it is hard to answer this question. The employee can wonder: what does my organization have to improve to have a better performance? How can I measure that? The answer is not objective, because an employee could have other opinions on it.
The fourth aspect is how lightweight the approach is. These research papers have a large difference on the number of questions or measurements. Besides that some have predefined answers. Predefined answers have the advantage to make it easier to compare the answers from different organizations. A
disadvantage is that the correct answer for an organization might not be given in the predefined answers. The approach of Karabacak is the only evaluation
framework that has maximal one question for each ISO control. This means that in the ISO 27001:2005 version, the number of questions is in total 133. An additional benefit of this evaluation framework is that it has several predefined answers. All other frameworks contain more questions. Further, all others have predefined answers except the approach of Wright.
Another requirement for the new evaluation framework is that the result of the evaluation framework shows how market conformant an organization is. All evaluation frameworks show the results of the framework in percentage of ISO compliance instead of market conformity.
The last aspect for the comparison is which methodology was used in the evaluation framework. In three of the four cases, a questionnaire is used. Only one uses something else, namely fact based measuring like how much time the system is down. In this case the evaluation framework does not base the results on answers of the employees, but on facts that can be measured. Measuring the implementation based on facts would be ideal to get an objective way of
measurement. However, it will much more time consuming to measure the security of organizational processes with facts.
As mentioned at the start of this section, the new evaluation framework has to meet the first five aspects of the comparison: ISO compliance, flexibility, measurability, light weightiness and market conformity. None of the four checked evaluation frameworks did meet all of the requirements.