3 Construction of the questionnaire 36
3.1 Phase 1: the trial questionnaire 37
During the first phase, the trial questionnaire was created. The idea of this trial questionnaire is to create a questionnaire of 25 questions. SIG has five
responsibility functions that are involved to the ISO 27002 implementation, namely: Management (CSO), IT, Software Development (Lab), HR and Facility management (Office). For each responsibility function there are five questions to answer to get insight about how good the trial questionnaire performs. The five groups are used for the validation inside SIG.
3.1.1 Development
During the development of the earlier prototype [6] it was already noticed that the checkpoints were not the right choice because of the large volume of the earlier prototype. In the earlier prototype it was tried to use questions with predefined answers instead of those checkpoints to reduce the volume. The implementation of questions satisfies the reduction expectations. The reason to use a questionnaire is a result of the satisfied expectations and the technique is the most used technique in other evaluation frameworks (see Section 2.3). The other technique that was used in other evaluation frameworks was measuring of facts. The advantage of this technique is that it produces objective result, but this technique is not chosen for several reasons. One reason is that fact-‐based
measuring assumes that an ISMS has already been well implemented. This does not have to be the case. Besides that the measurement is more dependent on the way the ISMS has been implemented, so when this technique is used then the evaluation framework has to be adjusted more to new technologies and ways to implement the ISO controls.
Another point what is learned from the earlier prototype was that the evaluation framework will increase very easily in size when too much details of the ISO controls are used.
After learning from the earlier prototype, there were some design constraints described for the trial evaluation framework:
• A maximum of one question for each ISO control
• Questions with pre-‐defined answers for making it easy to analyze the results
• Questions are posed in such a way that every organization could answer it, so the results can be compared.
• Answering the full evaluation framework may only take one day at maximum
During the start of the research, the trial questionnaire with the 25 questions is created. At first instance, it was tried to create one question for one control, but it was noticed that it was possible to reduce much more. At the end there were 25 questions, which were based on 38 ISO controls. There are 114 ISO controls
in total, so approximately a third of all ISO controls were covered in the trail questionnaire. One example of a question that was created can be found in Table 15. The question is asked in a relatively open manner, because otherwise
multiple questions have to be used to get the same amount of information. To help the participants understand the question and to get them to find the correct answer, pre-‐defined answers were used in the questionnaire.
Beside the four design constraints, the possibility to add some extra comments is created, because in some cases people prefer to give a bit more details or wants to describe an exception.
Table 15 Example -‐ question trial questionnaire
ISO control 9.2.1 User registration and de-‐registration
Control context A formal user registration and de-‐registration process shall be implemented to enable assignment of access rights.
Question Are there any actions used to manage the user IDs? Answer(s) ☐ No
☐ Yes:
☐ Usage of unique user IDs
☐ Immediately disabling or removing user IDs of users who left the organization
☐ Periodically identifying and removing or disabling redundant user IDs
Comments and/or extra actions:
………..…………..…………..…………..…………..…………..…… ………..…………..…………..…………..…………..…………..…… ………..…………..…………..…………..…………..…………..……
It is considered to do a selection of questions in the evaluation based on a risk assessment. In this way the trial questionnaire is more flexible and the questions are more relevant to the client organization. However, if you use different
questions for evaluations then it is hard to compare the results of the
evaluations. This means that SIG could only say how well is implemented, but not how market conform the implementation is. Because comparability and
objectivity is wanted, there is chosen not to pre-‐select questions based on a risk assessment. The same approach is taken in all evaluation models of SIG.
There was a design constraint of one question for one ISO control, but it is preferred to have fewer questions. That is why there is chosen to combine several ISO controls in one question. It was possible to combine ISO controls if the following requirements hold:
• The ISO controls share a related topic (e.g. both are questions about assets)
• One person could have the knowledge about both ISO controls.
A simple example of a combination of two ISO controls can be found in Table 16.
Table 16 Two combined ISO controls
ISO controls 6.1.3 Contact with authorities
6.1.4 Contact with special interest groups
Question Is there a procedure which says when, how and which organizations have to be contacted?
Answer(s) No Yes, for:
Authorities (e.g. law enforcement, regulatory bodies, supervisory authorities)
Special interest groups (e.g. specialist security forums) Suppliers (e.g. source development organization) Clients
Comments and/or extra actions:
………..…………..…………..…………..…………..…………..…… ………..…………..…………..…………..…………..…………..…… ………..…………..…………..…………..…………..…………..……
3.1.2 Validation inside SIG
The first phase validation was done for the trial questionnaire. This trial
questionnaire consists of 25 questions, which are equally divided under the five functions (Chief Security Officer, Lab, IT, Office and HR) for the ISO 27002 implementation inside SIG. During the validation, five one-‐hour sessions are done (one for each department).
During a session the employee that is responsible for the implementation of the security controls has to fill in the answers of the five questions that are related to their department. Further they have to give feedback on those questions. For feedback there were three statements described, where participants could agree or disagree on:
• The question is useful
• The question can be easily answered (no ambiguity and so on) • The question is complete
For each statement the employee can answer 1 (totally disagree) to 5 (totally agree). There is also an option to give comments on each aspect. The assessment of the three statements has to be given for all five questions so that it is know which questions had to be improved.
The assessment of the three statements for the 25 questions on the aspects usefulness, easiness and completeness indicated promising results for the full questionnaire if it was created similar to the trial questions. All three aspects obtained more than 50% of the answers agree or totally agree. Besides that in all three aspects the answers disagree and totally disagree are lower than 25%. Of course, there is room for improvement. The results of the assessments are shown in Figure 7. The questions the participants were asked had no overlap.
There is feedback given to improve the questions. The most notable remarks during the feedback will be discussed.
A problem encountered during the validation was that filling in the questions and the discussion were mixed. This made it harder to measure the exact time of filling in the questionnaire, although it was noticed that it took a lot of time. In the second phase of the development, it was needed to make sure that the questionnaire was reduced even further or something different to reduce the time needed to answer the evaluation form.
It was also noticed based on the given answers and the discussion that the right aspects of organizational process were measured.
Figure 7 Validation inside SIG phase 1 -‐ results
3.1.2.1 Usefulness
Multiple times it was mentioned that some questions or possible answers are only useful for a specific type of organizations (e.g. only banks). Possible
differences in companies or other organizations can be for example reliance on software, security level, location and whether the building is shared with other organizations. An example answer of a question was the AIVD screening for a new employee. This AIVD screening is only necessary in a highly secured environment. In ‘standard’ organizations there have to be some checks, but the AIVD screening is not needed for most organizations. The above-‐mentioned problem where questions and/or answers are not applicable for the
organization, the participant described the not applicable question as ‘totally disagree’. However, the same question could be potentially useful for other types of organizations. This means that the ‘totally disagree’ mark does not mean that the question has to be removed from the questionnaire.
3.1.2.2 Easiness
In the feedback on the questions whether the questions were easy to answer, it was shown that multiple times the participants preferred to have some
8 10 6 0 1
Useful
6 8 5 6 0Easy to
answer
5 10 7 3 0Complete
Totally agree Agree Neutral Disagree Totally disagreeexamples. For instance one question about contact with authorities and special interest groups would become much easier to understand by using examples. The question asks if there is a procedure which specifically say when, how and which organizations should be contacted. The initial two options are yes and no, but when you choose yes you can choose out of several options: the authorities, special interest groups, third party services and contractors. Somebody who has to fill in the questionnaire does not have to know what is in those four groups. The usage of the examples makes it clearer how to answer the question.
Besides using more examples, in some questions there were still some words that caused unambiguity. For example the word ‘system changes’ is used, which led to a discussion what is and is not covered under system changes.
3.1.2.3 Completeness
The three questions where participants were disagreeing about the completeness had two causes: missing examples (see Section 3.1.2.2) and missing predefined answers to the questions. One of those three questions was about the terms and conditions in contracts. At this moment the trial framework distinguishes two groups: employees and contractors, but there are more. In the organizations, there are for example also interns, self-‐employed workers and employees of an employment service provider. These groups may have different terms and conditions or regulation around the terms and conditions, so these are special cases that some organizations did not think about. The second question that missed predefined answers was the question about system security testing. The question has some general answers about how the system security test is done and how it is organized, but it is preferred to have more detail:
white/grey/black box testing, inside or outside the organization, code review or runtime test. The third question was also missing some predefined answers just like the second question.