3 Construction of the questionnaire 36
3.2 Phase 2: the full questionnaire 41
During the second phase, the trial questionnaire is extended to the full
questionnaire. The goal of this questionnaire is to cover all 114 ISO controls and use the feedback of phase 1 to improve the questionnaire. After finishing phase 2 several validations are scheduled:
• Validation inside SIG (similar to phase 1 validation only for full questionnaire)
• Validation with consultants
During the validation, the same aspects as in phase 1 of the development are validated. In these validation sessions, there is a special focus on these three aspects:
• Is the questionnaire useful?
• Is the questionnaire easy to answer?
• Could the questionnaire be answered in a short timeframe?
Further completeness is taken into account as extra information to improve the questionnaire on missing aspects.
In this phase, two validations with two different groups are done, because both groups have different functions. The validation inside SIG is done with people who have the knowledge of the ISO 27002 implementation inside SIG, so they can answer the questions. The validation with consultants is done, because those consultants will help clients fill in the questionnaire in the future consultancy. It is therefore important that the consults have experience with the questionnaire as well.
3.2.1 Development
In the validation of the trial questionnaire, it was shown that answering 25 questions already takes a lot of time. Therefore it was difficult to fulfill the design constraint of maximal one day to answer the full questionnaire. This resulted into an extra design constraint, a maximum of 60 questions. Of course having less questions is still preferable. This new constraint meant that the
questionnaire could not have 114 questions anymore. The rest of the design constraints remained intact.
In the first phase there were 25 questions for 38 controls. When this data is extrapolated then there would be 75 questions for 114 controls. This means that the design process has to be adjusted to be able to combine more ISO controls for meeting the project constraints. In some cases more ISO controls were able to be combined than in phase 1, so this made it possible to satisfy the design
constraints. Two ISO controls can be combined if they fulfill the same two requirements as used in phase 1:
• The ISO controls share a related topic (e.g. both are questions about organization assets)
• The person required to answer the question should have knowledge about both ISO controls.
A simple example of a combination of two ISO controls can be found in Table 16. Eventually, a questionnaire with 50 questions was created that fully cover the 114 ISO controls.
3.2.2 Validation inside SIG
The questions of the full questionnaire were divided under the five departments ISO 27002 implementation inside SIG (CSO, IT, Lab, HR, Office). During the validation there were in total five sessions (one for each department), where each session took 30 minutes. Only the IT department had a one-‐hour session, because they have more questions. Before the session, the questions were given to the employees. These questions had to be answered before the session, because the validation session is only intended for receiving feedback.
For feedback some statements are made again, to which the participants needed to give their opinion. This measurement was to determine how good or bad the questionnaire was overall. There are two statements described:
• The questionnaire is useful
For each statement the employee can give an answer from 1 (totally disagree) to 5 (totally agree). Furthermore there is an option to place some comments
concerning each aspect.
The assessment of the two statements indicates that the full questionnaire is accepted as a good option for the service inside SIG. They agreed on that it was useful and easy to answer. The results of the assessment are shown in Figure 8. The figure shows for each statement (useful, easy to answer) the collective results of the five sessions, which in each session a specific group if questions are given that are linked to the task of the participant. Besides the measurement the two opinions, there was also feedback supplied by the participants to improve the questions.
Figure 8 Validation inside SIG phase 2 – results
Everyone agreed that the (current) questionnaire is useful. This is a positive indication for the future. If people think the questionnaire is useful then there is a higher chance for it to be used properly.
Most of the participants also said that the questionnaire was easy to answer. Only two of the five were less positive. One person did not think that all
questions were easy to answer and this resulted in an answer between agree and neutral. This is showed in Figure 8 in the 0.5 agree and 0.5 neutral. The reason for this result was that in some cases the participant had to ask some small questions to get it clarified.
The IT department disagreed about the easiness. The reason to disagree on easiness was that the questionnaire is generally formulated. This means that if there is a question about the systems then the participant has to think of: which systems do we have? What do we do for all those systems? And so on. This means that you have to create an overview for yourself which systems you have and which security controls are implemented. This extra step makes it harder to give the answers. This is especially the case in the IT department questions, so that is why other participants (lab, office, CSO, HR) did not encounter this
1 4 0 0 0
Useful
1 2.5 0.5 1 0Easy to answer
Totally agree Agree Neutral Disagree Totally disagree
problem. In addition the time to fill in the full questionnaire was under in the predefined requirement (1 day).
3.2.3 Validation with consultants
A validation session with the consultants of SIG is done. Consultants are
employees that give professional advice on a specific subject. The selection of the participating consultants was based on their special security consultancy
capability. In total there were four consultants chosen. For each of those four consultants, a 30-‐minute validation session is done. Before the session, the full questionnaire is given to the consultants, so they could take a look and write some notes or feedback.
For feedback the consultants fill in their opinion on two statements about the quality of the questionnaire.
• The questionnaire is useful
• The questionnaire can be easily answered
For each statement the employee can give a rating ranging from 1 (totally disagree) to 5 (totally agree). Furthermore there is an option to place some comments.
The results of the small questionnaire indicate that questionnaire is accepted as a good option for the service inside SIG. All consultants agreed on the fact that the questionnaire was useful. However, they encountered some difficulties with the aspect ‘easy to answer’. The main solution to improve the easiness is to write a little introduction at the start of the questionnaire. The results of the
assessment are shown in Figure 9. The figure shows for each statement (useful, easy to answer) the collective results of the four sessions.
Figure 9 Validation with consultants – results
Summarizing the feedback from both security control owners and the
consultants, all participants agreed on that the questionnaire was useful. This results confirm the agreement on usefulness of the validation inside SIG.
Another outcome was that the consultants gave a lower mark to easiness. There were three major comments for the lower mark.
1 3 0 0 0
Useful
0 1 2 1 0Easy to answer
Totally agree Agree Neutral Disagree Totally disagree
-‐ More structure in the questionnaire based on for example subjects, and functions. This was already known, but not implemented yet.
-‐ An introduction before the questionnaire for the participants.
-‐ A document with information how a consultant could get the end result (star rating and actions).
Possible explanation for the differences between the validation inside SIG and the validation obtained from the consults is that the participants of the validation inside SIG have seen the questionnaire (in the trial questionnaire validation) earlier, so they are more familiar with questionnaire. Furthermore, the security control owners should be able to understand the questions better since most of the questions are about their job functions. Besides that, consultants has a different type of perspective, because they must do the consultancy and do not have to answer the questions. They view the questionnaire from a different perspective.
The reason that one of the consultants chose for the option disagree was that the questions are quite general. This is the same feedback as the IT department has given. There was some discussion about the subject and the result of the
discussion was that the organization has to describe the actions for the ‘weakest’ removable media. The weakest removable media is the removable media that has the lowest implemented security controls to protect the content. In case this is mentioned at the start of the questionnaire then it would be okay according to the consultant.