In Section 3, the construction of the questionnaire is described, but a translation from the answers of the questionnaire to a rating and actions is still to be
defined. In this chapter this aspect of the evaluation framework is described.
The translation from the chosen answers to an actual rating is based on how market conform the chosen answers are. This means that the rating scheme should be kept up to date. During the research process, the validations were tested in cooperation with only a limited number of external organizations. This made it difficult to give a market conformity rating at this stage in the project. The actual rating has to be further calibrated by SIG employees once more data points are available.
4.1 The SIG star rating
There are several options to calculate the SIG star rating. One option is to use the techniques that are already used inside other SIG services (‘The SIG approach’) [46]. However, there are other ways to do it. Another option that is considered was the risk approach, which is more adjusted to the organization situation. In the following two sections both approaches are described. In case of the evaluation framework, there is chosen for the SIG approach over the risk approach. The reason for this choice is that the added value of including risk in the approach is too low in comparison with the cost of time and the complexity. Besides that it would be difficult to compare organizations and the SIG approach is more consistent to other SIG services.
4.1.1 The SIG approach
In the research of the earlier prototype [6] it was decided to do a simple rating scheme to get to the SIG star rating. In this rating scheme, the answers for each question are classified in a specific box. There are five boxes from 1 star until 5 stars, where the 1-‐star box is easier to satisfy and the 5-‐star box is the most difficult one. An example of this is shown in Table 17. This approach uses
maturity levels of grading the question. This means that for two stars all answers from the one star box (answers 1 and 2) and also the answers of the two stars box (answer 5). The classification of the answers into the boxes are not given to the participating organization on beforehand due to the possibility to choose specific answer to get a specific result.
Table 17 Example – boxes
Stars Needed answers ✭ Answer 1, Answer 2 ✭✭ Answer 5 ✭✭✭ Answer 3, Answer 4 ✭✭✭✭ Answer 6 ✭✭✭✭✭ Answer 7
The SIG star rating is based on which boxes are met in the questions.
In the earlier prototype, the SIG star rating is made by taking the average of the box star (1 to 5) for each question. For example you have two questions: one question has all required answers to be in box 4 and the second question has the answers for box 2. The SIG star rating is then (4+2)/2 = 3.
The same technique as in the earlier prototype could be used, but it was found that there are some problems with the technique [44]. The problem was that when you take the average with many inputs (in this case 52 questions), the average would be coming to the middle. This means that mostly a 3 star rating is given. This is not what is desired for a security check.
This could be solved with a transition table. In SIG they use already another technique (transition technique), which is quite similar as the used technique of the earlier prototype. The only real difference is that they use a transition table instead of the average. This could be a good solution for the above-‐mentioned problem. This transition technique puts answers of the questions into five categories:
• Level 0 (Does not meet requirements for any level) • Level 1
• Level 2 • Level 3 • Level 4
This level system is almost the same as shown in Table 17, where each answer is in a specific box (or in this case a level) and for a specific level all answers of the lower levels have to be met. Depending on the number of answers, a question is in a specific category (see Table 18).
Table 18 Example -‐ level system
Level 1 Level 2 Level 3 Level 4
Question 1 Question 2 X X Question 3 X Question 4 X X X X Question 5 X X X
After classifying the questions into five categories, a transition table is used. The used transition table can be seen in Table 19. The percentages mean the amount of questions has to be at least that level or higher. The example data set of Table 18 is used to illustrate the idea. There you see four crosses in the column ‘level 1’, but there are five questions. This means that level 1 has a compliance of (4/5)*100 = 80%. If you also check this for the other levels, you will find the following results: level 2 at 60%, level 3 at 40% and level 4 at 20%. In the table below, it is possible to look up and see that all the requirements are met for a 3 star rating, but not for a 4 star rating. This means that the example has a 2 star rating.
Table 19 Transition table
compliance compliance compliance compliance ✭ 30% 0% 0% 0% ✭✭ 60% 35% 15% 5% ✭✭✭ 85% 55% 30% 10% ✭✭✭✭ 90% 80% 50% 15% ✭✭✭✭✭ 100% 95% 60% 20%
Not only the above-‐mentioned transition table is tried, but there are three scenarios used. It was checked which one fits the best:
• An easy to meet transition table: a table with low percentages to get a high star rating.
• An optimistic transition table: a table with high percentages to get a high star rating.
• A transition table that is in between the two transition tables mentioned above.
Firstly, the rating system is used to calculate the results of the questionnaire sessions of the three participating organizations (SIG and two external
organizations). The compliance percentages of those organizations can be found in Table 20. It is known that two of the organizations have an ISO certificate and all three have high security implementation. This meant that all organizations has to be at least three stars or higher. The three transitions tables were used and it was found that the transition in the middle has the most representative percentages. This transition table can be found in Table 19 and the SIG star rating results for the organizations are shown below.
Table 20 Results validations
Organizat-‐
ion Level 1 compliance Level 2 compliance Level 3 compliance Level 4 compliance SIG star rating
A 90,0 62,0 42,0 16,0 ★★★ (2.7)
B 93,9 83,7 51,0 16,3 ★★★★ (3.6)
C 94,0 76,0 36,0 12,0 ★★★ (2.8)
Besides the small transition table, a detailed transition table is made where the SIG rating can be determined more accurately (see Appendix B: Detailed
transition table). The results vary between 0.5 and 5.5. For each level there are six data points that are shown in Table 21. Between the data points, there is a linear function. In Figure 10, the piecewise linear function is given for Level 2.
Table 21 Six data points
Data
point Rate Level 1 compliance Level 2 compliance Level 3 compliance Level 4 compliance
1 0.5 30% 0% 0% 0% 2 1.5 60% 35% 15% 5% 3 2.5 85% 55% 30% 10% 4 3.5 90% 80% 50% 15% 5 4.5 100% 95% 60% 20% 6 5.5 100% 100% 100% 100%
Figure 10 Level 2 piecewise linear function
To know the exact grade, the following steps are done:
1. For each level, you check the position of the found compliance in the table. This gives you a specific rate (e.g. 3,5)
2. The end result is the minimum of the found rates in step 1. For example when you found the following ratings: 3,7; 3,5; 2,1; 2,2. Then the end result is 2,1
4.1.2 The risk approach
An option is to base the ratings on the risks of the organizations is called the risk approach. A bank for example has different risks than a telecommunication organization.
The risk approach uses weights for questions based on the risk. The weight can for example be set from one to ten. This risk is based on a risk assessment for the organization. The higher the risk, the higher the weight. Each different weight requires a specific percentage of implementation to be marked as ‘Good’/ ‘Ok’ / ‘Bad’. The higher the weight, the more difficult it is to be marked as ‘Good’.
The results of the questions (good, ok, bad) will actually be decisive for the final rating. This could be done with various techniques, like a point system where
goods give more points. At the end there is an overall rating.
This approach could also be used as described in the SIG approach, but when calculating the star rating you need to take the weights into account.
0 10 20 30 40 50 60 70 80 90 100 0.5 1.5 2.5 3.5 4.5 5.5
Level 2
Another implementation of the risk approach is possible, like for instance: • Adding more or less risk groups.
• Changing the weight factor of the implementation percentages based on type of organization. For example a bank would have to be more secure than a software development organization to get the same rating.
4.2 The actions for improving the security
Besides the SIG star rating, an organization will benefit from suggested actions to improve the security. This extra information will help to see what has to be done so that the vulnerabilities can be reduced to a minimum if the actions are
implemented well.
The answers of a questionnaire show the strong and weak points of the security implementations in the organizational processes. Based on those weak points, the necessary actions for the organization can be determined. In the overview of the questions with the associated category (see Table 18), you know to which questions the organizations score either good or bad. In case questions score badly, SIG could formulate some actions to help improve the company’s security. When there are many actions required then it is necessary to give an order for implementation, because an organization cannot implement everything at the same time. This order could be based on the costs, the likelihood that an incident related to the action occurs, the impact of the incident and so on. An example for such an action is that the organization has to use encryption on removable media.