There are many standards and evaluation frameworks that are related to this research. However as discussed in Chapter 2, none of those meet all
requirements (e.g. lightweight, measurable, flexible) that SIG desires. This research aims to meet all these requirements. However, it was a challenge to implement the combination of all requirements (e.g. lightweight and complete). Objective measurement was a difficult challenge even without the restriction of the other requirements. The final solution was created by looking at what is acceptable and what is not, coming to a compromise between the requirements. Ideally, the evaluation framework should be objective and quantitative, but in reality is always partially subjective and qualitative.
In an earlier prototype checkpoints were used that could be regarded as statements where either yes or no could be the answer. This concept does not work well in large questionnaires, because to obtain enough information a large number of checkpoints is required. The option to have several predefined answers to a question instead of only yes/no-‐checkpoints is in this case better. These predefined questions allow for a much smaller questionnaire. Predefined questions were used in this evaluation framework, but have to be used with caution. Using only predefined answers does not allow the interviewee to describe exceptions or give extra information. This possibly important
information would not reach the analyzers. For this reason comment lines were added to the questionnaire.
First a trial evaluation framework is created consisting of 25 questions divided over the five persons inside SIG that are responsible for the ISO 27002
implementation. The number of questions for each person was enough to verify whether the framework was designed with a good method. The validation inside SIG of this trial evaluation framework made it able to do a reduction to get a smaller, more compact evaluation framework. During this reduction several correlated ISO controls were combined. In the end, there was an evaluation framework with 52 questions.
In the validation sessions with external organizations, the questions are divided over three functions (CSO, head of IT, head of software development). All
organizations did not encountered problems with the division. In both external organizations, there was one person that could answer both the CSO part and the head of IT part. However, SIG does have separate employees for the two parts. Further it was noticed that these functions are connected with specific ISO chapters. This is very common, because in the ISO standard a specific type of controls are grouped together. These groups are associated with some tasks and processes, which are again associated with a specific function.
It was noticed that the questionnaire is more easily understandable to people who are familiar with the ISO 27002 standards, but other people should be able to understand the questions.
At the start of the research process, it was stated that this evaluation framework is especially for software development organizations and organizations that heavily rely on software. When looking back to this statement, it still holds that the full evaluation framework is suited for this group. However, if the
questionnaire part about the software development is left out, it could also be used for other type of organizations. This part about software development is around 15% of the full questionnaire.
Section 2.3 described four other evaluation frameworks. These evaluation frameworks have some similarities with the designed framework, for instance: the new evaluation framework uses questions and three of the other frameworks also use questions. They however also have big differences with the new
evaluation framework:
All frameworks except the new evaluation framework are based on an older ISO 27002:2005 version.
Furthermore the new evaluation framework provides a framework that is
lightweight. Three of the four frameworks ([34], [37] and [43]) are heavyweight. Only the approach of Karabacak [41] is lightweight, but that approach requires that multiple persons answer each question of the questionnaire. This means that the approach of Karabacak is also more time-‐consuming than the used approach in this master thesis.
Another difference is that this master thesis’ approach show how good or bad implementation of security processes in an organization is in comparison with other organizations (market conformance). All other approaches do not compare the results of organizations with each other, but they only check the level of implementation.
Future work can be done on this research in several topics.
• Refining the rating system based on more datasets than the used three datasets (SIG and two external organizations). The star rating has to be spread well (e.g. not all organizations have 4 stars). In case it is not spread well than the rating system has to be adjusted to allow for better
comparison with other companies. This could be done by using other percentages in the transition table or moving answers to another maturity level.
• Creating a tool that automatically calculates the star rating based on the answers of organizations. Possibly, the tool could give some suggestions of actions to improve. In a more advanced tool, the refinement of the rating system (mentioned in the previous point of future work) could be done automatically. This tool is a ‘nice-‐to-‐have’, because it helps reducing the analysis time.
• A research is required to investigate how trustworthy the evaluation framework and the results are.
References
[1] PricewaterhouseCoopers (2008). Information security breaches survey
2008: executive summary. Retrieved from
http://www.berr.gov.uk/files/file45713.pdf
[2] PricewaterhouseCoopers (2013). Information security breaches survey
2013: executive summary. Retrieved from
http://www.pwc.co.uk/assets/pdf/cyber-‐security-‐2013-‐exec-‐ summary.pdf
[3] Reuijl, A.; Koers, M.; Paans, R.; van der Veer, R.; Roukens, R.; Kok, C.; Breeman, J. (2014) Grip op SSD Het proces. Retrieved from
http://www.cip-‐overheid.nl/wp-‐content/uploads/2014/04/Grip-‐op-‐ SSD-‐Het-‐proces-‐v1-‐03.pdf
[4] Altena, J. (2012). ISO/IEC 27002 baseline selection. Master thesis, Nijmegen: Radboud University.
[5] German Bundesamt für Sicherheit in der Informationstechnik. (2007). IT
security guidelines. IT Security Management and IT-‐Grundschultz.
[6] Huijben, K. (2013). From ISO 27001 towards a flexible and light-‐weight
security evaluation framework. Research B project, Nijmegen: Radboud
University.
[7] McGraw, G., Migues, S., & West, J. (2013). Build Security In Maturity Model
V.
[8] Xu, H. (2013). ISO 27k controls & security process model. Not published. [9] International Organization of Standardization (2013). Information
technology – security techniques – information security management systems – overview and vocabulary. ISO 27000:2013
[10] International Organization of Standardization (2013). Information
technology – security techniques – information security management systems – requirements. ISO 27001:2013
[11] International Organization of Standardization (2013). Information
technology – security techniques – code of practice for information security controls. ISO 27002:2013
[12] International Organization of Standardization (2005). Information
technology – security techniques – information security management systems – requirements. ISO 27001:2005
[13] International Organization of Standardization (2005). Information
technology – security techniques – code of practice for information security controls. ISO 27002:2005
[14] International Organization of Standardization (2010). Information
technology — Security techniques — Information security management system implementation guidance. ISO 27003:2010
[15] International Organization of Standardization (2009). Information
technology — Security techniques — Information security management — Measurement. ISO 27004:2009
[16] International Organization of Standardization (2008). Information
technology — Security techniques — Information security risk management. ISO 27005:2008
[17] International Organization of Standardization (2007). Information
technology — Security techniques — Requirements for bodies providing audit and certification of information security management systems. ISO
27006:2007
[18] Lineman, D. (2013). ISO 27002:2013 Change Summary Heatmap. Retrieved from http://www.informationshield.com/security-‐
policy/2013/11/iso-‐270022013-‐change-‐summary-‐heatmap/
[19] PWC (2013). New releases of ISO 27001:2013 and ISO 27002:2013. Retrieved from
http://www.pwc.com.cy/en/publications/assets/iso27001-‐27002-‐ 2013.pdf
[20] Deloitte (2013). ISAE 3402 and SSAE 16 (replacing SAS 70):
Reinforcing confidence through demonstration of effective controls. Retrieved from http://www.deloitte.com/assets/Dcom-‐
Luxembourg/Local%20Assets/Documents/Brochures/English/2011/lu_en_ isae3402-‐ssae16_09092011.pdf
[21] National Institute of Standards and Technology. (2011) Managing
Information Security Risk. Special publication 800-‐39.
[22] National Institute of Standards and Technology. (2004) Standards
for Security Categorization of Federal Information and Information Systems. Federal information process standards publication 199.
[23] National Institute of Standards and Technology. (2008) Volume
I: Guide for Mapping Types of Information and Information Systems to Security Categories. Special publication 800-‐60.
[24] National Institute of Standards and Technology. (2008) Volume II:
Appendices to Guide for Mapping Types of Information and Information Systems to Security Categories. Special publication 800-‐60.
[25] National Institute of Standards and Technology. (2006) Minimum
Security Requirements for Federal Information and Information Systems.
Federal information process standards publication 200.
[26] National Institute of Standards and Technology. (2013) Security
and Privacy Controls for Federal Information Systems and Organizations.
Special publication 800-‐53.
[27] National Institute of Standards and Technology. (2011) National
Checklist Program for IT Products—Guidelines for Checklist Users and Developers. Special publication 800-‐70.
[28] National Institute of Standards and Technology. (2010) Guide for
Assessing the Security Controls in Federal Information Systems and Organizations. Special publication 800-‐53A.
[29] National Institute of Standards and Technology. (2010) Guide for
Applying the Risk Management Framework to Federal Information Systems.
Special publication 800-‐37.
[30] National Institute of Standards and Technology. (2011)
Information Security Continuous Monitoring (ISCM) for Federal Information Systems and Organizations. Special publication 800-‐137.
[31] British Standards Institution. (2013) Cyber security risk –
Governance and management – Specification. PAS 555:2013.
[32] Chaplin, M.; Creasey, J. (2011) The 2011 Standard of Good Practices
[33] Department for Business Innovation & Skills (2012) 10 steps to
cyber security guidance sheets. Retrieved from
http://www.bis.gov.uk/assets/biscore/business-‐sectors/docs/0-‐9/12-‐ 1121-‐10-‐steps-‐to-‐cyber-‐security-‐advice-‐sheets
[34] Wright, S. (2006) Measuring the effectiveness of security using ISO
27001, Whitepaper. SANS institute. Retrieved from
http://www.iwar.org.uk/comsec/resources/iso-‐27001/measuring-‐ effectiveness.pdf
[35] Information Systems Audit and Control Association (2012) COBIT
5 framework.
[36] Information Systems Audit and Control Association (2012) COBIT
5 for information security
[37] Bandopadhyay, S.; Sengupta, A.; Mazumdar, C. (2011) A
quantitative methodology for information security control gap analysis.
Proceedings of the 2011 International Conference on Communication, Computing & Security. Pages 537-‐540
[38] Järveläinen, J. (2013) IT incidents and business impacts: validating
a framework for continuity management in information systems.
International Journal of Information Management volume 33, issue 3. Pages 583-‐590.
[39] Breier, J.; Hudec, L. (2012). Towards a security evaluation model
based on security metrics. International conference on computer systems
and technologies ‘ 12.
[40] IT Governance Institute (2008) Aligning COBIT 4.1, ITIL V3 and
ISO/IEC 27002 for business benefit. Retrieved from
http://www.isaca.org/Knowledge-‐
Center/Research/Documents/Aligning-‐COBIT-‐ITIL-‐V3-‐ISO27002-‐for-‐ Business-‐Benefit_res_Eng_1108.pdf
[41] Karabacak, B.; Sogukpinar, I. (2006) A quantitative method for ISO
17799 gap analysis. Computer & Security. Volume 25, Issue 6. Pages 413-‐
419
[42] Software Improvement Group (2013). How secure is your
software?. Whitepaper -‐
www.sig.eu/blobs/Whitepapers/SIG_whitepaper_secure_software.pdf [43] Praxiom ISO IEC 27001 2005 Information Security Gap Analysis
Tool. Retrieved from http://www.praxiom.com/iso-‐27001-‐gap.htm
[44] Xu, H. (2014). Input research meeting at SIG
[45] International Organization of Standardization (2011). Systems and
software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models. ISO
25010:2011
[46] Heitlager, I.; Kuipers, T.; Visser, J. (2007) A practical model for
measuring maintainability. 6th International Conference on the Quality of
Information and Communications Technology (QUATIC 2007). Pages. 30– 39.