Consensus ranking – An ICT security awareness
case study
H.A. Kruger
a,*, W.D. Kearney
b,1aSchool of Computer, Statistical and Mathematical Sciences, North-West University, Potchefstroom Campus, Hoffman Street, Private Bag X6001, Potchefstroom 2520, South Africa
b
40 Shalimar Rise, Currambine, Perth, WA 6028, Australia
a r t i c l e
i n f o
Article history:
Received 22 October 2007 Received in revised form 26 May 2008
Accepted 9 July 2008 Keywords:
Information security awareness Consensus ranking
Assignment problem
Maximize agreement heuristic Decision making
a b s t r a c t
There are many disciplines where the problem of consensus ranking plays a vital role. Decision-makers are frequently asked to express their preferences for a group of objects, e.g. new projects, new products, candidates in an election, etc. The basic problem then becomes one of combining the individual rankings into a group choice or consensus ranking. The objective of this paper is to report on the application of two management science methodologies to the problem of identifying the most important areas to be included in an Information Communications Technology (ICT) security awareness program. The first methodology is based on the concept of minimizing the distance (disagreement) between individual rankings, while the second one employs a heuristic approach. A real-world case study from the mining industry is presented to illustrate the methods.
ª2008 Elsevier Ltd. All rights reserved.
1.
Introduction
Information security has become crucial to the continuous wellbeing of modern organisations and an information secu-rity solution should be a fundamental component in any organisation (Thomson et al., 2006). Information is regarded as an asset (Pipkin, 2000) and as such is exposed to a wide variety of threats and vulnerabilities that require a combina-tion of technical and procedural controls to mitigate risks. Companies often spend huge amounts of money and time on implementing technical solutions, while the human factor in information security receives less attention. Technical solu-tions are of course necessary to address vulnerabilities to viruses, denial of service attacks, etc.
However, the involvement of humans in information security is equally important and many examples exist where human activity can be linked to security issues. One such
example can be found in the area of social engineering, where ‘‘phishing’’ (fraudulent acquisition of sensitive information) has become one of the major problems associated with humans and their levels of awareness.Kerstein (2005) repor-ted that, according to Gartner, between May 2004 and May 2005 approximately 1.2 million computer users in the United States suffered losses caused by phishing. These losses were valued at $929 million. Companies in the United States also lose an estimated $2 billion annually as their clients fall victim to these scams. Statistics from the Association for Payment Clearing Services (APACS) revealed that losses from web banking fraud in the United Kingdom, which were mainly the result of phishing scams, rose by 90% from £12.2 million in 2004 to £23.2 million in 2005 (Finextra, 2006).
A key defence in the fight against security incidents that involve human activity, such as the phishing scams referred to above, is the use of ICT security awareness programs. In *Corresponding author. Tel.:þ27 018 2992539; fax:þ27 018 2992570.
E-mail addresses:[email protected](H.A. Kruger),[email protected](W.D. Kearney).
1Tel.:þ61 08 93054372.
a v a i l a b l e a t w w w . s c i e n c e d i r e c t . c o m
j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / c o s e
0167-4048/$ – see front matterª2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.cose.2008.07.001
general, the goal of such an awareness program would be to increase awareness of the importance of information systems security and the possible negative effects of a security breach or failure (Hansche, 2001). The importance of security aware-ness programs is also emphasized in the South African National Standard on Information Security, where one of the objectives of human resources security is given as ‘‘to ensure that all employees, contractors and third party users are aware of information security threats and concerns, their responsi-bilities and liaresponsi-bilities, and are equipped to support organiza-tional security policy in the course of their normal work, and to reduce the risk of human error’’ (SANS 27001, 2006).
The development and implementation of ICT security awareness programs imply that appropriate awareness material, activities and actions be developed, implemented and monitored. A wide variety of such material and possible actions is usually available to choose from, and a final deci-sion on what to use and where to focus attention is normally based on the different views obtained from different managers. To obtain the necessary resources in order to develop and implement an awareness program, it is necessary to identify the most important areas on which to concentrate effort and money. This is not always obvious – even if the areas have been identified, there may still be a problem in terms of which areas, if any, are more important than others. Questions such as whether all areas should receive equal resources or whether some areas should be regarded as more important and therefore receive more resources may be problematic. The same is true for awareness material – which material is more important and should be used more exten-sively? The problem then becomes one of combining the different management opinions into a group or consensus choice.
To assist in determining a consensus priority ranking of security awareness areas and/or security awareness promotion material, this paper investigates the application of two existing management science methodologies to obtain a consensus ranking from different role players. The first of the two methods that are briefly explained in the next section is based on work carried out byCook (2006)and makes use of the concept of minimizing the disagreement, or distance, between individual rankings through solving a linear assignment problem. Another overview of the technique can also be found inCook and Seiford (1978). The second method that was applied to the problem makes use of a heuristic called the maximize agreement heuristic (MAH) developed byBeck and Lin (1983).
The remainder of the paper is structured as follows: In Section 2 the two models to optimise disagreements are briefly introduced, while Section3implements the models in a real-world case study. Concluding comments are presented in the last section.
2.
Consensus ranking
The basic problem in a consensus priority ranking scheme is one of combining individual rankings into a group choice or consensus ranking given a set of individual rankings on a finite set of alternatives. Ranking problems can be classified into two basic categories, viz. cardinal problems and ordinal
problems. A cardinal ranking formulation requires an indi-vidual to express a degree of preference in the ranking, while this is not necessary in ordinal formulations. Ordinal prob-lems are called complete ordinal rankings when there are no ties in the ranking and when the transitivity property is present. The problem of combining individual ordinal rank-ings into consensus has been studied for many years and a number of procedures have been developed to deal with the problem. The simplest form of group consensus is majority rule.Kendall (1962) has proposed an approach where indi-viduals’ preferences, represented as priority factors, are simply added together and then the average is taken as the consensus choice.
Two different techniques in this study were applied to the problem of identifying the most important areas to be considered in an ICT security awareness program and the choice of awareness promotion material to be used. A brief introduction to each of the two techniques follows.
2.1. Distance-based approach
Cook and Seiford developed a theory of distance between Kendall’s priority factors and proposed a median consensus ranking based on distance. A good description of the axioms, mathematical representation and proof of existence of a unique distance function can be found inCook and Seiford (1978). An excellent overview of distance-based and ad hoc consensus models in ordinal preference ranking is given in Cook (2006).Shi et al. (1996)have already used this solution technique in a practical situation to determine consensus priority for information systems requirements. The nature of the security awareness areas prioritization problem described in this paper fits the framework of ordinal ranking problems and is suitable for the method developed by Cook and Seiford.
In general, the formulation can be described as follows. Considernindividuals andmobjects (security awareness areas). Letrijbe the rank of theith individual on thejth object
(i¼1,.,nandj¼1,.,m). Ifcjis the consensus rank for thejth
object, then theith individual’s absolute distance (disagree-ment) from the consensus ranking is represented by
di¼ Xm j¼1 rijcj ði¼1;.;nÞ:
The total distance of all individuals can then be expressed by
Xn i¼1 di¼ Xn i¼1 Xm j¼1 rijcj :
Ifcjis set equal to an index numberk(k¼1,.,m) the total
distance can be rewritten as
Xm j¼1 djk; wheredjk¼ Xn i¼1 rijk :
This represents the sum of distances between a consensus rankkand allnindividuals’ rank on thejth object. The best consensus ranking then becomes the one for which the total distance is a minimum.
The problem can now be represented by the following assignment problem.
MinPmj¼1 Pm k¼1djkxjk Subject toPmj¼1xjk¼1 ðk¼1;.;mÞ; Pm k¼1xjk¼1 ðj¼1;.;mÞ; xjk0; with xjk¼ 1 ifc j¼k; 0 otherwise:
The assignment problem is capable of handling large problems and can readily be solved by most linear programming software. Solution procedures and the struc-ture of an assignment problem are discussed extensively in the literature and details can be found inTaylor (2002), for example.
2.2. Heuristic approach
A simple procedure, called the maximize agreement heuristic (MAH), which can be used to arrive at a consensus ranking and that maximizes agreement among decision-makers, was developed by Beck and Lin (1983). Examples of how this heuristic was implemented in other studies can be found in Tavana et al. (1996), Tavana (2003)andKengpol and Tuominen (2006). The heuristic was also used in this paper for compar-ative reasons and the purpose of this section is to introduce briefly the mechanics of the MAH.
The MAH requires the construction of an agreement matrix A, where each elementaijrepresents the number of
decision-makers who ranked objectihigher than objectj. Positive and negative preference vectors, P and N, are then calculated using Pi¼ Xn j¼1 aij ði¼1;.;nÞ; and Ni¼ Xn j¼1 aji ði¼1;.;nÞ:
EachPiis a row total that represents the total agreement for
each objecti, i.e. the total number of times objectiis preferred over all other objects. Similarly,Niis a column total
repre-senting the total disagreement for each objectj, i.e. the total number of times objectjis not preferred when compared with all other objects.
If any entry in thePvector orNvector is zero, that object would be placed at the bottom or the top of the final consensus ranking, respectively. If no zero entries exist, the difference PiNi, for alli, is considered. The largest difference is
evalu-ated and if it is positive, the object is placed at the top of the final consensus ranking. If it is negative, the object will be placed at the bottom. It is often easier to complete the consensus ranking from the most to the least important ranking, in which case the largest positive difference (instead of the absolute difference) is used to indicate the next ranking. The placed object is now deleted from the agreement matrix and a new agreement matrix is constructed. The process is then repeated until all objects have been placed in the final ranking. Ties are dealt with arbitrarily.
In the following section, the two approaches are applied in a real-life case study to determine the best consensus ranking of selected information security awareness areas and aware-ness material used in an awareaware-ness program.
3.
Case study
3.1. Background
One of the largest international gold mining companies agreed to assist with the project. The company is a global African gold producer with 25 operations in 11 countries and is listed on a number of stock exchanges such as the Johan-nesburg Securities Exchange, New York Stock Exchange, etc. Over 6 million ounces of gold are produced annually, and it has one of the world’s largest reserves, resource bases and focused exploration activities around the globe. Operations include both deep and open pit mines and more than 62 500 people are employed in countries such as South Africa, Namibia, Ghana, Mali, Argentina, Brazil, USA and Australia.
Like any other organisation with ICT assets, senior management realized that a key defence against ICT security breaches would be to raise the general level of information security awareness and to educate all computer users in the basics of information security. The objective was to prevent, or at least reduce, human-related security incidents, for example, phishing. As a result, a comprehensive process was started to develop an ICT awareness program. During the last quarter of 2003 the roll-out of the programme commenced.
One of the priorities was to narrow the focus of the program into a manageable size and at the same time ensure that all important areas are covered. After careful deliberation and following a risk elimination process, the program was focused on six areas, viz.
always adhere to company policies,
keep passwords and personal identification numbers (PINs) secret,
use e-mail and the Internet with care,
be careful when using mobile equipment,
report incidences like viruses, theft and losses, and
be aware that all actions have consequences.
The program was rolled out to all computer users and awareness material was made available in English, Spanish, French and Portuguese. The six main awareness materials used included
video presentations,
personal presentations,
a website on the company’s intranet,
brochures,
posters in offices, and
articles in the company’s in-house magazine.
Following the implementation of the program a twofold business need arose. Firstly, there was a need to evaluate the success and effectiveness of the program, and secondly, a need to confirm that the six areas and six awareness
materials were the correct ones. The first concern was addressed through the development of a comprehensive tool to measure awareness levels of staff (Kruger and Kearney, 2006). The second issue was addressed through the use of consensus ranking techniques described in this paper and case study.
The motivation for reviewing the focus areas and aware-ness materials to determine whether new ones should be added or existing ones excluded from the awareness program can be found in ordinary business principles that impact ICT awareness programs. Business goals, technology and work environments are subject to constant change – to ensure that an ICT awareness program is properly aligned with changes and company objectives, periodic reviews of areas to be covered and material to be used should be conducted. Resources, such as money and effort, are necessary for any new or follow-up awareness campaigns. To ensure that they are effectively employed, it is important to know where to concentrate these resources. Once focus areas and material have been identified it is also necessary to determine the more important areas and material within the group of identified objects. It is very seldom that all identified aspects are of equal importance, and money and effort should not necessarily be evenly spread among identified focus areas and/or awareness material.
Another issue concerning the priority rankings is the measuring of the effectiveness of the awareness program. For example, theft of mobile equipment should be a higher risk in South Africa than in Australia. Priority rankings would therefore enable the incorporation of importance weights in a measuring tool and ensure more accurate measurements of awareness levels. One way of addressing these issues is to present a list of possible focus areas and awareness mate-rials to the right role players to rank them. The rankings should then be converted into a consensus ranking where the top x number of ranked objects are chosen for the program. The consensus ranking can also serve as an importance ranking from which importance weights can be derived.
3.2. Methodology
A very simple questionnaire was designed to present the six focus areas and six awareness materials to selected senior managers in each region (country). Respondents were then asked to rank them in order of importance from 1 (most important) to 6 (least important). In addition, they were asked to add any new items if necessary, and to include these new items in their importance rankings. Questionnaires and communications were translated into Spanish, French and Portuguese where appropriate.
A small number of senior decision-makers in each region were selected to participate. A personal e-mail from the Manager IT Risk and Compliance was sent to each of them, explaining the exercise and requesting them to complete and return the questionnaire. Twenty-two useable rankings were received, which represents a 63% response rate. The reason for the small number of participants was that only those senior managers in each region who had a direct influence on the company strategy and business goals were targeted.
3.3. Results
As per agreement with the company, the actual ratings of decision-makers may not be revealed.
None of the respondents have added any new items. This was seen as confirmation that the six focus areas and the six awareness materials used in the program were currently appropriate and relevant. Processing of the data was therefore focused on arriving at a consensus ranking to assist with providing importance rankings (weights) and thereby assist-ing with management information regardassist-ing the concentra-tion of effort and money. Responses received were converted into two distance matrixes, one for the focus areas and one for the awareness material, according to the discussion in Section 2. The Solver function of Excel was then used to solve the final assignment problems. For purposes of comparison the maxi-mize agreement heuristic was also applied to the responses. Table 1presents the results for the six focus areas andTable 2 the results for the six awareness materials used in the awareness program.
It can be seen from the two tables that there were no significant differences between the distance-based solution and the MAH. In both cases the top three rankings contain the same focus areas and awareness material although ‘video presentations’ and ‘posters’ exchanged first and second positions inTable 2. The middle column in each table indi-cates importance weights for each ranked object. The ranking orders were used to assign these importance weights to the areas and materials. The weights would be useful when measuring awareness levels, or they can be used to influence the allocation of resources. A very simple way of deriving the importance weights was used. The ranking orders were normalized to be between 0 and 1 and were then assigned in reverse order to the focus areas and the awareness materials. E.g. the focus area ‘keep passwords secret’ (Table 1) has the highest weight of 0.286 (6/(1þ2þ3þ4þ5þ6)) and ‘actions carry consequences’ the lowest weight of 0.048. This may then imply that 29% of the awareness budget should be spent on the focus area ‘keep passwords secret’, while only about 5% should go to the ‘actions carry consequences’ area.
Even though the distance-based approach may have more than one optimal solution in certain cases, it is also clear that
Table 1 – Consensus rankings for focus areas
Rank Consensus ranking based on the assignment problem Importance weight Consensus ranking using MAH 1 Keep passwords
and PINs secret
0.286 Keep passwords
and PINs secret
2 Adhere to company
policies
0.238 Adhere to
company policies
3 Use e-mail and
the Internet with care
0.190 Use e-mail and
the Internet with care
4 Report incidences 0.143 Careful when using
mobile equipment
5 Careful when using
mobile equipment
0.095 Report incidences
6 All actions have
consequences
0.048 All actions have consequences
the two techniques do not necessarily give the same consensus rankings. The fact that more than one optimal answer may be possible when using the distance-based approach was not seen as a problem in this case study – it simply means that if for example there are two optimal answers (consensus ratings) that are both of equal importance an arbitrary choice between the two can be made to allocate, for example, resources.
The case study suggested that the distance-based approach be considered for obtaining consensus rankings. Not only does it provide an optimal answer, but the application is extremely easy – calculating the distance matrix is easy and straightfor-ward, while any standard linear programming software (e.g. the solver function of Excel) can be used to solve the standard assignment problem. The MAH is also a useful tool with easy steps that can be successfully applied. However, in this study the application of the MAH was found to be somewhat tedious due to the iterative nature of calculations with different matrixes – a problem that will be aggravated when dealing with many objects that need to be ranked. The development of an automated tool should be considered when using the MAH. An optimal answer is also not guaranteed by the MAH. One of the limitations of the distance-based approach, as used in this study, is the fact that all rankings should be complete (no ties), while the MAH is capable of handling incomplete rankings. Missing values in responses should also be discarded. However, in this case study, there were only a few objects (six in each case) to be ranked and none of the participating deci-sion-makers had any problems in complying with the request of providing complete rankings with no missing values. In other cases or circumstances this might not be the case.
The problem of consensus ranking discussed in this paper is not limited to only a few or a small number of objects to be ranked. Although psychological research indicates that the maximum number of objects for which most decision-makers can make meaningful judgements will vary from five to nine (Patton et al., 1983), a greater number of objects can easily be handled by grouping them into groups of, say, nine objects each (Shi et al., 1996).
3.4. Management response to the consensus ranking
model
It was difficult to verify senior management’s response to the consensus ranking model with each manager involved in the
exercise as they were located in different parts of the world where they formed part of the senior strategic management team of the mining group. One of the senior managers, the Manager IT Risk & Compliance, who deals regularly with the respondents and who assisted in performing the case study, was used as a representative opinion from senior management. According to the Manager IT Risk & Compliance, one of the major advantages observed during the case study was the use of the simple and easy questionnaire that was distributed to respondents. It consisted only of a list of the six objects to be ranked in order of preference/importance – no weights or any other information or evaluations were requested. This may sound insignificant but it is a known fact that people, especially senior decision-makers, do not normally like to complete questionnaires. In a previous project, which is related to this one, an attempt was made to measure the security awareness levels of staff based on the six focus areas (Kruger and Kearney, 2006). Importance ratings were determined by using the Analytic Hierarchy Process (AHP), where a decision-maker would give his/her preference by means of pairwise compari-sons. Many practical problems arose from this, e.g. getting managers to participate as they did not understand the AHP completely; combining the different managers’ pairwise comparisons into one overall comparison; inconsistent pair-wise comparisons; and a waste of time redoing the exercise every time comparisons were inconsistent. None of these problems existed during this case study, something that can mainly be attributed to the use of the simple questionnaire.
Other positive feedback received can be summarised as follows: a short and easy process to obtain consensus from different role players; a process that is easily understood by all, which encouraged participation; no ‘comebacks’ in the form of additional meetings or questionnaires to try and resolve deadlocks; and the provision of a formal and trans-parent framework to achieve consensus – previously, attempts to achieve consensus were mostly during discus-sions, something that was often susceptible to problems caused by group dynamics.
4.
Conclusion
ICT security awareness programs have become one of the key defenses in the fight against security incidents involving the human factor, and sufficient material exists to assist organi-sations with delivering proper awareness programs. These programs are normally focused on specific areas of concern and may include a variety of awareness materials such as posters, presentations, brochures, etc. To obtain and justify resources needed for such an awareness program and to comply with the business principle of efficient and effective use of resources, while at the same time addressing business objectives, a consensus choice of focus areas, awareness material and importance rankings are required.
To address this problem, this paper described a consensus ranking method based on the concept of minimizing the distance between individual rankings. The method was demonstrated in a mining environment with satisfactory results. First, the traditional questionnaire was replaced by a single form containing only the list of objects to be ranked.
Table 2 – Consensus rankings for awareness material
Rank Consensus ranking based on the assignment problem Importance weight Consensus ranking using MAH
1 Video presentations 0.286 Posters in offices
2 Posters in offices 0.238 Video presentations
3 Personal presentations 0.190 Personal
presentations 4 Brochures 0.143 Brochures 5 Website on company’s intranet 0.095 Website on company’s intranet 6 Articles in in-house magazine 0.048 Articles in in-house magazine
Senior decision-makers were then asked to rank the objects in order of preference/importance, and, finally, the consensus ranking was calculated by constructing a distance matrix and solving a standard assignment problem. For purposes of comparison, the rankings were also heuristically evaluated. The distance-based method was extremely easy to apply and promises to provide management with information that will assist in identifying important security awareness areas, the allocation of resources to these areas and the provision of more accurate measuring opportunities of security awareness levels.
In general the study has shown that the use of a formal consensus ranking technique not only saves time and money, but may also provide a better understanding of the relevance and importance of those factors influencing an ICT security awareness program. Applying the techniques described in the paper assists in identifying and prioritizing improvement opportunities in an easy and transparent way and will enable decision-makers more accurately to address security aware-ness problems, such as phishing scams, through focused awareness programs. The techniques discussed are not limited only to situations described in the case study but could be used in any decision-making situation associated with ICT security.
Acknowledgement
The authors would like to thank the two anonymous referees for their constructive comments that helped improve the paper. The authors alone are responsible for any errors and omissions.
Part of this research was supported by the National Research Foundation in South Africa. Grant reference FA2007030800004.
r e f e r e n c e s
Beck MP, Lin BW. Some heuristics for the consensus ranking problem. Computers and Operations Research 1983;10(1):1–7. Cook WD. Distance-based and ad hoc consensus models in
ordinal preference ranking. European Journal of Operation Research 2006;172:369–85.
Cook WD, Seiford LM. Priority ranking and consensus formation. Management Science 1978;24(16):1721–32.
Finextra. UK phishing fraud losses double. Available from:
<http://www.finextra.com/fullstory.asp?-id¼15013>; 2006 [accessed June 2006].
Hansche S. Designing a security awareness program: part 1. Information System Security January/February 2001:14–22. Kendall M. Rank correlation methods. 3rd ed.; 1962. New York. Kengpol A, Tuominen M. A framework for decision support
systems: an application in the evaluation of information technology for logistics firms. International Journal of Production Economics 2006;101(1):159–71.
Kerstein PL. How can we stop phishing and pharming scams?. Available from:<http://www.csoonline.com/talkback/071905. html>; 2005 [accessed June 2006].
Kruger HA, Kearney WD. A prototype for assessing information security awareness. Computers & Security 2006;25:289–96. Patton JM, Evans JH, Barry LL. A framework for evaluating internal
audit risk. Research report number 25. Altamonte Springs, FL: The Institute of Internal Auditors, Inc.; 1983.
Pipkin DL. Information security. Protecting the global enterprise. Upper Saddle River, NJ: Prentice Hall; 2000.
SANS 27001:2006. South African National Standard. Information technology – security techniques – information security management systems – requirements. SANS 27001:2006, the identical implementation of ISO/IEC 27001:2005. 1st ed. Pretoria: Standards South Africa (a Division of SABS); 2006. Shi Y, Specht P, Stolen J, Vanwetering F. A consensus ranking for
information system requirements. Information Management & Computer Security 1996;4(1):10–8.
Tavana M. CROSS: a multicriteria group-decision-making model for evaluating and prioritizing advanced-technology projects at NASA. Interfaces 2003;33(3):40–56.
Tavana M, Kennedy DT, Joglekar P. A group decision support framework for consensus ranking of technical manager candidates. Omega 1996;24(5):523–38.
Taylor BW. Introduction to management science. 7th ed. Prentice Hall; 2002.
Thomson K, Von Solms R, Louw L. Cultivating an organisational information security culture. Computer Fraud & Security October 2006;2006(10):1–11.
H.A. Kruger is an Associate Professor in the School of Computer, Statistical and Mathematical Sciences at the North-West University (Potchefstroom Campus) in South Africa. He previously worked for a large international mining company and has a number of years experience in Informa-tion Risk Management. He has a PhD in Computer Science, a MCom (Information Systems) and an MSc (Mathematical Statistics). His current interests include decision modeling and the use of linear programming models.
W.D. Kearnycurrently works as a Manager, Risk and Assur-ance. He has over 20 years experience in Information Risk Management in a number of positions in large international companies. He has an MSc degree, numerous diplomas and earned a number of certifications, including CISA and CIA.
