18
AN APPROACH TO DESIGN SERVICES KEY PERFORMANCE
INDICATOR USING COBIT5 AND ITIL V3
1
Retno Ayu Widiyaningrum,
2Kudang B Sminar,
3Husniteja Sukmana
Department of Computer Science, Bogor Agricultural University, IndonesiaEmail: [email protected]
Abstract
Information Technology Infrastructure Library v3 is a best practice framework in Information Technology Service Management which enables organization to manage their service. It also has a comprehensive workflow which covers daily operation of Information Technology. COBIT5 is also a best practice framework which has been validated all over the world in the field of information technology governance. COBIT 5 also provides matrices that can be used to measure performance of the organization. In order to enabling the performance, the department needs key performance indicators (KPI's). In this paper, KPI's are derived from the process named goals cascade using COBIT 5 and validated using Lawsen's content validity method within 6 expert panelists and CVR value = 0.99. The result shows 6 relevant KPI's that can be implemented in development of service desk application based on ITIL v3. These 6 KPI's are belong to COBIT 5 APO09 and DSS02 process models. The result shows that achievement of KPI's through implementation of COBIT 5 and ITIL v3 helps organization achieve 11 stakeholder needs for enterprise goal : Stakeholder value of business investments, and 17 stakeholder needs for enterprise goal : Portfolio of competitive products and services. From content validation process, the result also shows the validity of key performance indicators derived from goals cascade.
Keywords
:
Key Performance Indicator, COBIT5, goals cascade, Performance
Measurement, ITIL v3, Service Desk
INTRODUCTION
IT department in an organization typically is a department that provide and manage information technology services related to the organization. This research takes place in a state university in Indonesia. IT department in the university has responsibility in development and maintenance academic information
system and other university-related systems,
websites, e-mail, computer networks, data support, etc. The department does not have performance indicators yet, so they find it is difficult to measure their service performance.
Performance measurement needs the
existence of Key Performance Indicators (KPI's). The performance measurement also should be in line with the objectives of the organization. COBIT 5 and ITIL
v3 are the best practice frameworks that used and validated all over the world in IT governance and IT
service management. Usage of standardized
frameworks gives advantages to the organization in case of collaboration and coordination [1]. COBIT 5 is a framework that used to control management strategy of information technology. COBIT 5 is a complete governance and management framework. Yet there is little academic research that leverage COBIT 5 as a research instrument[8]. COBIT 5 are developed by Information Technology Governance Institute in association with Information System Audit and Control Association[2]. COBIT 5 has a
process named goals cascade which helps
19
the matrices. COBIT 5 contains 37 process models which divided into 5 control objective categories :1. Evaluate, Direct and Monitor (EDM) 2. Align, plan and organize (APO) 3. Build, acquire and implement (BAI) 4. Deliver, service and support (DSS) 5. Monitor, evaluate and assess (MEA)
IT service management is defined as an operation set of information technology, which is characterized by an emphasis in the service of information technology, consumer, service level agreement, and daily activities of IT services[17]. And ITIL is defined as a best practice from IT functions that can be done to achieve IT service management. Even though it has been proven that implementation of ITIL can increase the quality of IT services, Arias and Monroy showed that ITIL has no clear measure yet. The manager has to build a complete ITIL metrics program. He must figure out which metrics are the most important and how to determine them (Arias. CA, 2007). In the need of collaboration and coordination, IT department should
follow best practices framework. Usage of
standardized frameworks also helps organization grows and survives in the middle of today’s competition. It also gives a complete and well structured services documentation to the organization that can be a meaningful data to improve the service quality.
In 2007, Thomas Schaff compared ITIL and NGOSS on business driven framework and stated that both have service level management component, but ITIL describes it more detail (Schaff, 2007). In 2007, Erna Infitharina analyzed the implementation of service desk as a part of ITIL and stated that it has improved IT service quality, faster problem solving, and well structured documentation (Erna, 2007). In 2008, Sahibudin. S, Sharifi. M, Ayat. M combined ITIL, COBIT and ISO 27002 to create a comprehensive framework design for IT service management. The paper said that organization can use ITIL to define strategy, planning and processes, and use COBIT to measurement matrices, benchmark and audit (Sahibudin S, 2008). In 2009, Bernroider and Ivanov attempted to add an empirically validated structure of internal control over IT project management using COBIT 5 and stated that the metrics suggested by COBIT are regarded as feasible and important by project management professional, and are regularly used in controlling practice (Bernroider, 2011). In 2013, Thomas PH Herz,
Florian Hamel, Falk Uebernickel, and Walter Brenner conduct research to refine how a corporation monitors and regulates the development of IT applications and monitor suppliers through the utilization of Key Performance Indicator (KPI) in a multi-supplier environment (Herz, 2013). In [9],
Hesham and Saad show that performance
measurement achieved highest score (4.57) within organization indicator, and help desk achieved highest score (4.71) within technology indicator. Spremic and Zmirak conducted a case study of ITIL implementation in finance service industry in [14], but in the research, KPI's are established by the organization and it means there is no guarantee that the KPI are formally aligned with organization goals. Wan, S.H.C. and Yuk-Hee Chan also implemented and evaluated IT service management process in campus environment with outsourcing IT service support. The result shows that implementation of ITIL has improved service target at average 13.4
percent, gave better communications and
transparency, better accountability which shown by traceable details for event and its respective handling
operator, and efficient event resolution
(Wan.HC.Stewart, 2007).
De Haes, in [8] stated that one of potential research question in COBIT 5 is to use it as a common measurement foundation for investigation of some particular aspect of EGIT or cognate areas of inquiry such as IT audit and assurance. The difference of this research is, unlike [15], which the KPI's are established by organization, in this research, COBIT 5 will be used to specify Key
Performance Indicators to measure service
performance of IT department in the university. Moreover, this research also brings out performance KPI's that can be embedded into ITIL v3 service desk as [9] stated that performance and service desk are important in service quality improvement and [4],[2] proved improvement service quality by implementing ITIL V3. In 2012, Jantti in [11] stated that the 9th lesson to improving IT service desk and service management is using CSF, KPI and metrics as a basis of a process measurement program. Therefore, as [5]
recommends to using COBIT for metrics
establishment, performance indicators are gained through COBIT 5 goals cascade process which derivate performance indicators from stakeholder needs into the form of measurement metrics. The selection is done to filter indicators that related to ITIL v3 service desk workflow. In the end, content
20
validation is conducted to make sure the indicators met organization needs.RESEARCH METHOD
The research methodology is divided into two sections : (1) Using COBIT 5 goals cascade to gain performance indicators, and (2) Validation process of performance indicator to become key performance indicator using Lawshe's content validation method.
COBIT 5 provides 17 enterprise goals that is commonly used by organizations all over the world. Each of these goals are related to one or more IT-Related goals (COBIT 5 also provide it). To derive performance indicator through goals cascade process, organization needs to gather their needs, in goals cascade it is called stakeholder needs. Stakeholder needs is defined as everything the organization expected from IT department. Figure 1 shows the
flow of goals cascade process.
Figure 1 Goals Cascade Process
According to figure 1, organization needs to map the stakeholder needs to enterprise goals. The relation between each stakeholder need and enterprise goals can be primary (P) or secondary(S). Primary relation is assigned when an enterprise goals related directly with the stakeholder needs, and secondary means indirect relations. In the end of this process, organization gained their performance indicators. In accordance with the scope of model development in this study which is limited to service performance measurement, COBIT 5 process models and measurement matrices used in the remaining part of this research are the metrics related to service agreements, services requests and service incidents.
Superimposing COBIT's conceptual model onto audit relevant assessments made by a panel of highly experienced IT auditors confirms the internal consistency between the underlying constructs of COBIT (Tuttle. Brad, 2007). Lawshe's content validation method is developed in 1975 and published in [13] to solve the problems by consensus of the expertise. The experts were asked to provide value for each solution candidate which can be in form of : (1) Essential, (2) Useful but not essential, or (3) Not necessary. From the result of consensus, a variable CVR is calculated to select the suitable candidates.
Where n is the number of panelist who determine a candidate as (1) essential or (2) useful but not essential and N is total number of panelists. The selection process is conducted by comparing CVR value of each candidate with minimum CVR (Bertea. Elena, 2013). Table 1 Minimum CVR, p=0.5 No of Panelist Min CVR 5 .99 6 .99 7 .99 8 .75 9 .78 10 .62 11 .59 12 .56 13 .54 14 .51 15 .49
Six panelists were involved in this research include the head of divisions from IT department in the university and the head of IT department itself. From table 1, with 6 panelists we get .99 as minimal CVR. Candidates with minimum CVR .99 will be accepted as key performance indicator.
RESULTS AND ANALYSIS
Goals Cascade
From observations, strategic documents are collected and selected as stakeholder needs. The
documents include vision and mission of
organization, and quality documents of the university and IT department. Then stakeholder needs are
21
mapped into enterprise goals using Primary or Secondary relation attribute.Figure 2 Stakeholder needs
Figure 2 shows a part of mapping table between stakeholder needs and enterprise goals. Summary of mapping results shown in table 2. From the result, enterprise goals which significantly primarily relevant to stakeholder needs are selected to the remaining of the process (primary count = 11 or more).
Table 2 Enterprise Goals Result BSC
Dimension Enterprise Goals Count
Financial Stakeholder value of business
investments 11
Financial Portfolio of competitive
products and services 17
Financial Managed business risk
(safeguarding of assets) 6
Financial Compliance with external laws
and regulations 2
Financial Financial transparency 4
Customer Customer oriented service
culture 13
Customer Business service continuity and
availability 9
Customer Agile responses to a changing
business environment 1
Customer Information based strategic
decision making 4
Customer Optimization of service delivery
costs 2
Internal Optimization of business
process functionality 13
Internal Optimization of business
process costs 3
Internal Managed business change
programs 0
Internal Operational and staff
productivity 8
Internal Compliance with internal
policies 6
Learning
and growth Skilled and motivated people 5
Learning and growth
Product and business innovation
culture 12
Each enterprise goal is related to one or more IT-Related goals with primary or secondary relation. Figure 3 shows the mapping table between enterprise goals and IT-Related goals.
Figure 3 Stakeholder Needs Mapping
In this research, only IT-Related goals which have primary relation with enterprise goals are selected. From the mapping process, resulted 10 IT-Related goals that will be used in the remaining processes as shown in table 3.
Table 3 IT Related Goals IT Related Goals
22
Alignment of IT and business strategyCommitment of executive management for making IT-related decisions
Realized benefits from IT-enabled investments and services portfolio
Delivery of IT services in line with business requirements
Adequate use of applications, information and technology solutions
Optimization of IT assets, resources and capabilities Enablement and support of business processes by integrating applications and technology into business processes
Delivery of programs delivering benefits, on time, on budget, and meeting requirements and quality standards
IT agility
Knowledge, expertise and initiatives for business innovation
Each IT-Related goals are belong to one or more COBIT 5 process models and each process model has its own matrices as explained in COBIT 5 process reference publication[7]. Process models for IT department then selected based on the mapping table. From COBIT 5 process reference [7], some of COBIT 5 processes are related to ITIL V3. To make a smooth integration between COBIT 5 matrices and the flow of ITIL V3 service desk, process models are selected again based on its relation with ITIL V3. Considering the focus of this research, there are two COBIT 5 process models that related to ITIL V3 service desk. There are APO09 (Manage Service Agreements) which related to ITIL V3 demand management, service portfolio management, service catalogue management, service level management and service reporting, and DSS02 (Manage Service Request and Incidents) which related to ITIL V3 incident management.
Performance indicator metrics from APO09 process models are :
1. Number of business processes with
undefined service agreements
2. Percent of customer satisfied that service delivery meets agreed-on levels
3. Percent of service being monitored to the service levels
4. Percent of service targets being met
And performance indicator metrics from DSS02 process models are :
1. Percent of incidents resolved within an agreed-on / acceptable period of time 2. Level of user satisfaction of service request
fulfillment
3. Mean elapsed time for handling each type of service request
Because of the similarity, the metrics then simplified into 4 indicators :
1. Number of services meet the SLA
2. Percent of customer satisfied with the services delivery
3. Percent of incidents resolved within an agreed-on / acceptable period of time and meet the targets
4. Mean elapsed time for handling each type of service request
Content Validation
Content validation process is conducted in two phases of processes, that are collecting the idea and validation through consensus. Content validation process conducted in the participation of 6 expertise in the fields. They are a head of IT department, and 5 head of divisions in IT department of the university.
Idea collecting phase resulted 8 additional
performance indicators. These 8 additional indicators joined with 4 previous goals cascade indicators then CVR value of these indicators are count as shown in table 4.
Table 4 CVR Values
Indicators CVR
Number of services meet the SLA .99
Percent of customer satisfied with the services delivery
.99 Percent of incidents resolved within an agreed-on / acceptable period of time and meet the targets
.99
Mean elapsed time for handling each type of service request
.99
Number of repeated problems in a period .99
Resolving time below the SLA .99
Number of out of catalogue service requests .33
Percent of SLA Increasing .99
Number of incidents that need no escalation .33
Number of services saved in knowledge base
.99 Percent of resolved requests in each levels which meet SLA
.99
23
Content validation process result shows that 10 indicators which have CVR value more than minimal value (.99) are determined as key performance indicators (KPI) of services provided by IT department in the university. As we can see the 3rd and 4th KPI and 6th KPI contents has been covered in KPI1. In order to avoid the overlap, 3rd, 4th and 6th KPI were included into KPI1. Furthermore, in ITIL v3, service category's SLA need to be declared before operational phase, so KPI10 is also eliminated.CONCLUSION
This research shows an approach to gain Key Performance Indicators for staff of IT department in the university. The input of goals cascade processes are vision and mission of the
organisation, strategy document and quality
document. The result shows that not all COBIT 5 process models are related to ITIL v3 service desk workflow.
The result of content validation method shows that all of the Key Performance Indicators gained through COBIT 5 goals cascade are relevant with organization needs. The result shows that CVR value of these 4 indicators are .99, which means the expertises agreed that these KPI's are essential for performance measurement of the organization. This result has conform Bernroider previous research[2]. From content validation phase, there are 6 additional KPI's which also validated through CVR value. But some of those KPI's are overlapped.
As the final result, there are only 6 KPI which can be used to the measure the performance of IT staff related to ITIL v3. These KPI's are :
1. Number of services meet the SLA
2. Percent of customer satisfied with the services delivery
3. Number of repeated problems in a period 4. Percent of SLA increasing
5. Percent of resolved requests in each levels which meet SLA
6. Number of services saved in knowledge base
REFERENCES
[1] Arias. CA, M. C. (2007). ITIL Methodology to
Manage Information Systems Departments:Benefits and Risks. International Conference on IndustrialEngineering and Industrial Management - CIO.
Madrid.
[2] Bernroider, E. W. (2011). IT Project Management Control and The Control Objectives for IT and Related Technology (CobiT) Framework.
International Journal of Project Management, vol. 29
, 325-336.
[3] Bertea. Elena, A. Z. (2013). SCALE VALIDITY IN EXPLORATORY STAGES OF RESEARCH.
Management and Marketing Journal vol XI issue 1 ,
38-46.
[4] Bin-Abbas. Hesham, H. B. (2014). Assessment of IT governance in organizations: A simple integrated.
Computers in Human Behavior, vol 32 , 261-267.
[5] CH, Lawshe. (1975). A QUANTITATIVE APPROACH TO CONTENT VALIDITY. Personnel
Pshycology vol 28 , 563-575.
[6] Conger. S, W. M. (2008). Service Management in Operations. Proceedings of the Fourteenth Americas
Conference on Information Systems. Canada.
[7] De Haes. Steven, G. W. (2013). COBIT 5 and Enterprise Governance of Information Technology: Building Blocks and Research Opportunities.
JOURNAL OF INFORMATION SYSTEMS, vol 27 no 1 , 307-324.
[8] Erna, I. (2007). Penerapan Information
Technology Infrastructure Library Framework pada Sistem Manajemen Service Desk (Studi Kasus PT Tridas Widiantara). Bogor: Bogor Agricultural
University.
[9] Herz, T. P. (2013). Toward a Model of Effective Monitoring of IT Application Development and
24
Maintenance Suppliers in Multisourced Environment.International Journal of Accounting Information Systems, vol 14 , 235-253.
[10] ISACA. (2012). COBIT® 5: Enabling
Processes. ISACA and IT Governance Institute®
(ITGI®).
[11] Jantti, M. (2012). Improving IT Service Desk and Service Management Processes in Finnish Tax Administration: A Case Study on Service
Engineering. LNCS 7343 , 218-232.
[12] OGC. (2007). The ITIL V3 Service Strategy. OGC Publisher.
[13] Sahibudin S, S. M. (2008). Combining ITIL, COBIT and ISO/IEC 27002 in Order to Design a Comprehensive IT Framework in Organizations.
AICMS .
[14] Schaff, T. (2007). Frameworks for
Business-driven Service Level Management; A Criteria-based Comparison of ITIL and NGOSS.
Ludwig-Maximilians University (LMU).
[15] Spremic Mario, Z. Z. (2008). IT and Business Process Performance Management: Case Study of ITIL Implementation in Finance Service Industry.
Proceedings of the ITI 2008 30th Int. Conf. on Information Technology Interfaces. Croatia.
[16] Tuttle. Brad, V. D. (2007). An empirical examination of CobiT as an internal control
framework for information technology. International
Journal of Accounting Information Systems, vol 8 ,
240-263.
[17] Wan.HC.Stewart, C.-H. (2007). IT Service Management for Campus Environment-Practical Concerns in Implementation. 10th IFIP/IEEE
International Symposium on Integrated Network Management. IEEE.
