In order to be universally accepted, implemented performance mea- surement criteria must be seen as quantifiable in the development and objective of the application. In order to achieve this, the organiza- tional culture must support accurate, comprehensive, and balanced feedback in order to facilitate effective decision making by providing the information necessary. The resources required to implement this should not be underestimated because it takes time and effort to be
introduced and maintained. A “one size fits all” approach cannot be used due to the extreme variations in architectures, methodologies, and approaches undertaken in developing and utilizing information systems.
Prior to implementation an impact analysis will be required in order to determine the ability of the organizational IS area to imple- ment such an approach as well as both financial and functional impact of such implementation. Based upon the results of this analy- sis a controlled pilot project would normally be launched within one of the operational areas of the IS function. The area chosen should be one in which performance measurement points can be readily deter- mined and the life cycle of improvement is comparatively short. The starting point, as ever, is the long-term business objective of the area to be improved. This may seem obvious but determining agreed objectives with a common interpretation can be problematic. Once the objectives have been agreed an appropriate measurement archi- tecture can be developed to assess the performance area from the four perspectives mentioned earlier. Once the architecture has been agreed a strategy of transition to the new measurement approach must be developed. The pilot project will be used to adapt and streamline the performance measurement architecture prior to implementation of the full IS area.
The use of a balanced scorecard helps to create a Strategy-Focused Organization (SFO), rather than a Metrics-Focused Organization (MFO), which thus facilitates the alignment of its daily activities to strategy, and communicates that strategy throughout the organiza- tion. This is critical because many organizations utilize key perfor- mance indicators, which are tactical in nature rather than strategic. Because the balanced scorecard is designed to ensure achievement of strategic objectives, a strategic focus is essential. Many of the benefits of information systems are intangible in nature making it difficult to measure their relative contribution to overall business performance. There is a tendency to conduct project management at a technical level while ignoring the original business proposition that created the project in the first place. Many IS benefits such as improved customer service and increased flexibility are difficult to interpret into financial benefits. It has been suggested7that as many as 75% of IS interven-
tions offer no demonstrable business value. The key word in this sit- uation is demonstrable because traditional financial evaluation methods may not tell the whole story in an IS environment.
In order to adequately express measurement criteria within the balanced scorecard, the starting point must always be the statement of business intent or mission for the entity being evaluated. In such a case the customer orientation intent may be “to deliver customer response times superior to competitors.” This could then be translated into measurement criteria by reducing the intent into measurable objectives such as customer satisfaction, acquisition of new customers from com- petitors, or retention of existing customers, leading to measurements such as numbers and severity of customer complaints, percentage of repeat customers, number of new customers acquired from competi- tors, and so forth. These are not the traditional financial measures but nevertheless express the strategic content of the information system. In a similar manner measurement criteria may be established for the other balanced scorecard perspectives. Internal business processes could, for example, be measured by percentage availability of systems, frequency of system crashes, response times, downtime, and so on. Innovation and learning could be measured by determining skills acquisitions or independence from consultants. Critically, the IS bal- anced scorecard measurement criteria should be able to indicate cause and effect of performance drivers rather than solely business out- comes. Kaplan and Norton recently released a follow-up to The Bal- anced Scorecard,8in which they noted:
The Balanced Scorecard approach retained measures of financial performance, the lagging indicators, but supplemented them with the measures on the drivers, the lead indicators, of future financial per- formance. But what were the appropriate measures of financial per- formance? If financial measures were causing organizations to do wrong things, what measures would prompt them to do the right things? The answer turned out to be obvious: Measure the strategy! Thus all of the objectives and measures on a Balanced Scorecard— financial and nonfinancial—should be derived from the organiza- tion’s vision and strategy.
Control Risks and Outsourcing
Outsourcing of IS services is a route chosen when: An organization determines that it does not have sufficient competence to tackle the work itself, where the costs of in-house processing are determined to be
greater than the cost of outsourcing the activity, or where the risks asso- ciated with the executor function is determined to be too high to retain in-house. IT areas frequently considered for outsourcing include:
■ Project functions such as systems analysis, design, and programming ■ Running of selected application systems such as payroll
■ Data capture or transformation prior to processing ■ Operation of the IT facility
In all cases measurement and performance criteria must be devel- oped prior to evaluation of the most appropriate process. Consider- ations such as the scope of services, level of service requirements, quality measurements, and cost must be taken into account in advance because choosing to outsource such services can be a higher risk intervention and the decision may be difficult to reverse.
Whether or not the outsourcing partner retains the organization’s original staff as part of the outsourcing agreement, there may be insufficient skills and abilities retained to rebuild an internal IT ser- vice if required.
The selection of an outsourcer is a critical part of the process requiring that checks be made on both the quality of service received by the existing customer base as well as their market stability and track record.
A common mistake is approaching a selected number of out- sourcers to inquire as to the variety and level of services they can offer. As with any project, business and technical requirement specifications should be designed with evaluation criteria before outsourcing agen- cies are approached.
Outsourcing contracts for information services will typically involve a longer duration than normal commercial contracts due to the longer duration of systems development and a higher cost in setup of specific architectures. As such, it is essential to ensure their con- tracts clearly spell out the variety of services to be provided, mea- surement criteria, and penalties for unacceptable performance. This would normally take the form of a service level agreement (SLA) which, in an outsourcing contract, would typically involve more detail than those in normal internal use. Performance areas such as security, right of inspection by audit, and contingency planning arrangements must be incorporated within the SLA.
As with any partnering agreement the intention and the belief is that all will go well. Nevertheless, a fundamental reason for the con- tract is to prepare for the eventuality that things do not go well and therefore options and remedies in the event of a partial or complete failure of the agreement need to be decided in advance. This is par- ticularly true where IT services are involved because the technical nature of such services can result in lengthy court cases regarding per- formance measurement and contract terms. In cases of disputes many users of such services attempt resolution by arbitration in order to avoid costly and drawn-out court battles.
Where software is involved, escrow may be required for the pro- tection of software interests. This involves the lodging of a copy of the software with an independent third party so that, in the event of the software provider failing to comply with the terms of contract or ceasing to exist, software service can be continued legally by obtain- ing the software from the third party.
Auditing IS Management
In reviewing a typical IS function, the auditor will use a standard set of indicators to determine whether management is performing at an acceptable level. These indicators could include:
■ Operational
● Degree of end-user acceptance of performance levels ● Actual to budget comparisons
● Frequency of hardware and software problems ● Acceptability of computer response time
● Frequency of unauthorized purchases of hardware and soft-
ware
● Frequency of upgrades of hardware and software ● Degree of capacity planning
● Adequacy of business continuity planning
■ Systems development
● Cost and budget comparisons ● Achievement of deadlines
● Achievement of end-user objectives
■ General management
● Experience and training of staff ● Staff turnover
● Staff motivation
● Reliance and key individuals
Such indicators would be examined using the conventional audit techniques of documentation review, observation, and interviewing in order to determine management’s intent as compared to actual per- formance.
ENDNOTES
1. Project Management Institute (PMI) Standards Committee, A Guide to the Project Management Body of Knowledge (PMBOK® Guide), Third Edition, Sylva, North Carolina: PMI
Publishing, 2000.
2. R. D. Archibald, Managing High-Technology Programs and Pro- jects, New York: Wiley, p. 19.
3. B. Boehm, “A Spiral Model of Software Development and Enhancement.” IEEE Computer (May 1988): 61–72.
4. E. Fish, An Improved Project Lifecycle Model, Pandora Consult- ing, http://www.maxwideman.com/guests/plc/intro.htm (Guest Department), 2002, updated 2003.
5. International Organization of Standardization, ISO 8402: 1986. 6. Robert S. Kaplan and David P. Norton. The Balanced Scorecard, Boston: Harvard Business School Press, 1996. “Using the Bal- anced Scorecard as a Strategic Management System,” Harvard Business Review (January-February 1996): 75–85.
7. M. Raisinghani, “A Balanced Analytic Approach to Strategic Electronic Commerce Decisions: A Framework of the Evaluation Method,” in W. Van Grembergen, Information Technology Eval- uation Methods and Management, Hershey, PA: Idea Group Pub- lishing, 2001, pp. 185–197.
8. Robert S. Kaplan and David P. Norton. The Strategy-Focused Organization: How Balanced Scorecard Companies Thrive in the New Business Environment, Cambridge, MA: Harvard Business School Publishing, 2000.