Performance Measures to evaluate
the impact of Best Practices
M.H. Jansen-Vullers, M.W.N.C. Loosschilder, P.A.M. Kleingeld, and H.A. Reijers
Department of Technology Management, Eindhoven University of Technology P.O. Box 513, NL-5600 MB, Eindhoven, The Netherlands.
{m.h.jansen-vullers, p.a.m.kleingeld, h.a.reijers}@tue.nl,
Abstract. Business process redesign best practices help to generate al-ternative process redesigns. Best practices are often used by practitioners to repeat design techniques that have proven effective in practice, but it is not clear why they are the best, and in what respect. More importantly, it is unclear why a best practice might be “best” for one process and not for another. In this paper, we propose a framework for performance measures that can be used to evaluate the effect of a redesign with the dimensions of time, cost, quality and flexibility. For each dimension we propose a set of measures, which have been operationalized as much as possible. We conclude that a simulation approach is very well applicable to estimate the impact of redesign best practices, especially for the time, cost and flexibility measures. For quality measures additional methods are necessary to complement the estimation.
Keywords: Business Process Redesign, Best Practices, Performance Measurement.
1
Introduction
The domain of business process redesign can roughly be divided in two differ-ent approaches: revolutionary and evolutionary. In the revolutionary approach, a redesign starts from a clean sheet. In the evolutionary approach, the exist-ing business process is taken as a startexist-ing point. Application of redesign best practices fits within this approach. Reijers provides an overview of all heuristics currently encountered in literature [21]. Further, a rough qualitative estimation of the expected effect is given [22]. However, quantitative research is necessary to determine a more concrete impact of a best practice on the performance of a workflow.
Although not much is known about the impact of redesign heuristics on the performance of a workflow, some papers have been found that are based on a simulation study. These studies include several heuristics: knockout heuristic [1], extra resources heuristic [9], specialist-generalist heuristic [9, 19], flexible assign-ment heuristic [19], task composition, triage and case types heuristic [26].
The main shortcoming of the above-mentioned literature is that none of the authors provide guidelines for the redesign of workflows: what heuristic should be applied in what situation, process or setting? Other deficiencies are a lack of a general approach to quantify the impact of heuristics, the limited number of different dimensions of performance, and the limited number of aspects per measured dimension. Further, none of them, with the exception of [19], quantifies the impact of the implementation of more than one heuristic at the same time. In our research, we aim to quantify redesign best practices within as many dimensions as possible. This paper provides an overview of available performance frameworks and related performance measures. The resulting performance mea-sures can be used in simulation studies to quantify the impact of a redesign best practice and to evaluate the adequateness of a business process design.
The paper is organized as follows. In Section 2 the dimensions of performance are investigated. Next, each performance dimension is discussed separately (Sec-tion 3 – Sec(Sec-tion 6). The performance measures have been specified for every dimension based on the findings in literature and metrics are proposed. The paper concludes with a reflection on and discussion of the results.
2
Performance measurement
This study focuses on the quantification of a redesign heuristic’s impact on the performance of a business process. Subject of study is the business process that is being redesigned, in contrast to, for example, the performance of individual employees or entire organizations.
In the last twenty years a variety of performance measurement systems has been developed. We assessed this literature to find out what dimensions of per-formance they discern and which are suitable for measuring business process performance. Six different systems have been considered. Here we summarize the dimensions (in some approaches also called categories or perspectives) they discern.
– Performance pyramid: four dimensions on the operational level, i.e. quality, delivery, process time and cost [6].
– Performance measurement matrix: four different dimensions, i.e. internal and external, cost and non-cost measures [12].
– Results/determinants matrix: six dimensions, i.e. competitiveness, financial performance, quality of service, flexibility, resource utilization and innovation [5].
– Balanced scorecard: financial and non-financial measures are combined, re-sulting in four dimensions, i.e. the customer, internal business process, inno-vation and financial perspective [11].
– Devil’s quadrangle: four dimensions, i.e. time, cost, quality, and flexibility. The quality dimension further distinguishes internal quality (pertaining to the workforce) and external quality (of the process and the output) [4]. – Performance prism: this approach indicates what areas an organization must
We assessed the dimensions of the above-mentioned performance measure-ment approaches for possible evaluation of the impact of best practices. In the performance pyramid quality, process time and costs can be considered as per-formance dimensions of business processes. Delivery is a measure that is con-cerned with the timely delivery of the required quantities, which overlaps with the process time from the viewpoint of (administrative) business processes. In the performance measurement matrix the three relevant performance pyramid dimensions can be mapped onto three dimensions of the matrix; the external cost measures dimension is less suitable, because it focuses on customers instead of business processes. In the results/determinants matrix quality of service and flexibility are suitable performance dimensions. Resource utilization is more a measure of the cost or time dimension than a performance dimension on its own. The remaining dimensions refer to organizational performance. From the balanced scorecard and the performance prism only the internal business process is relevant, though it does not add any specific performance dimension. When considering the devil’s quadrangle, we conclude that time, cost and flexibility already have been identified as suitable dimensions. The (internal and external) quality dimension are also relevant when depicting trade-offs and effects of a business process redesign effort.
It can be concluded that the dimensions of the devil’s quadrangle are suitable for measuring the performance of a workflow (see Figure 1). The dimensions cover the concept of performance and the other performance measurement systems do not provide additional relevant dimensions. The next four sections give the performance measures for each of the discerned dimensions.
Time Cost
Quality
Flexibility Fig. 1.The devil’s quadrangle [4]
3
The Time dimension
Time has been described as both a source of competitive advantage and the fundamental measure of performance [18]. Based on the information on time measurements found in the literature [4, 15, 18, 21, 25], a set of performance mea-sures for the time dimension, specifically for workflows, has been derived. Figure 2 shows the relation among these measures.
Client Task 1 Task 2 Client
Move time Queue time Service time Wait time Throughput time Lead time Setup time
Fig. 2.Performance measures - time dimension
Lead time is the time it takes to handle an entire case. Throughput time is the time between the moment a task is completed and the moment the next task is completed. This is composed of:
– Service time: the time that resources spend on actually handling the case – Queue time: the time a case spends waiting in a queue
– Wait time: all other delays for a case, e.g. the time a case has to wait in a parallel branch for completion of all other branches, in order to be able synchronize
– Move time: the time it takes to move a case between tasks
– Setup time: the time it takes to setup a task for a case, for example the time to get acquainted with the case
Based on standard event logs with time information, the above-mentioned measures can easily be calculated (see e.g., [2]).
4
The Cost dimension
The most commonly used performance targets are of a financial nature. The cost dimension is closely related to the other dimensions. For example, long lead
times can result in a more costly process; low quality can lead to expensive rework and low flexibility can also result in a more costly process execution. We focus on the direct costs of running the process. Most of the literature [4, 14, 15, 18, 25] is about cost measurement in production processes, here viewed from a workflow perspective. This resulted in:
– Running costs: costs for executing the workflow
• Labor costs: costs of the workforce (salary×worked hours)
• Machinery costs: an investment in machinery per new automated task • Training costs: costs for training employees (based on the number of
employees and the number of new tasks)
– Inventory costs: costs of keeping records and products (e.g. file cabinets) – Transport costs: costs for moving (intermediate) products and sharing
infor-mation (distance×price per km)
– Administrative costs: costs for keeping the entire workflow intact
– Resource utilization: the ratio of in use time and available time of resources These measures can be categorized as either constant costs or variable costs. Constant costs are costs that are independent of production volume, e.g. invest-ments or labor (size of the workforce is constant) [13]. Variable costs vary with the level of the output, e.g. for part time employees.
5
The Quality dimension
The quality of a workflow can be judged from at least two angles [21]. External quality is defined from the customer’s side, the person or organization that ini-tiates the workflow and will receive the output. Internal quality is defined from the worker’s side.
External quality can be measured as client satisfaction with either the prod-uct or the process. Satisfaction with the prodprod-uct is the degree to which the customer feels that the product is according to specification or feels satisfaction with the delivered product. With the advancement of total quality management (TQM) the emphasis has shifted from “conformance to specification” towards “customer satisfaction” [18]. The satisfaction of a customer with the process concerns how a workflow is executed [21]. Literature has been found on both the quality of a product and the quality of a process [7, 14, 15, 25]. This resulted in the following set of external quality performance measures:
– Quality of the output
• Performance: the degree to which a product’s primary operating charac-teristics meet customer’s requirements
• Conformance: the degree to which a product’s design and operating char-acteristics meet established standards
• Serviceability: the speed, courtesy, competence and ease of correcting mistakes
• Information on application status: the degree to which information on the application status is provided
• Bureaucratic language simplification: the clearness in the presentation • Information availability: the time required to get updated on the status Measuring external quality in a workflow model is less straightforward than measuring time or costs. Many different factors influence and determine the ex-ternal quality. Performance and serviceability may be influenced by: the number of specialists that work on a case, the number of control tasks per case, and the assignment of a case manager, to name a few. The quality of the process may be influenced by two aspects: the assignment of a case manager and the num-ber of employees that work on a case. The more employees work on a case, the lower the expected quality of the process, as the commitment of the employee increases.
Whether a specific aspect influences the external quality of a process and the degree to which it affects the quality is highly dependent on the type of process. It is therefore not possible to capture the measures of external quality in a metric. The same problem came up during the operationalization of internal quality and process modification flexibility (see below). To settle this, we have decided to consider such aspects of external quality as proxies: a change in one or more of the aspects will have some impact on the external quality dimension. However, the exact size of impact cannot be determined in a simulation model. Internal quality can be seen as the quality of a workflow from an operator’s perspective. In this context, internal quality involves the conditions of working [21]. Psychological and social factors are very important. High internal quality can result in high motivation, high job satisfaction, high external quality and low absenteeism. Internal quality is commonly measured with a survey among employees, although some measures might be obtained by simulation as well. The literature contains slightly different, but comparable internal quality measures [10, 16, 17, 20, 21]. This resulted in the following set of performance measures:
– Skill variety: the degree to which a job requires a variety of different activities in carrying out the work, which involves the use of a number of different skills and talents of a person (number of different tasks and case types).
– Task identity: the degree to which the job requires completion of a whole and identifiable piece of work; that is, doing a job from beginning to end with a visible outcome (ratio of number of executed tasks and total number of tasks per workflow).
– Task significance: the degree to which the job has a substantial impact on the lives or work of other people, either in the immediate organization or in the external environment.
– Autonomy: the degree to which the job provides substantial freedom, inde-pendence, and discretion to the individual in scheduling the work and in determining the procedures to be used in carrying it out the work (ratio of number of authorized decisions and total number of decisions).
– Feedback: the degree to which carrying out the work activities required by the job results in the individual obtaining direct and clear information about the effectiveness of his or her performance.
– Co-worker relations: the quality of the relations between an employee and his/her co-workers.
The operationalization of internal quality is comparable to external quality in the sense that it is not possible to capture the measures in metrics. Differences among people moderate how they react to the complexity and challenge of their work [10]. The aspects that can be used as a proxy for internal quality are: the number of executed tasks and case types per resource, the number of contact moments, the number of teams a resource is participating in, the number of authorized decisions and the number of new tasks and case types per resource.
6
The Flexibility dimension
Flexibility is the least mentioned criterion to measure the effect of a redesign effort. Flexibility can be defined as “the ability to react to changes” [21]. This section describes the performance measures for the flexibility dimension found in literature [8, 18, 21, 23–25, 27]. It appears that flexibility can be identified for individual resources, for individual tasks, and for the workflow (process) as a whole. The following set of flexibility performance measures is proposed:
– Mix flexibility: the ability to process different kinds of cases • for resources: number of case types a resource can handle • for tasks: number of case types a task can handle
• for the workflow: number of case types that can be handled – Labor flexibility: the ability to perform different tasks
• for resources: number of executable tasks
• for the workflow: available resources per task per case
– Routing flexibility: the ability to process a case using multiple routes (num-ber of different sequences in the workflow)
– Volume flexibility: the ability to handle changing volumes of input (available time per employee)
– Process modification flexibility: the ability to modify the process (number of subflows in the workflow, complexity, number of outsourced tasks, etc.) For each of these types, different aspects of flexibility can be considered, such as range (the range of variations that can be handled), time (the amount of time required to adapt to change) and cost (the amount of money required to adapt to change).
Another way of looking at flexibility mentioned in literature is to discern run time flexibility and build time flexibility. “Run time flexibility concerns the pos-sibilities to handle changes and variations while executing a specific workflow. Build time flexibility concerns the possibility to change the workflow structure” [21]. This distinction is already present in the set of measures. Process modifi-cation flexibility is comparable to build time flexibility and the other measures are all run time flexibility types.
7
Discussion
The application of heuristics in a redesign effort is a fruitful approach. However, little quantitative data is available to predict the impact of the application of a best practice. A few studies provide quantitative data, but these have a limited scope. This paper provides an overview of different dimensions for performance measurement and for each dimension an overview of possible performance mea-sures. The dimensions are taken from literature on performance measurement systems, and especially from the devil’s quadrangle [4].
For the dimensions of time and costs, the computation of the measures in a simulation model is quite straightforward. Determining the external quality of the output and the process with a simulation model is much more difficult, or even impossible. We identified a large number of different proxies, of which a few were mentioned in this paper. Although this set of proxies can be computed in a simulation study, this does not do complete justice to the real complexity of external quality in practice. Something similar holds for the internal quality measures; these can be implemented, but also in this case this will provide a lim-ited view on what will happen in reality. Furthermore, internal quality is highly dependent on the perception of the person involved. With respect to the flexi-bility measures, all measures can be implemented (except process modification flexibility). Here we expect that actual changes in our metrics’values accurately reflect what will happen in practice.
The research is suitable both for quantifying the impact of redesign best practices, as well as for measuring the adequateness of a process design in general. The discussion above provides a twofold conclusion when applying simulation in this context. For a large part the approach is very well applicable and several measures have been identified for this purpose. For a smaller, but very significant part, we conclude that a simulation approach does not provide the desirable insight. Since the respective identified performance dimensions are all relevant, different methods are necessary to complement the analysis of the impact based on the devil’s quadrangle.
References
1. W.M.P. van der Aalst. Reengineering Knock-out Processes. Decision Support Systems, 30(4):451–468, 2001.
2. W.M.P. van der Aalst, B.F. van Dongen, J. Herbst, L. Maruster, G.Schimm, and A.J.M.M. Weijters. Workflow mining: a survey of issues and approaches.Data and Knowledge Engineering, 47(2):237–267, 2003.
3. C. Adams and A. Neely. Prism Reform. Financial Management, 5:28–31, 2002. 4. N. Brand and H. van der Kolk. Workflow Analysis and Design. Kluwer
Bedrijf-swetenschappen (In Dutch), 1995.
5. T.J. Brignall, L. Fitzgerald, R. Johnston, and R. Silvestro. Performance Measure-ment in Service Businesses. Management Accounting, 69(10):34–36, 1991. 6. K.F. Cross and R.L. Lynch. The “SMART” Way to Define and Sustain Success.
7. D.A. Garvin. Competing on the Eight Dimensions of Quality. Harvard Business Review, 1987.
8. D. Gerwin. An agenda for research on the flexibility of manufacturing processes. International Journal of Operations & Production Management, 7(1):38–49, 1987. 9. R.H.J.J.M. Goverde and H.A. Reijers. Resource Management: a Clear-Headed Approach to Ensure Efficiency. Workflow Magazine, 4(6):26–28 (In Dutch), 1998. 10. J.R. Hackman and G.R. Oldham. Motivation through the design of work: Test of a theory. Organizational Behavior and Human Performance, 16(2):250 –279, 1976. 11. R.S. Kaplan and D.P. Norton. The Balanced Scorecard: Measures that Drive
Performance. Harvard Business Review, 70:71–79, 1992.
12. D.P. Keegan, R.G. Eiler, and C.R. Jones. Are your Performance Measures obsolete? Management Accounting, 70(12):45–50, 1989.
13. E. Lekanne. Bedrijfseconomie Gids: Begrippen uit de Bedrijfseconomie. Uitgeverij NIB, Zeist (In Dutch), 1988.
14. G.K. Leong, D.L. Sneyder, and P.T. Ward. Research in the process and content of manufacturing strategy. Omega, The International Journal of Management Science, 18(2):109–122, 1990.
15. A. Longo and G. Motta. Design Processes for Sustainable Performances: a Model and a Method. InProceedings of the Workshop on Business Process Design, 2005. 16. F. van de Looij and J. Benders. Not just money: Quality of working life as
em-ployment strategy. Health Manpower Management, 21(3):27–33, 1995.
17. V. Mehta and H. Shah. Characteristics of a work organization from a lean per-spective. Engineering Management Journal, 17(2):14–20, 2005.
18. A. Neely, M. Gregory, and K. Platts. Performance measurement system design: A literature review and research agenda. International Journal of Operations & Production Management, 15(4):80–116, 1995.
19. M. Netjes, W.M.P. van der Aalst, and H.A. Reijers. Analysis of Resource Con-strained Processes with Colored Petri Nets. In K. Jensen, editor, Proceedings of the Sixth Workshop and Tutorial on Practical Use of Colored Petri Nets and the CPN Tools, 2005.
20. C.H.P. Pava. Old and new sociotechnical approaches. National Productivity Re-view, 4(1):87–89, 1984.
21. H.A. Reijers. Design and Control of Workflow Processes: Business Process Man-agement for the Service Industry. Springer Verlag, Berlin, 2003.
22. H.A. Reijers and S. Limam Mansar. Best Practices in Business Process Redesign: an Overview and Qualitative Evaluation of Successful Redesign Heuristics.Omega, The International Journal of Management Science, 33:283–306, 2005.
23. A.K. Sethi and S.P. Sethi. Flexibility in manufacturing: A survey. International Journal of Flexible Manufacturing Systems, 2(4):289–328, 1990.
24. N. Slack. The flexibility of manufacturing systems. International Journal of Op-erations & Production Management, 7(4):35–45, 1987.
25. A. De Toni and S. Tonchia. Performance measurement systems: Models, charac-teristics and measures. International Journal of Operations & Production Man-agement, 21(1):46–70, 2001.
26. M. Zapf and A. Heinzl. Evaluation of Generic Process Design Patterns: An Ex-perimental Study. In W.M.P. van der Aalst, J. Desel, and A. Oberweis, editors, Business Process Management: Models, Techniques, and Empirical Studies, volume 1806 ofLNCS, pages 83–95. Springer-Verlag, Berlin, 2000.
27. Q. Zhang, M.A. Vonderembse, and J. Lim. Manufacturing flexibility: Defining and analyzing relationships among competence, capability, and customer satisfaction. Journal of Operations Management, 21:173–191, 2003.