Process Modeling for Process Improvement

(1)

-A Process Conformance -Approach

Sigurd Thunem

(2)

(3)

Abstract

There is a continuous pressure on organizations to improve their work processes. In order to improve these processes, knowledge about them is necessary. To support process improve-ment the organization should collect process data, transform process data into knowledge and then insert this knowledge back into the organization.

The goal of this thesis is to support process improvement in software development organi-zations. Our approach is based on process modeling, and focuses on process models as a medium for knowledge transfer. We also emphasize that such models must be usable in real life, making the relation between audience and model extremely important.

Process modeling has different foci. These range from human understanding to process au-tomation, and a single process model cannot support all foci. For process improvement the social factors are dominating, and process models should therefore focus on human interpre-tation, not machine interpretation.

If we want to support process improvement with process models, process conformance is es-sential. The thesis develops a theory for process conformance, and examines the different aspects of conformance. In a process improvement context, the focus should be on fidelity. Fidelity ensures that model interpretations match the abstractions of the process performers, and thus supports model validation. Fidelity is supported by variable complexity, by navi-gation support and by different views of the process.

To construct process models a process modeling language is needed. The thesis presents a set of goals and requirements for such a language, making a tradeoff between conflicting re-quirements. From these requirements a software process modeling language is developed. This language is optimized for support of process improvement, and therefore aim to develop models with high fidelity.

All processes must execute within the context of an organization. Processes are influenced by their context, and thus the relations between the organization and its processes are essential. Our language provides an owner relation, connecting organization elements and processes. Processes are also organized into hierarchies, enabling abstractions and specializations. The language supports different views using the process and activity concepts. The process con-cept represents the external view of a task, while activities represent the internal view. Thus different viewpoints are supported within the same model.

The language is extended with rules and deontic operators. These provide dynamic process-organization relations, including the notion of time. These mechanisms can be used to repre-sent complex control flows or constraints not easily expressed using the basic relations. The extensions are optional, and not necessary to produce fully operational process models. Thus our language supports variable complexity.

A method for language design has also been developed. This method can be used to con-struct modeling languages, using sound engineering principles. The goals, requirements and rationale of the design processes are captured and stored during development. This method is used to construct our modeling language, enabling readers to examine the whole construc-tion process. We suggest that this will communicate far more insight than just the finished language.

(4)

(5)

Preface

This thesis did not end up as I once envisioned it. When I started my Ph.D. work back in 1992, my main interests were object-orientation and reuse. I was working on the REBOOT project, an ESPRIT project focusing on reuse of software components. This project developed both tools and methods enabling reuse, and my effort contributed mainly to the method parts. This work was later published as “Software Reuse - A holistic approach” [94].

My idea at that time was to look at the characteristics of software reuse. Some types of soft-ware are easier to reuse than others, and some companies are better at reuse than others. While this was an interesting topic, it never got further than the initial stages.

Instead my interests gravitated towards process modeling. Process modeling was (and still is) the main focus of the group for Programming and Systems Development, and as I was part of this community, I was influenced by the work done here. At that time development of the EPOS environment was central in the process modeling activity, but the focus was gradu-ally changing. Where technical issues had been dominant, the non-technical and social issues were slowly gaining acceptance.

I have always been interested in stories and rationale, and the most interesting question for me was alwayswhy ?. While process modeling was an interesting and demanding topic, I needed a good reason for spending effort on it. At the same time I was supervising two M.Sc. students examining the use of quality assurance methods in Norway. While quality assur-ance is a large and multi-faceted topic, many organizations focus on process improvement. And suddenly it all made sense. If process models can be used in process improvement, the effort of model development is certainly well spent.

The match between process improvement and process modeling is definitely not original. Most process modeling approaches state improvement as one of their goals, but I intended to go further than that. If we accept that process improvement is the goals of process model-ing, this must certainly influence the language used to develop these models. But while most languages stated improvement as a goal, the actual influences were not discussed.

Thus the link between process improvement and process modeling became the nucleus of my interest. I wanted to show how the requirements of process improvement influenced process modeling languages, and how process modeling could support process improvement. Most of this thesis is dedicated to showing the characteristics of this link. While it will present a process modeling language, this language is less interesting than the reasons for why it ended up as it did. Thus I allocate far more space and effort to explanations and rationale than usual for process modeling languages.

(6)

I envision this thesis as input to an ongoing process, not as the end result. The perfect pro-cess modeling language has not been developed, and will never be developed. I have put much emphasis on explanations, and hope that these enable knowledge transfer to my read-ers. While process modeling is getting out of its infancy, we have still some way to go before we reaches our target.

Acknowledgments

There are several people who have been essential to the completion of this thesis. I would like to thank my advisors Guttorm Sindre and Reidar Conradi for their guidance and comments throughout this work. Reidar has been the rock, always pushing me a bit further towards completion. And Guttorm has been truly exceptional, giving guidance for my thesis work as much as two years after he officially quit his job at the university (and thus ended the official period as my advisor).

Furthermore, I want to thank Geir Høydalsvik and Sivert Sørumg˚ard. We started on our the-ses at the same time, and even shared the same office space at times. We have had many in-teresting discussions (not all of them academic), and I have many good memories from these years.

I spent the last year of my thesis work at PAKT,1_{and it has been interesting to sample a}

dif-ferent research community. There are three people at PAKT that deserve special mention, the first is my “boss” Hans Tilset. Without his flexibility this thesis would not have been finished. In addition Terje Totland and Tor G. Syvertsen have been very inspirational, and great part-ners in afternoon discussions.

The final thanks go to those who made it all possible. It is many years since my parents put me on this track, and they have given invaluable support all this time. They never doubted me, and without their belief I am not sure I could have done it. But the heaviest burden was put on my wife, Hilde. She has shouldered much of my worries, and I am eternally grateful for her support when I needed it most.

Trondheim, September 12, 1997.

Sigurd Thunem

(7)

12.2.5 Model quality . . . 251 12.3 General applicability . . . 252 12.4 Chapter summary . . . 254 13 Conclusions 255 13.1 Summary of contributions . . . 255 13.2 Further work . . . 256 13.3 Conclusion . . . 258 IV Appendices 259 A Syntax of our language 261 A.1 Syntax of the Organization Structure View (OSV) . . . 262

A.2 Syntax of the Process Definition View (PDV) . . . 265

A.3 Syntax of the Deontic Rule View (DRV) . . . 269

B Case study: a software development organization 273 B.1 The organization . . . 273

B.2 Available data . . . 274

B.3 Topics of interest . . . 274

B.4 OSV model . . . 275

(12)

(13)

Chapter 1

Introduction

1.1 Thesis motivation

In today’s world of lean and mean corporations, organizations must continuously improve in order to stay competitive. Most of this improvement comes from working smarter, i.e.process improvement.

The search for improvements has resulted in techniques likeBusiness Process Reengineering (BPR) [67]. While such techniques show some promise, they are seldom based on controlled, scientific principles. Thus organizations use guidelines and loosely defined rules to optimize their processes. From an engineering point of view it is quite surprising that this topic is not treated more seriously.

This thesis discusses process improvement from an engineering point of view. We envision process improvement as a controlled, well defined task, where results and consequences can be predicted with reasonable confidence.

The approach of the thesis is to merge theprocess improvementandprocess modelingparadigms. While many improvement approaches rely on models (and process modeling should result in some improvements), few have examined the relations between these topics. There is cer-tainly a connection between improvement and modeling, but the properties of this relation are unknown.

1.2 An engineering approach to process improvement

To improve the process a method for improvement is necessary. This thesis proposes an en-gineering approach, i.e. to use enen-gineering techniques to improve the process. In this section we limit our interest to the merger of engineering methods and process improvement, the next section will examine the merger of process improvement and process modeling. The term engineering is defined by the Oxford dictionary as“the practical application of scien-tific knowledge in the design, construction and control of ...”[39]. Thus an engineering approach

(14)

must fulfill these criteria:1

The knowledge relevant for the task is collected.

Scientific principles guide the collection and utilization of this knowledge. Thus an engineering approach to process improvement must investigate:

The knowledge necessary for process improvement.

Methods for collection and utilization of process knowledge.

We use the termknowledge perspectivefor this view of process improvement. To ensure that the approach results in some improvements, we add a third topic of interest:

Quality criteria for processes and process knowledge.

These topics are the foundation for our approach to process improvement.

1.3 Problem definition

The previous section examined the merger of process improvement and engineering princi-ples. We promoted aknowledge perspectiveto process improvement, including methods for collection, analysis and utilization of process knowledge. This section examines the integra-tion of process modeling and process improvement.

An approach based on the knowledge perspective can be divided into three phases:

Knowledge collection.Data from the process are collected and transformed into knowl-edge.

Knowledge analysis.The process knowledge is analyzed, and possible improvements identified.

Knowledge transfer.Knowledge about the process and possible improvements is trans-ferred to the organization.

Process modeling supports all three phases. The first phase is equal to model building, where process data are transformed into process models. The second phase is equal to model anal-ysis, where process models are analyzed in order to obtain more specific knowledge about the process. The last phase is equal to model utilization, where process models are used to transfer knowledge in the organization.

Process modeling relies on a representation of the process. The format of this representation varies according to the objective of the modeling effort. Some representations are well suited

(15)

1.4. OBJECTIVES AND APPROACH 3

for reflecting upon individual situations,2_{while other representations lend themselves to}

for-mal analysis or knowledge transfer. The thesis examines howprocess modeling languagescan be used to build such representations. We use the requirements of process improvement as input, and examine if we can construct a process modeling language supporting the chosen focus of process improvement.

We summarize the problem addressed by this thesis as:

How can process modeling support process improvement?

The improvement must be based on a knowledge perspective, and a process modeling lan-guage will be used to develop these models.

1.4 Objectives and approach

The nucleus of our approach is an evaluation of how process modeling might support pro-cess improvement. This evaluation is based on the belief that propro-cess modeling is a useful tool for such improvements. Thus we focus on evidence that points to process modeling as a valid approach. A more general approach would be to evaluate all possibilities for support of process improvement, and then use this evaluation to select the best solution. We have two reasons for rejecting this approach:

The effort needed for such an evaluation is enormous. This would severely limit the resources available to investigate the actual solution.

We would like to integrate process modeling and process improvement. While there might be other solutions to process improvement, we are satisfied with this solution. The disadvantage of our approach is that the focus might become too narrow. We focus on one solution, and relevant knowledge from other approaches might be lost. Thus we cannot claim to support all aspects of process improvement, just the aspects of improvement related to process modeling.

The first step in our evaluation is to survey process improvement approaches. This survey includes quality criteria for processes and process improvement. It also examines how cur-rent process improvement approaches use process models, and the requirements of process improvement.

The next step is to examine the concept of process modeling. The characteristics of process models are explored, and we compare these to the requirements of process improvement. We point out the different uses of process models, and how process models have been used to support improvement. Using this background, we survey some process modeling approaches. Their support for process improvement is evaluated, and strengths and weaknesses identi-fied.

2_{When thinking about the process, each individual builds an internal representation of that process. This}

rep-resentation is informal, and not necessarily complete. We will later use the termabstractionfor this internal rep-resentation.

(16)

The final step is to merge process improvement and process modeling. We define a set of as-sumptions for how process models can support process improvement. The most important assumption is thatprocess conformanceis necessary for support of process improvement. We develop a theory for process conformance, and examine how a process modeling language might support various aspects of conformance. Of these aspects,fidelityis identified as es-sential for support of process improvement.

Our assumptions and theories are used to develop a set of goals for a process modeling lan-guage. Based on these goals, a set of language requirements are defined, and we make a tradeoff analysis between conflicting requirements. From these requirements we develop a process modeling language. This language is optimized for support of process improvement, and is used to strengthen our confidence in the usefulness of process models for process im-provement.

Our investigation must fulfill the criteria of a Ph.D. thesis. In such theses, the only valid ap-proach to problem solving is thescientific approach. This means that the problems addressed by the thesis should be attacked with scientific tools, and the results of the thesis must with-stand scientific scrutiny. We will not go into a detailed discussion of research methods here, but a scientific research process should at least contain the following elements:

The definition phase:What is the problem, and why is this problem important? The research phase:How was the problem solved, and what paradigms were used in this process?

The validation phase:Is the proposed solution a valid solution for the given problem? An important requirement for our work wasa controlled development process. A controlled pro-cess makes validation easier, as it provides a trace of the propro-cess from problem to solution. This trace is present in the thesis as intermediate results. These results are:

Problem statement:A definition of the problem. The survey:A survey of other solutions. Language goals:Goals for the language.

Language requirements:Requirements for the language.

Tradeoff analysis:How requirements were traded against each other. The language:The resulting language.

These results can be used to validate individual steps in the process. If all steps have been validated, we can claim that our approach produces a valid solution to the given problem.

(17)

1.5. CLAIMED CONTRIBUTIONS 5

1.5 Claimed contributions

The objective of this thesis is to explore the suitability of process modeling for process im-provement. Within this focus we claim that the thesis provides unique knowledge. The claimed contributions are:

A framework for how process models can support process improvement. We have surveyed process improvement approaches and process modeling languages, examin-ing possible relations and weakly supported goals. From this survey we developed a set of assumptions for how process models might support process improvement. A theory for process conformance.One of the most central assumptions is that process conformance is essential for successful support of process improvement. A new theory for process conformance has been developed, examining the different aspects of con-formance. This theory surveys the relations between processes, audience abstractions, process models and interpretations, defining a taxonomy for process conformance. A set of goals and requirements for a software process modeling language.Based on our theory and assumptions, we defined a set of goals for a software process model-ing language. We developed a set of requirements for these goals, and made a tradeoff analysis between conflicting requirements.

A software process modeling language suitable for process improvement. Our re-quirements were used to construct a software process modeling language. The aim of this language is to support process improvement, and the language focuses on knowl-edge transfer and high fidelity. We introduce three novel features in this language. The first feature is to use organization structures for navigation in process descriptions. The second feature is the concept of variable complexity. The third feature is the notion of community dependent interpretations. We assume that each community validate their own processes, and allow different interpretations of our models.

A development method.Our language is supported by a development method. This method gives guidelines for how to develop models in our language, and how these models can be used to support process improvement.

A well documented development process.We used a language design method3_{in the}

development of our language. While this method introduces few new concepts, we still claim a contribution. Without this method we would not have the necessary data to de-scribe our language, and still this documentation is extremely uncommon for modeling languages. We therefore claim theideaof presenting this information as our contribu-tion, not the actual information.

We do not claim that the perfect process modeling language has been developed, but it is our opinion that the language provides valuable input to the topic of process modeling. We hope that the features of this language can inspire other developers, and point out that the documentation provided by this thesis makes impact analysis much easier.

3_{This method should not be confused with the method described in the previous point. The language design}

method defines how to develop modeling languages, while our development method gives guidance for model development.

(18)

1.6 Thesis overview

The structure of the thesis corresponds to the phases of our research outlined in section 1.4. The results of the definition phase are reported in the first part, the results of our research in the second part and the validation in the third part. This section describes each part in more detail.

1.6.1 Part 1: Current Practices

The first part of our thesis examines current practice for process improvement and process modeling. We survey several approaches, and examine relevant theory. The purpose is to detect possible relations, and to examine how current process modeling languages support process improvement.

Chapter 2: Process Improvement.This chapter surveys process improvement approa-ches. We define the relevant terms, and examine the relations between process improve-ment and quality. The final part of the chapter surveys four process improveimprove-ment meth-ods. The surveyed methods are the Capability Maturity Model (CMM) [73], the Expe-rience Factory (SEL/EF) [13], ISO-9000 [82] and Business Process Reengineering (BPR) [67].

Chapter 3: Process Modeling.This chapter contains our survey of process modeling. The chapter defines the central terms, and examines different views and paradigms for process modeling. We also explore the relations between process modeling, process im-provement and model quality. The final part of the chapter surveys different process modeling approaches. The surveyed approaches are PPP [65], ALBERT [48], RIN [145], TEMA [154],

[4] and SLANG [9].

Chapter 4: Baseline.The baseline defines the basic theory for our approach. We define a set of assumptions for how process modeling might support process improvement. One of the most central assumptions is that process conformance is essential, and we develop a theory for process conformance. Fidelity is identified as the central aspect for support of process improvement, and we suggest navigation, variable complexity and different viewpoints as mechanisms supporting high fidelity in process models. 1.6.2 Part 2: Process Modeling

The second part contains the main contribution of the thesis. It presents our process modeling language; from goals, through requirements and tradeoffs, to the finished product. The pur-pose of the language is to examine how process modeling might support process improve-ment, not to produce a generic process modeling language. The previous part examined the suitability of process modeling on a theoretical basis, this part explores its practical applica-tion.

Chapter 5: Designing Process Modeling Languages. This chapter defines a design method for process modeling languages. The method is built on well know software

(19)

1.6. THESIS OVERVIEW 7

development methods, and the purpose of this chapter is to introduce the method used in the design of our language.

Chapter 6: Goals and Requirements.This chapter contains thegoalsandrequirements for our language, these are organized into hierarchical structures. We identify three clusters of requirements, and make atradeoff analysisfor each cluster. The language is divided into three views, where each view corresponds to a cluster.

Chapter 7: The Organization Structure View. This chapter presents the first of the three language views. This view is used to build thestructuresof theorganization ex-ecuting the process. We present the concepts used in the view, and discuss syntax and semantics. The view is also evaluated against its requirements.

Chapter 8: The Process Definition View.This chapter presents the second view of our language. This view is used to represent theprocessesof the organization. We present the concepts used in the view, and discuss syntax and semantics. The view is also eval-uated against its requirements.

Chapter 9: The Deontic Rule View. This chapter presents the third view of our lan-guage. This view extends the expressive power of the previous views usingrules. We present the new concepts of this view, and discuss the syntax and semantics of these concepts. The view is also evaluated against its requirements.

Chapter 10: The Development Method.This chapter contains the method for building models in the language. It presents guidelines for model development, including how to use models developed in our language.

1.6.3 Part 3: Evaluation and Conclusions This part contains the validation of our approach.

Chapter 11: Case Studies.This chapter contains two models developed using our guage. The purpose of these models is to show the practical application of our lan-guage.

Chapter 12: Evaluation.The evaluation chapter contains the evaluation of our work. This evaluation considers both the validity of the language with respect to its goals and requirements, and the validity of our solution with respect to the requirements and frameworks of others.

Chapter 13: Conclusions.The purpose of this chapter is to summarize the major find-ings of the thesis. This includes a summary of the major contributions and suggestions for further work

1.6.4 Part 4: Appendices

(20)

Appendix A: Syntax of our language.This appendix presents the formal syntax of our language using BNF-syntax.

Appendix B: Case study: a software development organization.This appendix con-tains the textual descriptions for our largest case.

(21)

Part I

Current Practices

(22)

(23)

Chapter 2

Process Improvement

The purpose of this chapter is to give an introduction toprocess improvement. This requires an examination of the various views of process improvement, including different goals and foci of improvement.

We start with a discussion of the terminology and definitions of process improvement. In or-der to measure improvement a notion of process quality is required, and we look at possible definitions. We survey generic process improvement approaches, including a short evalu-ation of each approach. We also investigate whether software process improvement might have a different focus than process improvement in general.

2.1 What is process improvement

Process improvement can be done in all organizations, and there exist numerous approaches to process improvement. These approaches have different goals, and focus on different as-pects of the process. There is also considerable confusion about the exact meaning of the termsprocessandimprovement, and this section contains the definitions used in this thesis. If we consider the terms,improvemeans“to bring into a more desirable or excellent condition” or“increase in quality or value”[163]. Thus process improvement involves an increase in the value or quality of the process. We therefore need a definition of process quality, and this will be discussed in section 2.3.

The Webster dictionary defines aprocessas“a systematic series of actions directed to some end” [163]. Thus processes are structured and goal-oriented. They can be broken down into smaller fragments, where the sum of fragments (and their order) equals the process. This definition is fairly common for the general process concept [32, 54].

However the combination of“improvement”andprocess”is not interpreted uniformly. The interpretations can be separated into two views [149]:

Improvementofthe process.

Improvement of somethingby means ofthe process. 11

(24)

These alternatives are a result of different foci for the process. One focus is improvement of theactionsof the process, i.e. the process itself. Another focus is thegoalof the process, i.e. improvement of what the process produces.

If we look at process improvement as a vehicle for improvement, there must be a direct re-lation between the process and what we aim to improve. The most common target for im-provement is theproductof the process. In this case we aim to improve the quality of this product, e.g. defect rate, performance or customer satisfaction. In these approaches the pro-cess is a tool used to measure the product quality. Examples of this approach are Business Process Reengineering (BPR) [67] and the ISO-9000 standards [82].

The other focus for improvement is the process itself. This approach relies on aprocess quality definition, and aims to improve aspects like efficiency, predictability, scheduling and confor-mance. The product of the process might have some importance in this approach, but only as an indicator for process quality. The Experience Factory (SEL-EF) [13] and the SEI Capability Maturity Model (CMM) [73] both confirm to this view.

This thesis supports the process centered view of improvement. While the product is an im-portant target for improvements, the focus should be on the actual process. Thus products and product quality are of secondary importance in this work.

Regardless of the focus chosen, most improvement approaches use the model of figure 2.1 for improvement. The main difference is the knowledge collected, either process or product

Collect Knowledge Identify Improvements Execute Analyze

Figure 2.1: A model of process improvement.

knowledge. We see that improvement never ends, when a cycle is completed we just start again from the beginning. Each cycle infuses additional knowledge into the organization, something emphasized by SEL-EF [13].

2.2 Software process improvement

If software processes are substantially different from processes in general, they might require a different focus for improvement. In order to investigate this claim, we examine some

(25)

def-2.2. SOFTWARE PROCESS IMPROVEMENT 13

initions of the software process, and whether these make software processes fundamentally different.

It is important to remember that processes only exist in the real world. They can be observed, described and reflected upon, but a process is always equal to a sum of real-world actions. These actions need not be purely physical; observation and reflection are examples of actions with no physical component. When we observe the process, we build anabstractionof the process. This abstraction is what we use when we improve the process, however it is usually not part of the process itself.

Figure 2.2: A process and its abstractions. With this distinction in mind, we define a software process as:

the actions performed by an organization in order to produce software.

This definition is similar to those of Dowson“a set of related steps both for creating, and for main-taining and evolving software systems”[45] and Humphrey“that set of tasks that, when properly performed, produces the desired results”[73], but with one important distinction. Those defini-tions donotseparate the process and its abstraction, in fact the software process is equal to an abstract set of steps/tasks that can be used to produce software. In our opinion the dis-tinction between a real-world process and its abstractions is important, and we will use this division in our theory for process conformance.1

There exists another, somewhat different view of software processes. An example is the def-inition provided by Lonchamp, where a software process is equal to“A set of partially ordered process steps, with sets of related artifacts, human and computerized resources, organizational struc-tures and constraints...” [110]. In this view, the software process includes everything needed to produce software. In our opinion it is important to distinguish the tasks from their envi-ronment, and thus we reject this definition because its too broad.

Software processes do not exist in a vacuum. They are always executed by an organization, this is depicted in figure 2.3. The actions of the process are done bypeople, possibly playing differentroles. It produces a set ofproducts, consumingresourcesduring execution.

(26)

Process Product Resources People/Roles ORGANIZATION Produces Context Organization Structures Execute Consumes

Figure 2.3: A process and its environment. 2.2.1 The software crisis

By rejecting the broad definition of the software process, we made software processes similar to processes in general. However there might be another source of difference. If software development has special problems, this might result in another focus for improvement. The major problems of software development are [55]:

Cost:Software development frequently results in cost overruns, and software develop-ment is generally considered too expensive.

Productivity:The productivity of software development is lower than the demand, and the requirements are considered to grow faster than the improvement in productivity [20, 45, 103].

Quality: The quality of software is generally low, and ensuring high quality is often costly.

We claim that one of the major reasons for the problems of software development is the de-sign riskinherent in current development methods [25]. This is reflected in the emphasis on predictabilityin some process improvement approaches (e.g. CMM and SEL-EF). With an un-known process and a high percentage of design work, the predictability of cost, productivity, and quality becomes very low. Thus software development projects often fail.

If we aim to improve the predictability, knowledge is essential. Based on the assumption that low predictability is a major problem in software development,process knowledgebecomes the key focus for improvement of software processes. Without this knowledge, predictability will stay low, and improvements will be a matter of luck. This observation fits well with our knowledge perspective to process improvement, and strengthens our belief that knowledge is essential to successful improvements.

(27)

2.3. PROCESS IMPROVEMENT AND QUALITY 15

2.3 Process improvement and quality

We previously defined that process improvement is equal to an increase in process quality. To progress in our investigation, we need a definition of process quality. This section provides this definition, based on the work of Fenton [55] and Sørumg˚ard [149]. We also examine a more general quality framework, including product and resource quality.

2.3.1 Quality in software development

The term quality is hard to define. It encompasses different aspects (e.g. “superiority”and “personality trait”[163]), and varies according to the subject studied. In software development the term is usually interpreted as [32]:

The degree to which a system, component or process meets specified requirements. The degree to which a system, component or process meets customer or user needs. The first interpretation defines anabsolutequality criterion; the degree of requirements fulfill-ment. This assumes that a definition of quality is somehow included in these requirements. The second interpretation is a relative measure, it regards quality as the relation between user needs and provided results.

Regardless of approach, the quality of an object can be broken down intoquality factors[32]. These factors vary according to the purpose of the object. Thus we have severalpoints of view [149], with a set of quality factors for each view.

Developer’s view Manager’s view Customer’s view

Product

quality Resourcequality Process_quality Quality matrix

Process quality

Flexible Developer’s view

Figure 2.4: A multidimensional view of quality (adapted from [149]).

The basic entities for software development areproducts, processesandresources[55]. Com-bined with the views ofmanagers, developersandcustomers, they provide a multidimensional quality matrix depicted in figure 2.4.

(28)

2.3.2 Process quality

One approach to process quality is to select product quality as the most important factor for process quality.2_{While this has some merit, it is too coarse-grained if we aim to do}

improve-ment of the process. While product quality is an indirect measure of process quality, we need more detailed descriptions in order to isolate areas of interest.

From figure 2.4 it is also clear that different users have different quality goals. This influences the quality factors selected, and we reject one uniform quality definition for processes. The quality factors for each view are shown in figure 2.5. This model will be used to evaluate the

Conformance Measurability Repeatability Predictability Efficiency Cost Challenge Bureaucracy Usability Flexibility Adaptability Stability Complexity Traceability Conformance

Manager’s View Developer’s View Customer’s View

Figure 2.5: Process quality.

support provided by the process improvement approaches surveyed.

2.4 Process improvement approaches

There are many lanes of approach to process improvement. In addition to the difference be-tween improvementofthe process and improvementusingthe process, different approaches focus on different parts of the process, differ in paradigm and goals, and argue the exact def-initions of process and improvement.

In the previous sections we discussed the terms and definitions used in this thesis. We also limited our scope of interest, and examined possible foci within this scope. This section sur-veys four process improvement approaches fitting this scope. These approaches share some common characteristics. They providegeneralframeworks, and propose an approach cov-ering most aspects of improvement. Their focus is on the organization, and less on techni-cal support and solutions. This is necessary for general approaches, and not considered any drawback.

The selected approaches are:

2_{This is not uncommon in process improvement approaches using the process as a medium for change (see}

(29)

2.4. PROCESS IMPROVEMENT APPROACHES 17

The Capability Maturity Model:CMM is the most widely used improvement paradigm for software development organizations.3_{It is developed by the Software Engineering}

Institute (SEI), and provides a framework for processmaturity. This framework can be used forassessmentorevaluationof software processes.

ISO-9000:The International Organization for Standardization has developed the ISO-9000 series of standards. These standards provide a framework for quality assurance in organizations, and is widely used in Europe. The standards are used to certify orga-nizations, and are often compared to the evaluation part of CMM.

The Experience Factory:The Software Engineering Laboratory has developed the Goal/-Question/Metric (GQM) and Quality Improvement Paradigm (QIP) methods. The sum of these methods are the Experience Factory, providing a framework for process im-provement.

Business Process Reengineering: Business Process Reengineering (BPR) has been a popular approach to process improvement. We examine the concepts of this approach, and its applicability to process improvement.

Two of the approaches were developed specifically for software process improvement (CMM and EF), while the other two can be used for any process. While there are other methods with process improvement aspects (e.g. Statistical Process Control [42] and Total Quality Manage-ment [53]), these differ little from the surveyed methods. Thus our survey presents a balanced view of the domain.

2.4.1 The Capability Maturity Model

The Capability Maturity Model is developed by the Software Engineering Institute (SEI). It proposes amaturityparadigm for organizations, where mature organizations have a greater capacity for process improvement. It is widely used, especially among organizations con-nected to the U.S. defense industry. This survey is based uponManaging the Software Process [73] andThe Capability Maturity Model[130]. Other references are [40, 72, 75, 76, 126, 127, 128, 129].

History and motivation

The work on CMM began in November 1986. The purpose was to develop“a process matu-rity framework that would help organizations improve their software process”[130]. The first major publication on the framework wasManaging the Software Process[73], and version 1.0 of the framework was released in 1991 [127]. The experiences from assessments and evaluations were incorporated into version 1.1, released in 1993 [128, 129].

The motivation for CMM was the“growing perception that software quality is the weak link in de-veloping high-quality products”[130]. The promises about productivity and quality gains were unfulfilled, and new methodologies and technologies ineffective. A new focus was necessary

(30)

in order to keep pace with the growing complexity of software development. CMM focuses onmanagingthe process, usingengineeringpractices. The mission of the SEI is to“provide lead-ership in advancing the state of the practice of software engineering”[130], and CMM is their main contribution towards this goal.

Improvement paradigm

The improvement paradigm of CMM is built upon the concept of organizationmaturity. The capabilities of an organization are separated into [129]:

Software-process performancerepresents the actual results achieved by following the soft-ware process. This is equal to the quality of the softsoft-ware produced by the current pro-cess.

Software-process capabilityis the range of results that can be achieved with the current process. It represents the possible products that can be produced, and can also be used to predict the most likely outcome of the next project.

Software-process maturityis the potential for growth. It indicates the possibility for ex-tending the current capability, and is the result of an explicitly defined, managed, mea-sured, controlled and effective process.

Managed 5 Optimized 5 Defined 3 Repeatable 2 Intial 1 Disciplined process Standard, consistent process Predictable process Continuously improving process

Figure 2.6: The five levels of software process maturity [126].

CMM is a framework describing the capability and characteristics of mature organizations, and can be used to determine the maturity of an organization (evaluation) and for improve-ment (using assessimprove-ments).

Figure 2.6 shows the five levels of maturity defined by CMM. An immature organization starts at level one, and must progress towards level five one step at a time. Thus the improve-ment paradigm in CMM is quite narrow, there is only one way to maturity.

(31)

2.4. PROCESS IMPROVEMENT APPROACHES 19

To describe each level, CMM defines a set ofkey process areas(KPA). These define issues that the organization must address to achieve the maturity required for that level. The key process areas of CMM 1.1 are shown in table 2.1. As all organizations manage level one by default,

Repeatable Defined Managed Optimizing

Software Configuration Peer Reviews Software Quality Process Change

Management Management Management

Software Quality Intergroup Qualitative Process Technology Change

Assurance Coordination Management Management

Software Subcontract Software Product Defect

Management Engineering Prevention

Software Project Integrated Software Tracking and Oversight Management Software Project Training Program Planning

Requirements Organization Management Process Definition

Organization Process Focus

Table 2.1: Key Process Areas of CMM.

there are no KPA’s for this level. To increase their maturity, organizations must perform the set of activities defined for each KPA. These activities were observed in successful software production organizations, and CMM postulates that they are necessary for mature organiza-tions. Thus CMM proposes a prescriptive approach to process improvement.

Process quality view

The process quality view of CMM has a clear bias. While the quality view of managers is strongly supported, developers have almost no support. The customer view was strength-ened in CMM 1.1, but is still less pronounced than in ISO-9000.

The different maturity levels provide different views of process quality. Figure 2.7 presents our evaluation of the support provided for the different quality factors, and the maturity level it is introduced. From this figure we find that only the higher levels of maturity provide a well developed quality view.

Evaluation

This evaluation focuses on CMM as a process improvement framework. This rules out some of the criticism of CMM, e.g. problems related to the evaluations [25].

In our opinion, CMM is a concrete, well defined and consistent approach to process improve-ment. The published material is very well written, and the approach well suited for practical

(32)

Conformance Measurability Repeatability Predictability Efficiency Cost Challenge Bureaucracy Usability Flexibility Adaptability Stability Complexity Traceability Conformance Manager’s view Developer’s view Cust. view

No Support Full Support

Level 2

Level 3

Level 4

Level 5

Figure 2.7: The process quality view of CMM.

applications. If one accepts the paradigm, we doubt that any better approach exists. Some questions have been raised about the validity of the approach [149], but in our opinion the necessary effort has been made to confirm the validity.

There are however some problems with the paradigm of CMM. We will examine some weak-nesses, and their influence on process improvement:

Prescriptive:The paradigm of CMM is both prescriptive and narrow. There are some activities that must be done in all mature organizations, and there is only one way up the maturity ladder. This has raised doubts whether CMM is too strict in its approach to process improvement [25], and while local adaptations of CMM are encouraged [73], little support is provided for such adaptations.

Managerial focus:CMM focuses on management activities and support for the man-ager’s view of quality (see figure 2.7). Developers have little support, and especially troublesome is the lack of support for complexity and bureaucracy reduction. These factors are quite important for the developer’s view of process quality, and CMM can not claim adequate support for these aspects of process improvement.

Overhead:In the CMM-approach, many essential process improvement factors are only applicable for level four and five organizations (e.g. technology change management and control of process performance). These levels are burdened by administrative over-head, and at the present time only large organizations have reported success at these levels. Most organizations are at level one to three, and only time will show whether the higher maturity levels can be reached for all kinds of organizations.

(33)

2.4. PROCESS IMPROVEMENT APPROACHES 21

Thus CMM is a valid approach to process improvement, but it is definitely notthesolution.

2.4.2 ISO-9000

ISO-9000 is a series of standards developed by the International Organization for Standard-ization. These standards define“quality systems requirements that can be used for external quality assurance purposes”[82]. The focus of the standards is to provide a supplier with a means to “demonstrate its capability, and for the assessment of such supplier capability by external parties”.

Overview and motivation

The ISO-9000 standards are general manufacturing standards, and are used by all kinds of manufacturing organizations. In this survey we limit our interest to the application of ISO-9000 for software development organizations, the relevant standards are [80]:

ISO-9000-3 [81]:Guidelines for how to interpret the ISO-9001 standard for software de-velopment.

ISO-9001 [82]:Specification of a quality management system for use in design, devel-opment, production, installation and service. The focus is on disciplines which involve design.

ISO-9126 [83]:Software product evaluation, quality characteristics and guidelines for their use.

In addition to these, the ISO-9002 and ISO-9003 standards can be used in software develop-ment. These are complimentary subsets of ISO-9001 [61] and we focus on the latter. The usual application of ISO-9000 is as a standard forcertification, and as ISO-9001 is the basis for such certifications it will dominate this presentation. The guidelines of ISO-9000-3 and ISO-9126 provide additional information, but change no fundamental concepts.

Improvement paradigm

ISO-9001 is a quality assurance standard, and thus focuses less on process improvement than the other surveyed approaches. However a well-defined quality assurance system must con-tain some elements of process improvement, and this is reflected in the standard. These as-pects are the focus of this survey, and asas-pects of ISO-9001 without any influence on process improvement will not be mentioned.

The main purpose of ISO-9001 is to regulate the relations between supplier and customer. This is done by certification of suppliers, customers are then assured that the supplier follow the rules of the standard.4_{The standard only regulates the quality system of the supplier, and}

the improvement focus is therefore quite narrow.

(34)

The standard aims“primarily at achieving customer satisfaction by preventing nonconformity at all stages”[82]. It contains twenty requirements for the quality system, and all of these must be fulfilled to achieve certification. Descriptions of these requirements can be found in [61, 80, 82]. ISO-9001 is strictly product centered, and the quality of the development process is of secondary importance. The standard includes no requirements for a particular level of qual-ity or performance, that must be arranged by the supplier and customer for each contract. The process improvement aspects of ISO-9001 are coupled to the requirements:

REQ 9: Process control.The organization must plan and verify all processes affecting quality. This can be used to gather the knowledge necessary to start improvement. REQ 14: Corrective and preventive action.The standard requires procedures for han-dling of nonconformities. While removal of nonconformities is a somewhat narrow im-provement paradigm, experience has shown that companies consider this area the most difficultandthe most useful of all requirements [109].

REQ 17: Internal quality audits.Internal audits are used to assess the current situation and potential for improvement.

Thus improvement using ISO-9001 focuses on measurements of nonconformance. It has no support for how to avoid nonconformance, and how to implement changes is not mentioned. Process quality view

Not surprisingly thecustomerview of quality is strongly supported by ISO-9000. The support for the other views are harder to assess, as the standard is quite abstract. The assessment depends on the interpretation of the requirements, and an evaluation therefore becomes quite subjective. The evaluation reported in figure 2.8 is based on the wording of the standard, and an evaluation based on current practice might look somewhat different.

From this evaluation we find that ISO-9000 provides little support for the quality view of de-velopers, while managers are a bit better supported. The customer view of quality has full support for all aspects.

Evaluation

It is hard to give to a thorough evaluation of ISO-9000. Because of the lack of details in the standards, there is considerable room for interpretation, and practice varies considerably. This evaluation focuses the published standards, and we limit our comments to the process im-provement aspects.

The main problem with using ISO-9000 for process improvement is the focus on quality assur-ance. Process improvement is only indirectly supported, and an organization can be certified for ISO-9001 with very little process improvement support. Other problems are:

Focus on documents:ISO-9000 depends on documents for traceability and demonstra-tion of capability. How to prepare documents and how to use their contents are weakly

(35)

2.4. PROCESS IMPROVEMENT APPROACHES 23 Conformance Measurability Repeatability Predictability Efficiency Cost Challenge Bureaucracy Usability Flexibility Adaptability Stability Complexity Traceability Conformance Manager’s view Developer’s view Cust. view

No Support Full Support

Figure 2.8: The process quality view of ISO-9000.

supported, and this might lead to a large bureaucracy without any impact on the pro-cess.

Product centered:While customers are mainly concerned with the product quality, sup-pliers should consider process quality just as important. The focus on product quality makes process quality harder to assess.

Minimum requirements:ISO-9001 is often considered a minimum criterion for an ad-equate quality system [126]. While this depends on the interpretation of the standard [109], current practice has shown this to be true.

Certification:Certification is problematic in a process improvement method. It makes the organization focus on the certification requirements, and it is hard to go on from there. As ISO-9000 is fairly lenient on process improvement requirements, this might result in certified organizations not bothering with improvements.

Thus ISO-9000 is far from ideal as a process improvement approach. However it is widely used, and surveys have reported that implementation has resulted in process improvement [109]. While other methods might be just as good, it is far better than nothing.

2.4.3 The Experience Factory

The Experience Factory is a quality improvement methodology developed by the Software Engineering Laboratory (SEL). SEL has been active since 1976, and is a partnership between NASA/Goddard Space Flight Center, the University of Maryland and Computer Sciences

(36)

Corporation. The main goals of SEL are to understand the software process in a production environment, to determine the impact of available technologies and to infuse these methods back into the development process [114]. This survey is based on [12, 13, 16, 114].

Motivation and overview

The Experience Factory (EF) is a virtual organization supporting the Quality Improvement Paradigm (QIP). The termExperience Factoryis used both for the actual organization and for the complete methodology.5 _{In the rest of this survey, the term refers to the methodology}

unless otherwise noted.

The structure of the Experience Factory is shown in figure 2.9. Experience Factory

Organization Quality Improvement Paradigm (QIP)

Goal/Question/Metrics (GQM)

Uses Supports

The Experience Factory

Figure 2.9: The Experience Factory.

Our interest is the process improvement aspects of the Experience Factory, and we focus on QIP. Because the Experience Factory is a tightly integrated approach, some explanations of the other elements will be included.

The purpose of the Experience Factory is to present an integrated approach,“combining tech-nical and managerial solutions”[13]. The business needs of the organization must be analyzed, and process and product qualities defined. The Experience Factory defines software develop-ment as evolutionary and experidevelop-mental, and emphasizes the differences from ordinary pro-duction. The paradigm relies on experiments and measurements, with an emphasis on reusing experiences. The successes and failures of the past are analyzed, and the lessons learned are packaged for use by future projects.

The Quality Improvement Paradigm (QIP)

The Quality Improvement Paradigm is a variant of the standard process improvement model (see figure 2.1). It focuses on continuous improvement and feedback into the process, based on packaging and reuse of experience. The six basic steps of the approach are [13]:

1. Characterize the current project and its environment with respect to model and metrics. 5_{The sum of the EF-organization, QIP and the Goal/Question/Metric (GQM) approach.}

(37)

2.4. PROCESS IMPROVEMENT APPROACHES 25

2. Set quantifiable goals for successful project performance and improvement. This is usu-ally done using the GQM method.

3. Choose the appropriate process model, including supporting methods and tools for the project.

4. Execute the processes, collect and validate the prescribed data, then analyze the data to provide real-time feedback.

5. Analyze the data to evaluate current practices, determine problems, record findings and make recommendation.

6. Package the experience in form of updated and refined models, and save these in an experience base.

QIP is project centered, and each improvement cycle corresponds to the execution of one project. It is also very flexible; goals, process model, method and tools are selected by each project.

The Goal/Question/Metric (GQM) Approach

The GQM approach is used by QIP to define and interpret the measures used to collect data. It defines a top-down approach, where goals are related to measures using questions indicating whether the goal has been reached. The basic structure is shown in figure 2.10.

Metric 1.1 Metric 1.2 Metric 2.1 Metric 2.2 Metric 2.3

Question 1 Question2

Goal

Figure 2.10: The Goal/Question/Metrics approach [13]. A GQM-model consists of three different elements [12]:

1. Goals:The goals characterize the purpose of the measurements. Goals areconceptual and relative to an environment.

2. Questions:The questions are used to decide whether the goals have been reached. 3. Metrics:Metrics are used to answer the questions in a quantitative way. All measures

relate directly to one or several metrics.

The GQM approach shares the flexibility of QIP, it can be used in any organization and for any set of goals.

(38)

The Experience Factory organization

An organization is needed to support QIP. This organization is built upon the principle of continuous improvement, and recognizes that improvement“requires continual accumulation of evaluated experiences (learning), in a form that can be effectively understood and modified (expe-rience models), stored in a repository of integrated expe(expe-rience models (expe(expe-rience base), that can be accessed/modified to meet the needs of the current project (reuse)”[13].

The Experience Factory is divided into two separate organizations. Theprojectis responsi-ble for the actual development process, using models and experiences from the experience database. This includes goal definitions and measurements, and corresponds to the first four steps of the QIP model. The analysis and packaging of experiences are the responsibility of theexperience factoryorganization, and this organization is independent of the ongoing projects. Relevant models and experiences are supplied to the projects from the experience base, usually in the first (characterize environment) and third (choose model) phases of QIP. This separation of concerns is emphasized by SEL, and considered one of the key character-istics of the approach [16].

Quality view

The Experience Factory is flexible enough to support all quality views. The goals are selected by each project, and“should be defined from a variety of points of view: user, customer, project man-ager, corporation, etc.” [13]. The methodology therefore emphasizes that different views of quality are important.

On the other hand, the experience base of the EF contains onlydomain specificdata. According to Basili [16], one of the key lessons of the Experience Factory (as used by NASA/SEL) is that the“goal of experimental software engineering is self-improvement; not external comparison”.Thus the Experience Factory promoteslocalquality views, there is no general, ideal view of quality. This contrasts with the other surveyed approaches.

Evaluation

The Experience Factory is adescriptiveapproach to process improvement. It provides a frame-work for process improvement, without the prescription found in most other approaches. The tasks of improvement are integrated into the actual development process, unlike CMM and ISO-9000 where much of this work is done by the quality assurance organization. While there exists a separate EF-organization, its purpose is to package and provide experience, not the actual improvement. Thus the EF integrates previously disparate activities like training, standards, technology insertion and measurements [16].

The Experience Factory is an engineering approach, and built on sound scientific principles. It considers software development alaboratory science[13], and treats it accordingly. The ex-perimental approach relies on measurements, and improvements have been measured in real organizations [15, 16]. The method is used actively within NASA/SEL, and have good inter-nal validation [149].

(39)

2.4. PROCESS IMPROVEMENT APPROACHES 27

Thus the Experience Factory is a valid approach to software process improvement. However there is a potential for problems in the approach:

External validation:The Experience Factory has only been used by the SEL. While this has resulted in goodinternalvalidation, it lacksexternalvalidation [149].

Local optimizations:The reliance on domains might introduce local optimizations. If each project uses only experience local to its domain, there is a possibility of each do-main reinventing the wheel. In addition introduction of new technology can be diffi-cult.

Overhead:The reported overhead for SEL is about 11 %. This overhead is measured in a mature organization with good knowledge of the EF, and the overhead for more im-mature organizations is not known. On the other hand this is the only hard data avail-able for process improvement, and the cost of other approaches is unknown.

Academic orientation:The approach has an academic orientation, and is documented partly by internal reports and papers. The lack of one, concrete specification makes the Experience Factory hard to implement outside SEL.

Time to impact:The most serious problem is the time necessary to build an experience database. Without any experience data, no process improvement can be made. Thus the time until impact for the Experience Factory is usually 1-2 years [16]. In addition the initial baseline is extremely important [16], as optimization of a poorly designed baseline can take considerable time.

While the Experience Factory has been a good approach to process improvement for the SEL, there is still doubts about its general applicability.

2.4.4 Business Process Reengineering

Since it was coined by Hammer in 1990 [66], Business Process Reengineering (BPR) has gained widespread following. As with all popular management approaches, this resulted in many different approaches claiming to be BPR, and no universal definition exists. This survey is based on the book by Hammer [67], but we believe that the characteristics presented here are common for most BPR approaches.

Improvement paradigm

BPR is first and foremost a management tool. It is process-centered, and concerned about improvement of business processes. Its improvement paradigm is defined by Hammer as [67]:

The fundamental rethinking and radical redesign of business processes to achieve dramati-cal improvements in critidramati-cal, contemporary measures of performance, such as cost, quality, service and speed.

(40)

The keywords here arefundamental rethinkingandradical redesign. BPR proposes that the busi-ness processes of current organizations are fundamentally flawed, and BPR is an“all or noth-ing proposition that produces dramatically impressive results”[67].

Thus BPR does not fit the improvement paradigm of figure 2.1. It proposes to throw away all old processes, and make radically new ones.6 _{The BPR approach then gives guidelines}

for what these new processes should look like. We will not go into details here, but common elements are empowerment, concurrency, horizontal compression and customer orientation. Few BPR approaches incorporate any notion of continuous improvement. In [2] an attempt is made to merge BPR and TQM, but it is hard to merge two fundamentally different paradigms. Thus what happens after a (successful) reengineering is left unanswered. It is also quite pre-scriptive, either you do it the BPR way, or you don’t.

Finally, BPR proposes anintegratedview of process improvement. Changes to the process must be followed by changes in jobs and structures. These changes are then followed by changes in management and measurement systems, which result in changes in the values and beliefs of the organization. While this view might be somewhat extreme, it does emphasize that process improvement must include more than just changes in a few proces

Process Modeling for Process Improvement - A Process Conformance Approach