Integrating Management and Engineering Processes in Software Product Development

(1)

Integrating Management and

Engineering Processes in Software

Product Development

Daniel Karlström

Department of Communication Systems

Lund Institute of Technology

(2)

Integrating Management and Engineering Processes in Software Product Development

2

ISSN 1101-3931

ISRN LUTEDX/TETS- -1069- -SE+230P Printed in Sweden

E-husets tryckeri Lund 2004

(3)

(4)

4

This thesis is submitted to Research Board FIME - Physics, Informatics, Mathematics and Electrical Engineering - at Lund Institute of Technology (LTH), Lund University, in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Engineering.

Contact Information: Daniel Karlström

Department of Communication Systems Lund University Box 118 SE-221 00 LUND Sweden Tel: +46 46 222 08 84 Fax: +46 46 14 58 23 email: [email protected] http://www.telecom.lth.se/Personal/danielk/

(5)

5

Abstract

The intangible nature of software means that traditional processes for managing product development are not sufficiently effective. This, in combination with the increasing share of software in technical products, has had the effect of making it difficult to identify improvement proposals for engineering processes in many cases. This thesis uses both qualitative and quantitative research methods to investigate methods that involve the entire development organisation in software process improvement.

The research is performed in cooperation with several companies developing software products in different countries. The studied organisations range from small development companies, using light, informal methodologies, to large corporations, with thousands of developers, using large frameworks of formal process models. The need for organisations to use both local experience as well as generally accepted best practice in process improvement is addressed.

A method that uses input from the whole organisation for strategic software process improvement is created and evaluated. The introduction of one agile methodology, Extreme Programming, is studied in a small team and a decision support method is evaluated for introducing the methodology. Also, a framework based approach to process improvement is applied to testing practices. The approach allows a small, rapidly evolving, company to introduce suitable practices that solve problems apparent in the current state of the company. This allows the processes to evolve in small steps without introducing formal practices too early in the company’s evolution.

An integration of agile methods and high-level product management processes is proposed and evaluated on both the engineering level and the product management level of organisations. The integration allows contrasting elements of the processes to remain effective within each respective domain and in some instances even perform synergetically.

(6)

(7)

7

Acknowledgements

This work was partly funded by The Swedish Agency for Innovation Systems (VINNOVA), under a grant for the Centre for Applied Software Research at Lund University (LUCAS). The research performed in Japan was funded using contributions from the Sweden-Japan Foundation, Per Westling’s memorial foundation, Hosei University Tokyo, the Ernhold Lundström scholarship, and the Saskawa foundation. Travel support was also received from the Royal Physiographic Society in Lund.

First and foremost I would like to thank my supervisor, Dr. Per Runeson, for supporting, guiding, inspiring and restraining me during the work presented in this thesis. I would also like to thank my second supervisor, Dr. Martin Höst for always being available to answer my questions.

I would like to take this opportunity to thank Prof. Claes Wohlin for convincing me to start as a PhD student in the group.

I would also like to thank my co-authors and all other people who have contributed to the research performed in the thesis.

A special thanks to all my colleagues at the department of Communication systems for creating a professional working environment. Especially I would like to thank the members of the Software Engineering Research Group, Carina Andersson, Enrico Johansson, Lena Karlsson, Johan Natt och Dag, Josef Nedstam, Bengt Ljungquist, Dr. Björn Regnell, Dr. Thomas Thelin, and former group members Dr. Magnus C. Ohlsson, Dr. Tomas Berling Dr. Håkan Petersson and Thomas Olsson for providing the opportunity for many rewarding discussions both on and off the topic of software engineering. Thanks also to the Telecom floor ball players for allowing me to clobber them every Monday evening.

Finally, I am grateful to Kerstin, my friends outside the department and my family. Thank you for putting up with me, especially during the last year while performing my studies away from home and, of course, during the last few months of finishing the thesis.

(8)

(9)

9

Introduction

Software has become a part of almost every product being developed today. The intangible nature of software means that traditional processes for managing industrial projects are not effective. This, in combination with the increasing size and complexity of software products, has had the effect that improvement strategies for development processes are in most cases hard to identify.

The field of software processes and process improvement has had to evolve constantly in order to keep up with changes in the type, size and complexity of the software being developed. In addition to this, the number of people performing the development has increased greatly as well as the variety of different types of software. An excellent example is the extent of Internet software developed in the early 1990s compared to early 2000. Currently there are two quite different approaches to process improvement being used in the industry, (1) methods based on maturity models, and (2) agile methodologies. The maturity models approach is regarded a more formal and ‘heavy’ approach and has evolved from the Capability Maturity Model (CMM) from the beginning of the 90’s. The agile approaches have evolved partly as a reaction to the first and are considered more lightweight. Currently the agile approach is thought less compatible with large-scale software product development, often performed in strict, staged, management models. The research performed in this thesis indicates that there may indeed be several advantages both on the engineering and management levels of the organisation by using agile concepts instead of traditional heavyweight concepts.

The research presented in this thesis is performed in both small development organisations, using light, less formal methodologies, known as agile methodologies, and large organisations, with thousands of developers, using large, more formal development process frameworks.

(14)

14

A method that uses input from the whole organisation for strategic software process improvement is created and evaluated. This method allows the process owners to base software process improvement on input from the entire development organisation. This is thought to anchor the decision more firmly in the organisation.

The introduction of one of the main agile approaches, Extreme Programming, is studied in a case study of a small development project in a real world context as well as in a large stage gate based product development context in two other cases.

A decision support method is evaluated for identifying practices within Extreme Programming that might be perceived to be more or less effective in the developer group before the introduction. Hence the introduction of the practices can be tailored based on the results of the analysis.

A framework approach to process improvement for testing is investigated that allows the company to introduce suitable practices that solve problems apparent in the current state of the company. This allows the test processes to evolve in small steps without introducing too many formal practices too early in the company’s evolution.

The research presented in this thesis attempts to reduce inherent resistance to process changes in organisations by increasing the involvement of the whole organisation in the improvement work and thereby increasing acceptance for process improvement.

The seven research papers in the thesis can be classified into research performed in a context of a small or large company and into research in the use of traditional or agile methods. This classification is shown in Table 1.1. The papers could be read in any order, but have been numbered chronologically as the studies have been performed. The papers are provided exactly as they have been published, only the layout has been altered to suit the format of this thesis. This has the advantage of making each paper more understandable as a unit but has the disadvantage of some repetition of material in the introduction and background .parts of the papers.

Large context Small context Traditional Paper 1 Paper 4

Agile Paper 5

Paper 6 Paper 7

Paper 2 Paper 3

(15)

15

The first part of this thesis is an introductory part that summarises and packages the work performed. It is divided into the following sections: 1. Background – This section briefly explains the background of software system engineering and software engineering processes, the structure of the discipline, the terminology involved and the challenges present today.

2. Research Methodology – This section discusses research methodology used in software engineering research.

Parts of this section are based on the following publication.

Karlström, D., “Effects of Introducing Agile Methodologies in Software System Engineering”, 3rd National Conference on Software Engineering Research and Practice in Sweden, SERPS'03, Lund, Sweden, October 2003.

3. Research Presentation – This section presents the research performed in this thesis, the strategies used, the main contributions made and an outline for further research in the field.

Parts of this section are based on the following publication.

Karlström, D. Runeson, P., “Combining Stage-Gate Product Development with Agile Software Development: Observations from three industrial cases”, Submitted the special issue of IEEE Software on software engineering project management, May/June, 2005.

The second part of this thesis contains the seven research papers that constitute the main contribution of the thesis:

Paper 1 - Aggregating Viewpoints for Strategic Software Process Improvement – A Method and a Case Study

Karlström, D, Runeson, P. , Wohlin, C.,

Published in IEE Proceedings Software, October 2002.

Paper 2 - Introducing Extreme Programming - An Experience Report

Karlström, D.,

Published in proceedings of the third International Conference on Extreme Programming and Agile Processes in Software Engineering, 2002.

Paper 3 - Decision Support for Extreme Programming Introduction and Practice Selection

Karlström, D., Runeson, P.,

Published in proceedings of the fourteenth International Conference on Software Engineering and Knowledge Engineering, 2002.

(16)

16

Paper 4 - Incremental Introduction of a Structured Test Practice Framework in Emerging Software Development Organisations

Karlström, D., Runeson, P., Nordén, S.,

Accepted for publication in the Journal of Software Testing Verification and Reliability, preliminary publication issue: June 2005.

Paper 5 - Agile Methods and Software Product Management – An Integrative and Hybrid Approach

Runeson, P., Karlström, D.,

Technical report, Lund Institute of Technology, CODEN : LUTEDX (TETS-7203) / 1-34 / (2004) & local 16, 2004.

Paper 6 - Integrating Agile Software Development into Stage-Gate Managed Product Development

Karlström, D., Runeson, P.,

Submitted to the Journal of Empirical Software Engineering, 2004.

Paper 7 - Exploring Agile Influences in Stage-Gate Management of Software Product Development

Karlström, D.,

Submitted to IEEE Transactions on Engineering Management, 2004.

1 Background

The background to this thesis starts with an introduction to software engineering (SwE), system engineering (SE) and software system engineering (SwSe). The area of the thesis is mapped into the software engineering body of knowledge (SWEBOK) [1] followed by an introduction to SwE processes, software process improvement (SPI). Process maturity models are described in a separate section as are Agile Methodologies and project management models as these areas are central to the thesis topic.

1.1 Software Engineering and Software System

Engineering

Software system engineering (SwSE) is considered the application of system engineering (SE) on software development [2]. SE is the practical application of scientific, engineering and management skills required to move from a need to a technical realisation of that need. At the SE level the product is generic, i.e. the product does not necessarily have to include software at all, can be part software or a complete software

(17)

17

product. A standard for SE is provided in IEEE Std. 1220-1998 [3]. Here, five aspects of SE are identified: Problem definition, Solution

analysis, Process planning, Process control, and Product evaluation.

Relationships between SE, SwSE and software engineering are well described by Thayer [2] and are illustrated in Figure 1.1. Here, a sequential V-type model [4] relationship is implied for the activities on all levels. However, there are activities on all levels throughout the course of the product development. System safety is often a very important factor in large systems. The software part of the system is often the hardest part in which to ensure safety. Other aspects of system engineering such as logistic engineering, integrated logistics support (ILS) and logistics support analysis (LSA), although they may be remotely affected by a software development methodology change, fall outside the scope of this thesis. The Software Engineering Body of Knowledge (SWEBOK) [1] lists systems engineering as a related discipline to software engineering.

Figure 1.1 Engineering relationships between system engineering, software system engineering and software engineering [2]

1.1.1 The Structure of the Software Engineering Discipline

Software Engineering is a relatively new discipline and it is only recently that structure and terminology has become well established. The collaborative effort creating the Software Engineering Body of Knowledge (SWEBOK) [1] has begun the important task of creating an introduction

System analysis System design Software requirements analysis Architectural software design Detailed software design Code and unit test System testing System integration testing Software system testing Software integration testing Software subsystem testing System engineering Software system engineering Software engineering

(18)

18

to the body of knowledge. This is, however, a difficult task as the discipline is constantly changing. The overview provided by this initiative is provided in Figure 1.6.

SWEBOK divides the discipline into ten knowledge areas,

requirements, design, construction, testing, maintenance, configuration

management, engineering management, process, tools and methods, and

quality. The knowledge areas represent areas where research is being

performed today and have been established at different times during the evolution of software engineering processes up to the present. The first five knowledge areas are ordered according to the steps in the traditional waterfall development process [6]. The knowledge areas themselves, however, are not on the same level of abstraction as each other and many are very closely related. Some sub-areas are even part of several other areas. For instance, the software process knowledge area is a meta-area compared to the first five knowledge areas. All the areas however play a significant role in the development of software today.

The research presented in this thesis can be categorised into the

software engineering process knowledge area of the SWEBOK [1], but parts

of the software engineering management area are also closely related to the work. An overview of the guide to the SWEBOK is shown in Figure 1.2 and relevant areas are highlighted. The process knowledge area is subdivided into the following sub-areas:

• Process implementation and change

This sub-area focuses on organisational change. It describes the infrastructure, activities, models and practical considerations for process implementation and change.

• Process definition

This sub-area focuses on how processes are defined. This can be as procedures, a policy or a standard.

• Process assessment

This sub-area focuses on how process assessment or appraisal is carried out.

• Process and product measurement

This sub-area focuses on how measurement is performed both to evaluate the need for process change and the effects of process changes made.

(19)

F igu re 1.2 Th e Softwar e Engine er ing B od y of Knowl ed ge Ov er vi ew [1] (H ighlig ht s in dicat e t he areas m os t re le va n t f or t his t he sis ) Sof tw ar e Re quire m ents Sof tw ar e De si gn Sof tw ar e Con str u cti on Sof tw ar e Te st ing Sof tw ar e Maint enan ce Sof tw are Co nf ig u rat io n Ma n ageme n t Sof tw are En gin eer in g Ma n ageme n t Sof tw ar e E n gi n eer in g Pr oc es s So ft war e En gin eerin g T ools and M eth ods Re la te d D isci p li n es Sof tw ar e Qual it y So ft war e R equirem en ts fu ndame n tals R equirem en ts P roce ss R equirem en ts E licit at io n R equirem en ts Analysi s R equirem en ts Sp ecif ica tio n R equirem en ts V ali dati o n Pr ac ti ca l Co n sid era tio n s So ft war e Desi gn F u ndame n ta ls Key Issu es in So ft war e Desig n So ft w are S truc ture an d A rc h it ec ture So ft war e Desi gn Q u ali ty analy si s and Ev alu ati on So ft war e Desi gn Notati ons So ft war e Desi gn Str ate gi es and Met h od s So ft war e Co n st ruc ti on F u ndame n ta ls Ma n ag in g Co n st ruc ti on Pr ac ti ca l Co n sid era tio n s So ft war e Test in g F u ndame n ta ls Test L ev el s Test Techniqu es Test Rel ate d Mea sures Te st P roce ss So ft war e M ai n te nance F u ndame n ta ls Key Issu es in So ft war e M ai n te nance M ai n te nance P roce ss Techniqu es f or M ai n te nance Ma n ag em en t o f th e SC M Pr oc es s So ft war e Co n fig ura ti on Id en ti fi ca ti on So ft war e Co n fig ura ti on Co n tro l So ft war e Co n fig ura ti on St at us A cc oun ti n g So ft war e Co n fig ura ti on Au d iti ng So ft w are R el ea se M anage me nt an d D el ivery In it ia ti on a n d Sc ope D ef ini ti on Sof tw ar e Pr oj ect Plan ni ng Sof tw ar e Pr oj ect Enactme n t Re vi ew and Ev alu ati on Cl os ure Sof tw ar e Engi ne er in g Mea surem en t P roce ss Imple me n tati on and C h ange Pr oc ess D ef ini ti on P roce ss Asse ssme n t Pr oc es s and Pr odu ct M anage me nt So ft w are T oo ls So ft war e Engi ne er in g Met h od s So ft w are Qua li ty F u ndame n ta ls So ft w are Qua li ty M anage me nt an d P roce sse s Pr ac ti ca l Co n sid era tio n s C ompu te r Engi ne er in g C ompu te r Sc ie nc e M anage me nt Ma th em at ic s Pr oj ect M anage me nt Qua lit y M anage me nt So ft war e Er gonomi cs Sy ste m s Engi ne er in g Guide t o t h e Soft wa re En gi n ee rin g Bo dy o f Kn owle dge (2004 Vers ion)

(20)

20

1.2 Software

Engineering Processes

As software engineering processes have developed, the terminology has evolved successively. Sommerville defines the software process as: “The software process is the set of activities and associated results which produce a software product” [5]. We can model this basic concept by using an elementary process model adapted to software development which gives us a basic software process. This is illustrated in Figure 1.3.

Figure 1.3 An illustration of the software process [5]

Any software process needs to at least address the following activities in some form in order to develop software [5]:

• Specification. Decide what is to be developed.

• Development. Develop the product.

• Validation. Ensure the product meets the specification.

• Evolution. Handle changes in the product.

In today’s methods, the execution of the software process is done in many different ways. An example of software development that does not address these activities is the methods that sparked the software crisis, also known as ‘code and fix’ methods. The lack of planning and requirements in these methods, made the following problems common:

• Poor structure.

After each fix the structure of the code is destroyed making subsequent fixes and additions more and more difficult.

• Inaccurate results.

The resulting program is hardly ever what the customer desired in the first place due to the lack of requirements.

Software process

Product idea Software Product

(21)

21

• Expensive.

Because of poor structure and lack of planning all modifications and fixes become very expensive.

When these problems were observed, new models that addressed planning, requirements, test and modification were developed.

Sommerville identifies four different types of such software development models currently in professional use [5]:

• Waterfall type [6].

Each activity - specification, development, validation and evolution - is executed and signed off sequentially, one by one.

• Evolutionary type [5].

The activities are interleaved to rapidly produce prototypes of increasing complexity and correctness with regards to customer requirements.

• Formal transformation [7].

The system is specified as a mathematical system and is then transformed into the finished product using formal mathematical transformations.

• Component based [8].

The software system is assembled from pre-developed parts.

Of these, the waterfall and evolutionary types are most widely used in industry today, but the potential effectiveness of software reuse has caused much interest in component based software engineering. In the classification above, agile methodologies [9] are classified as evolutionary by Sommerville. These are, however, commonly regarded as a separate type. These methodologies are very new and there is a need for more research into the area. The agile methodologies are discussed in section 1.5.

One of the current problems within software engineering is the use of many different terms for similar items within different areas of the discipline. This has arisen over time as certain terms have been associated with certain methodologies or have become negatively charged due to their usage in different circumstances. New software processes have changed the terminology in order not to be associated with old values or just in plain ignorance of existing terminology. The terms that are used within this thesis are shown in Table 1.2

(22)

22

1.2.1 Software Engineering Process Evolution

Software engineering processes can be divided into three distinct periods chronologically and we are currently moving into a fourth period [10]. The periods are illustrated in Figure 1.4 and a brief summary is provided below

The first period can be referred to as ‘code and fix’ [11]. Much programming was performed using low-level programming languages, program size was relatively small and the software environment was less complicated with fewer external interactions [12]. This implied that many of today’s problems were not apparent during this period, although surprisingly many current problems were mentioned by Brooks in his classic book [13] in 1975. Performing software development was relatively simple compared to today’s elaborate systems as it meant that bugs could be fixed. Systems could actually be developed that were completely fault free.

Term Definition

Process A sequence of steps performed for a given purpose [14].

Model A model is a simplified representation of a system or phenomenon with any hypotheses required to describe the system or explain the phenomenon, often mathematically. It is an abstraction of reality emphasizing those aspects that are of interest to someone [15].

Software process The set of activities and associated results which produce a software product [5].

Practice A specific activity within a process or method.

Practice framework A collection of practices creating a form of a software process.

Method A reasonably complete set of rules and criteria that establish a precise and repeatable way of performing a task and arriving at a desired result [16].

Methodology A collection of methods, procedures and standards that defines an integrated synthesis of engineering approaches to the development of a product [16].

Improvement method Defines the strategy and the process by which improvement is pursued. It specifies how to assess a process, to initiate corrective actions, to evaluate them, and to further promote new improvement initiatives. Maturity model Maturity model defines the ideal profile of a process, i.e., the

(minimum) set of requirements that a process should satisfy to qualify for a specific status.

(23)

23

The second period can be referred to as the structured methods period [11]. This period started after the software crisis in the late 1960’s when new hardware advances enabled software systems to be developed that were too large for the traditional development methods used at the time [5]. Elementary software development models were developed and used in industry. A major characteristic of this period is that the software engineering process was being described graphically. Examples thereof are waterfall and spiral processes [6, 17]. A large portion of practicing software development companies today are still using gate processes similar to these early processes although they are tackling functionality several orders of magnitude larger than in the end of the 70s. This evolution of the waterfall method is commonly referred to as gate or milestone models [18, 19, 5, 20].

Figure 1.4 Periods in software engineering process evolution

The third period, process maturity, introduced a meta-level to the development processes. During this period, software process improvement (SPI) processes and models were developed and implemented [10]. The essence of SPI is to use an improvement process, functioning on a meta-level with respect to the traditional software development process. The current development process is assessed compared to a reference process model or framework and then adjusted accordingly. The most well known example of work originating from this period is the Capability Maturity Model (CMM) from the Software Engineering Institute (SEI), Carnegie Mellon University [21]. Another example is ISO 9000 and 9001 [22, 23, 24, 25]. Most of the third period models are appropriate for larger companies though the research community has made several attempts at adapting them to smaller organisation types [26, 27, 28].

The fourth period was predicted to be the industrialised software engineering phase during the third period [29]. The engineering methods used would resemble the production methods present in other more

~1970 ~1990 ~2000 Period 2: Structured methods Period 3: Process Maturity High Maturity Agile Method-ologies

?

Period 1:

(24)

24

established engineering disciplines such as civil engineering or manufacturing. This fourth period, however, is currently only beginning to take form. There has been a distinct change in the software process community with the introduction of the processes known as the agile methodologies that appear to be in contrast to the continued evolution of the maturity models towards working with higher maturity levels. Both of these have evolved from software engineering using third period methodologies but with different approaches to the problems discovered.

Table 1.3 compares an example of inherent properties of software compared to traditionally industrially manufactured items. The properties indicate some of the problems that were thought to be solvable with fourth period methods. Due to the different nature of software, however, the industrialisation may never be possible.

Software Traditional Industry

1 Build once Build all the time

2 Reproduction low cost Reproduction high cost

3 Very flexible Inflexible

4 Intangible Physically present

5 Customer often undecided Customer determined

6 High maintenance cost Rel. low maintenance cost

Table 1.3 Comparisons between inherent properties of software and traditional industrial artefacts [30]

The creation of software can be compared to preparing for industrial production of a product. Here the community has identified some of the most useful items to apply in the field of SE, such as Total Quality Management (TQM) and Plan-Do-Check-Act (PDCA) [31, 32].

The fourth period has started with a division of the SE community. The successful practitioners of the third period models have continued their improvement into high maturity practices. The companies that failed in their intent to introduce third period processes, or never even attempted this, have taken to agile methods as the natural next step. In this second group, there is a strong reaction towards unnecessary formalism and overhead in the third period processes.

The development over the next few years will show the approaches converging and being adapted to different circumstances in different environment and it will become more apparent that both groups are striving towards a common goal of producing quality software as efficiently as possible, with as much control as possible, and with content people within the organisations. To do this, influences will be needed from within the entire scope of the practice, and theory, of SE today and

(25)

25

that each organisation producing a different type of software will require a highly adapted methodology for their software type, organisation type and individual people types involved in development. This can be identified in Brooks’ early paper [33] from which we can quote the classic “There is no silver bullet for software development”.

1.3 Software Process Improvement

Software process improvement (SPI) involves working with the existing software process in a company and improving it. The area can be divided into the process for performing the improvement work and the reference needed for comparison [34].

1.3.1 Software Process Improvement Methods

A generic SPI method can be simplified to three steps: start, assess and

change. The SPI initiative starts by the insight that the current

development process can and needs to be improved. In this first step,

start, sponsorship from senior management is ensured. This is in order to

make sure that support and resources are available and to spread knowledge and acceptance about the effort. During this initial step a general plan is created and the goals for the initiative are also drawn up.

Assessment of the software process in the next step, assess, concerns making an evaluation of the current process and comparing it to a reference of some kind. This can be performed both internally by the company, externally by assessors or a combination of both.

The actual change is performed in the final step, change. Typical methods for introducing change include training and introduction of new process documentation.

Typical examples of process improvement methods are SEI’s IDEAL method [35], specialized for self-assessment and capability evaluation, CMM based appraisal for internal process improvement (CBA-IPI) [36], Software Capability Evaluation (SCE) [37], and the forthcoming SPICE/ISO15504 standard [38].

The IDEAL method was developed by SEI as a way of using the CMM based appraisal method for process improvement. As the method does not refer to a specific model for comparison, this model could be used in combination with other models than the CMM. The IDEAL method is illustrated in Figure 1.5. Note the similar structure to the basic phases described earlier in this section. Here the initialising phase of the

(26)

26

IDEAL model corresponds to the start phase, diagnosing corresponds to

the assess phase and establishing, acting and leveraging correspond to the

change phase.

Figure 1.5 SEI IDEAL process improvement model [35]

A process improvement should be relevant, i.e. actually improve the process, as well as applicable, i.e. possible to introduce both technically and socially in the target context.

In order to be effective a process improvement should therefore contain both improvements based on the local situation as well as general improvements. This dual nature of process improvement is not reflected clearly in the SPI methods referenced. It is referred to as tailoring by some but no real guidelines are provided.

Improvements based on local situation have the positive characteristics that they are easily accepted in the organization and are also effective in that organization. They have the negative characteristics that they are most probable not found in general SPI frameworks and so can be difficult to identify. Secondly they have a low degree of transferability. A generic improvement process based on the local situation in a company is shown in Figure 1.6. sponsorship improvement Stimulus for and establish Set context infrastructure improvement Establish strategy Set phase results and document recommendations Develop characterize Appraise and lessons Plan actions action teams practice current priorities and Plan and and measures processes Define process Establish and analyse Document installation and track execute Plan, pilots execute Acting Leveraging Initialising approach Establishing Diagnosing organisational Revise

(27)

27

Improvements based on general frameworks have positive characteristics that their need may not be apparent in the organisation despite their relevance. Secondly they have a high degree of transferability. They have the negative characteristics that they can be perceived to be forced onto the practitioners in the organisation and may not be as effective. A generic improvement process based using a general framework is shown in Figure 1.7.

Figure 1.6 Improvement based on local situation

Figure 1.7 Improvement based on general framework

The most effective SPI process therefore would attempt to use improvements based on both the local situation and general frameworks. This hybrid process is illustrated in Figure 1.8.

Actual

behaviour Observe Improve

Decide What & How

Actual

behaviour Compare Improve

Decide How

(28)

28

Figure 1.8 Improvement based on both local situation and general framework

A further issue here is identifying general frameworks that apply to different context within SE. For example frameworks for small vs. large companies, product development vs. contract based development and so on. This creates an in-between to completely general frameworks such as the CMM [16] and completely local based improvement methods such as TQM [31, 39, 40].

Looking at SPI and SPI models on another level we can examine the origin of the reference framework and improvement method and how these were established. Most frameworks are established as collections of best practices based on the aggregated experience of a varying number of individuals or developing organisations. This ranges from frameworks without any source traceability what so ever to the CMM [16], which can be classified as one of the largest collections of software practices performed.

The generic SPI processes are composed of three types of steps: The first is either observe or measure, the second decide and the third do. Each step can potentially be improved using a different strategy from SwE research methodology, presented in Section 2, due to that the goal of the work involved are similar to that of the research performed:

• Observe – Qualitative research validity theory (see Section 2.2).

• Measure – Quantitative research validity theory.(see Section 2.1)

• Decide – using Internal & external experts or analysis, different

sources of experience and decision support methods.

• Do – improve anchorage, motivation and support for process changes. Actual behaviour Observe Improve Decide

What & How

Compare Decide

How

(29)

29

In addition, the analogy can be extended to flexible research being similar to non-framework based SPI and fixed research being similar to framework based SPI. The synergy effects of integrating fixed and flexible research approaches can therefore be analogous to integrating the different types of SPI methodologies. We can use this analogy to improve both the application of software process improvement in the target context as well as to improve creation of software process frameworks.

One of the most difficult steps is the decision step. Based on observations made, and potentially comparisons to a framework, a decision is taken based on the decision takers’ knowledge, experience and beliefs. The person or group making the decision as well as the apparent basis he takes the decision on affects this decision significantly. This aspects is often not included in SPI methodologies

A general process framework is created through a large number of observations and/or experiences generalised to a framework. The generalisation step involves deciding how to incorporate differences between cases, which can be difficult. The process, which is parallel in nature, is illustrated in Figure 1.9

Once created, a general process framework needs to evolve to keep up with industry practice. This process is more serial in nature and is illustrated in Figure 1.10.

Figure 1.9 Genesis of a general framework

Actual behaviour Observe Generalise Actual behaviour Actual behaviour General Framework Observe Generalise Observe Generalise D iffe re n t C on te xt s ( sim ila r a n d d iffe re n t i n n at ure )

(30)

30

Figure 1.10 Evolution of a general framework

Two cases, CMM [16] and agile methodologies [9], are discussed here to demonstrate the evolution of two different software process models that are of significance in the third and fourth periods respectively. It is interesting to note the opposite direction of introduction in the two cases. In the case of agile methodologies the source of the methodology is industry and the introduction is bottom up from developer to manager whereas in the CMM case the source of the method is academia in cooperation with the US Department Of Defence (DoD).

The cases illustrate different roles that academia has taken during the evolution of two different software processes. The current trends in software processes must affect not only what the academia is researching, but also how the research is performed. For instance, in the CMM case the intent of each part of the process is clear, but the actual industrial effect is unclear and needs to be investigated. This was done for the lower maturity levels in the CMM as many companies reached the lower levels relatively quickly. The effects of the higher levels have not really been seen until the latter part of the 90’s and the content thereof can be seen as largely speculative based on what was though to be needed once the initial practices at the lower levels were in place. This has resulted in updates due to experience gathering at for example high maturity workshops [41].

In the second case however the intent of the practices are not as clear to the academic community as the formulation of the practices have not involved the academic community. As in the CMM case the actual effects of the practices need to be studied, but the fact that the models from an academic point of view seems to contain several ‘dangerous’ activities and excludes several activities proven to be effective and are still used needs to be investigated.

Framework Compare Improve

Decide How

Actual behaviour Framework

Compare Decide Improve

How

Actual behaviour Framework

Compare Decide Improve

How

Actual behaviour

(31)

31 1.3.2 Challenges in SPI

Humphrey describes six basic principles that need to be followed during the course of process improvement. These provide an overview of some basic issues involved in software process change [42]:

1. Major changes to the software process must start at the top.

Senior management leadership is required to launch the change effort and to provide continuing resources and priority.

2. Ultimately everyone must be involved.

Software engineering is a team effort and everyone who does not participate in the improvement will miss the benefits and may even inhibit progress.

3. Effective change requires a goal and knowledge of the current process.

To use a map you have to know where you are.

4. Change is continuous.

SPI is not a one shot effort; it involves continuous learning and growth.

5. Software process changes will not be retained without conscious effort and

periodic reinforcement.

Effort must be taken not only to introduce changes, but also to retain them.

6. SPI requires investment.

SPI takes planning, dedicated people, management time and capital investment.

One of the goals of the improvement effort is to improve the relationship between actual required resources compared to the estimation of resources. The process can be affected in two ways:

• Predictability

Moving the estimation of the required resources towards the average of the actual required resources.

• Control

Decreasing the variance of the actual required resources.

The official process described in a company exists in different forms depending on which phase of the process improvement we investigate it. A presentation of the different kinds of processes that can be present in an organisation is given in Table 1.4 [43].

(32)

32

Process Name Description

Reference The sources from which the process

ideas originate (Theoretical papers, books, etc).

Desired The process that the person

interpreting the references wants to introduce.

Official The process described on paper by the organisation.

Perceived The process that the performing

subjects believe they should perform

Actual What is actually performed.

Observed The observed process.

Table 1.4 Different types of the same process [43]

Each process type is related to the next by a mechanism of transfer. For example the perceived process is obtained from the official process through teaching, peer information and self-study. The mechanisms are summarised in Table 15.

It can be seen that three of the processes shown in the table are transferred by interpretation. Interpretation is highly subjective, i.e. highly dependant on the person involved. A person’s subjective interpretation of the process is known as that person’s viewpoint [44].

From To Mechanism Description

Reference Desired Interpretation Reference processes are interpreted by the person(s) creating the official process. Desired Official Formulating The official process is formulated from the

desired process and written down as the official process.

Official Perceived Interpretation The user interprets the official process based on what he is taught, what he reads and how his co-workers influence him. Perceived Actual Performing What is actually performed is a result of

the perceived new process and the old process.

Actual Observed Interpretation The observed process is a result of an interpretation of the actual process.

(33)

33

The actual process is unique in the sense that this is the process that

actually creates the product. In many ways this makes it the most important process, as this is the process that we must ultimately improve. It is observed through the interpretations of observers but cannot be objectively evaluated by observation alone. The use of some form of metrics, however, offers an objective unbiased picture of the results of the actual process [45]. This indicates the importance of metrics in SPI work.

SPI involves changing existing practices, processes and structures. The study of the phenomena observed when introducing changes to a group of people belongs more to the social sciences than engineering science. It is however very important to remember that there are humans involved in the development process and any action made needs to take this fact into account. All changes must therefore be practised, improved and will then eventually become a natural part of the company’s practices after the introduction. A typical response to change and the effort exerted by the subject is illustrated in Figure 1.11.

It must also be remembered that the current process being used at a company is the result of many years of experience, and most of it is usually good practice.

Figure 1.11 Typical human response to change [10]

1.4 Process Maturity Models

There are several process maturity models used in practice, both commercial and non-commercial. Without any doubt the most used is the CMM model [16]. The framework quagmire [46] shows an overview of many of the alternatives that exist. How these relate to each other also gives an idea of how they have evolved. Here the term framework is used as a collective terms for process standards, quality standards, maturity or capability models, appraisal methods and guidelines so as to compare them directly. The choice of framework is, however, often much more limited due to pre-set requirements from, for example, contractors, management, the current process and the improvement goal itself. The

Time Stunned paralysis Anger, Rage Denial Bargaining Testing Depression Effort Acceptance Status Quo

(34)

34

most influential frameworks in the quagmire are the CMM-based models [16], ISO 9000 based standards [22], and various military standards. These frameworks are also the most widely used today and many companies are assessed in several standards.

Process improvement models are not only different in their composition and content. They also have different purposes and uses. Some models are intended for use in one special project because the customer demands it, while others are intended to be used consistently across a whole organisation, such as the CMM.

The Software-CMM (SW-CMM) [16] contains a framework of practises that define different levels of maturity in the software development process. The model is composed of five levels of increasing maturity. Each level contains practises that would normally be expected in a software process at the corresponding maturity level. Organisations that do not implement CMM are said to be at level 1, the initial stage.

Each level, except for level 1, is divided into several key process areas (KPA), which describe the areas that the level addresses. For an organisation to be performing at a certain CMM level, it must successfully implement the practices described at that level and at all lower levels. The CMM can be used in five different ways [16]:

1. CMM based assessment to identify process strengths and weaknesses.

2. Evaluating subcontractors for selection.

3. Developing new appraisal methods based on the CMM. 4. Teaching upper management about SPI introduction.

5. As a guide for technical staff and process improvement groups in their work to define and improve their processes.

Since its introduction the SW-CMM [16] has evolved in to the Capability Maturity Model Integrated (CMMI) [47], by being updated and integrated with the system engineering CMM (SE-CMM) [48]. This model is available both in a staged version such as in the original SW-CMM, or a continuous version from the SE-CMM.

When using the CMM models it should be remembered that they were created at the initiative of the Department of Defence and that their way of acquiring software is usually to select one of several contractors to complete the project at a fixed price. The projects involved are usually quite long, not market driven or of a product type and have relatively stable requirements. The software industry has changed radically during the 90s with the popularisation of the Internet and exponential growth of

(35)

35

amount of software being developed.

1.5 Agile

Methodologies

Agile methodologies have arisen as a reaction to the more strict processes employed during the third period of software engineering processes The development of these occurred in parallel at the end of the 90’s. The most widespread of these methodologies are:

• Extreme Programming (XP) [49, 50]

XP was created by Ward Cunningham and Kent Beck and involved 12 practices that individually are not new to the SE community. The methodology became widespread around 2000 and a new updated version is currently being released.

• Cockburn's Crystal Family [51]

Crystal was developed by Alistair Cockburn in the early 1990s with the primary focus of improving communication in software projects. It was intended to include three versions for different sizes of projects, but only the smallest version has been completed so far.

• Adaptive Software Development (ASD) [52]

ASD was created by Jim Highsmith. The method focuses more on the philosophy behind the development approach instead of individual practices. Currently considered complementary to Crystal, where the details in Crystal complement the philosophy in ASD.

• Scrum [53]

Scrum was created by Ken Schwaber. The method is considered a meta-process or ‘wrapper’, independent of the actual development methodology used. It is, for instance, quite common to use Scrum and XP in combination.

• Feature Driven Development (FDD) [54]

FDD was developed by Jeff De Luca and Peter Coad. FDD development is a process designed to produce frequent, tangible, results and is a collection of best practices.

• Dynamic System Development Method (DSDM) [55]

DSDM is a framework focused on delivering a quality solution quickly. The method focuses specifically on

(36)

36

incremental development, user involvement and most important first.

1.5.1 The Agile Manifesto

The Agile methodologies have many common aspects and in 2001 a core group of people from the agile community formulated the agile manifesto [9] describing the most fundamental aspects of agile development. The principles of the agile manifesto are shown in Table 1.4. These principles indicate the difference in focus of agile methodologies compared to the maturity models. Agile methodologies focus on [9]:

• Individuals and interactions over processes and tools.

• Working software over comprehensive documentation.

• Customer collaboration over contract negotiation.

• Responding to change over following a plan.

This means that there is still value placed in the practices on the right, but those on the left are valued more. A common misconception of the agile methodologies is that the practices to the right of the list above are strictly forbidden. This is not the case. All of the methodologies described in the previous subsection share these values and the principles behind them. Indeed, it is possible to use only the principles to influence the current practice in an organisation without actually using any of the methodologies mentioned.

The agile methodologies recognise the importance of making the path from the person(s) that are familiar with what functionality is desired (i.e. the customer) to the developers that are going to produce the product as short as possible in order to make communication as quick and effective as possible. As the agile methodologies propose a different perspective on scheduling development, based on fixing time and resources and varying functionality, changes to requirements late in the development cycle has no effect on the schedule. This different focus of scheduling is illustrated in Figure 1.12. By maximising work not to be done the team can focus on the most important functionality without being distracted by less important features.

(37)

37

Principles of the Agile Manifesto:

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

4. Business people and developers must work together daily throughout the project.

5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

7. Working software is the primary measure of progress.

8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

9. Continuous attention to technical excellence and good design enhances agility. 10. Simplicity, the art of maximizing the amount of work not done, is essential. 11. The best architectures, requirements, and designs emerge from self-organizing

teams.

12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.

Table 1.4 Principles of the Agile Manifesto [9]

By focusing on delivering an actual product in working software as quickly as possible and consequently delivering increments often, a concrete measure of progress is achieved compared to traditional methodologies. Agile methodologies also recognise the need for good communication both within the development team and with management. This is best achieved through frequent face-to-face communication, a much richer form of communication compared to documents. Through communication trust is established between team members as well as between the team and management. Agile methodologies promote self-organisation and self improvement in the development teams. This makes sure that improvement is performed at the level where the product is created and not imposed from above.

The development team is expected to work at a pace which they can sustain indefinitely. It is considered that overtime or death march projects only lead to lower quality products, increasing the lead time and cost in the long run. As the software product is a technical product it is essential to promote technical excellence and good design.

(38)

38

Figure 1.12 Requirement flex in Agile Development [55]

These agile approaches have only been in widespread use since around 2000 as a result of the amount of media attention received by the extreme programming methodology and we have only recently begun seeing experience reports on the usage of such methods [56, 57]. A brief introduction to Extreme Programming is provided in the introduction to Paper 3 in this thesis and a complete case study on introducing Extreme Programming is provided in Paper 2.

1.5.2 Effects on the Development of Software Systems

The effects on SwSE and SwE of introducing an agile approach in the software development will, of course depend on many factors. For instance whether the change is introduced in a small group within the software developers’ organisation or if all the software developers in a large project are to use a more agile approach. Another factor will be the extent of software in the product being developed. If the product is largely hardware with an embedded software part, the software development must be strongly coordinated with the hardware development. The fixed nature of hardware makes hardware development a less volatile process.

If we examine the four aspects of the agile manifesto we can discuss how each of these in turn is thought to affect the development of the software system. The issues discussed will affect the software system development in different ways in specific instances, but the most general principles will most probably be similar.

Functionality Time Functionality Resources Time Resources Fixed Variable Traditional Agile

(39)

39

Individuals and interactions over processes and tools

The shift of focus towards interactions between humans must ensure that communication previously handled by the process and its artefacts must still be in place in order to successfully develop the product. Instead of putting that responsibility on the process, the responsibility is given to the people working in the project. Empowering them to optimise the way they work together. Tools will be necessary to complete certain tasks and interactions although it is the completion of the tasks and results of interactions per se that are truly important. An example of easing interactions between people can be to make sure that a working version of the product is available to discuss over.

Working software over comprehensive documentation

The shift from a document oriented development strategy to a software-oriented strategy may sound dangerous to many system development organisations. Documentation is required to synchronise development between development units, both software and non-software, for future maintenance of the system. Often documentation can be for the sake of complying to quality standards themselves. The shift implies reducing unnecessary documentation. Documents that are not updated as the product evolves or document the code line by line are examples of candidates for unnecessary documentation, but documents can still be essential when fit for their purpose

Customer collaboration over contract negotiation

The shift towards communicating with a customer stakeholder is part of increasing the feedback rate within the project. Instead of the project being one big feedback loop, with the goal set at the beginning and feedback provided by the stakeholder at delivery, the project becomes a series of rapid feedback steps with feedback provided as often as is effective. The SwSE environment implies many complex customer relationships, e.g. software groups may be customers to hardware groups and visa versa, and it may be important to have customer groups and customer group coordination activities as well as an extended project community.

Responding to change over following a plan

The shift towards responding to change over following a plan seems contradictory to the ordered flow shown in Figure 1.1 and it is a large challenge to introduce this focus shift in such a context. However, if all

(40)

40

units producing the product are tightly coordinated with short feedback loops and proceeding at optimal pace, then the plan becomes less important than ensuring that the project continues to proceed optimally. This is not saying that a software system should be developed without a plan, plans are important to further coordinate different units within the product.

1.6 Project Management Models

A project management model (PM) defines a framework for setting up and managing a project [18]. The framework as such is general, applicable to any kind of project. The PM focuses on monitoring, management and decision issues on a business level, rather than the specific technical work, and typically defines:

• A set of phases.

• Criteria for decisions related to the phases.

• Stakeholder roles.

• Project management artefacts.

In Figure 1.13, the principal constituents of a generic PM are outlined. In this model, there are four phases; prestudy, feasibility study, execution and completion phases. This is the terminology used in the PROPS model by Ericsson [19] but most PM models used have phases with similar intent. Project progress is measured through the passing of predefined milestones (MS) and at the end of each phase a tollgate (TG) must be passed, which involves a decision whether the project is to continue and be assigned further funding, or if it is to be terminated.

Stakeholder roles are defined within the PM, such as project manager, project sponsor, quality assurance manager, and for the internal project organization, subproject managers. The customer stakeholder is typically only involved at the start of the project and after delivery. Project documentation standards are defined in the PM, such as the content and format of project specifications, project plans, quality plans, progress reports, etc.

(41)

41 Figure 1.13 Stage-Gate™

model [18]

In order to control the actual software development in the project, the organization integrates the management model with a software development process. This process is much more technically oriented and fits into the specific application domain for the project. There may be several sub-processes for different logical parts of the software and even sub-processes for hardware development that must be coordinated. The general phases and decision criteria are defined in terms of a technical content and technical documentation is added and linked to the different phases.

The project management model can thus be seen as a layered model. The technical development process is still on a rather high level of abstraction even though it describes work of technical nature.

An incremental approach can be defined, where development products are split into increments, delivered to other stakeholders according to a defined increment plan [58]. Milestones are defined for each of these incremental deliveries similar to those shown in Figure 1.13, and these are reported and monitored on the management level using the same reporting mechanism.

Launch

Discovery

Stage 1 Stage 2 Stage 3 Stage 4 Stage 5

Post launch review G2 G1 G3 G4 G5 Scoping Build business case

Development Testing & validation

(42)

42

2 Research Methodology in Software

Engineering

Research is the expansion of human knowledge, and research methodology deals with methods for doing this. Traditionally, empirical research methodology has been divided into two main paradigms, the qualitative and the quantitative methodologies. Each has been used with success in different disciplines and applications [59]. These paradigms can also be referred to as fixed and flexible designs respectively [59, 60]. These terms regard the amount of preparation performed by the researcher before the study takes place.

Research can be performed from various philosophical perspectives. These can be said to be positivist, interpretive and critical. Positivist research assumes that the researcher can perform the research without affecting the studied subject or environment. It also assumes that the subject can actually be measured upon and is accessible for direct measurement. Interpretative research assumes that the subject can only be accessed indirectly through tools such as language and nomenclature. Critical research assumes that social reality is historical and that it is produced and reproduced by people. This research focuses on the oppositions, conflicts and contradictions in society and strives to solve these.

Examples of other distinctions made in research are: objective versus subjective, being concerned with general laws (nomothetic) versus being concerned with uniqueness of each particular situation (idiographic), as aimed at prediction and control versus being aimed at explanation and understanding, as taking an outsider (etic) versus taking an insider (emic) perspective [61].

When preparing a research project the researcher must choose a research methodology. This can be done by discussing the purpose of the research and the epistemological status of the area. If there is a low level of previous knowledge, an exploratory study is probably the best selection of methodology. If a little more is known and a more detailed result is required, a descriptive methodology might be the better selection. If a lot is known and relationships are to be confirmed an explanatory methodology is probably the best selection. This can be illustrated as a research method staircase, as shown in Figure 2.1. In the figure, the type of knowledge in the research area changes as you go up the staircase.

(43)

43

Qualitative methods are generally lower in the staircase and quantitative methods are generally towards the upper part of the staircase [62]. Table 2.1 shows appropriate strategies for research depending on the aim of the research.

Figure 2.1 Research method staircase [62]

Type Research aim Requires control

over behavioural events?

Experiment How, why Yes

Survey Who, what, where,

how many, how much

No

Case study How, why No

Table 2.1 Relevant situations for different research strategies, adapted from Yin [63]

2.1 Quantitative

Research

Quantitative research has evolved through the natural sciences in order to explore the world. This scientific approach has the common feature of starting with a theory, which is then proven or disproved by an experiment. The scientific researcher is concerned with a certain theory and only this theory during the course of testing it. This implies that there is already much knowledge about the researched area and explicit relationships are to be proven as opposed to a

Integrating Management and Engineering Processes in Software Product Development