Enhancing the Quality of Planning of Software Development Projects

(1)

ENHANCING THE QUALITY OF PLANNING OF

SOFTWARE DEVELOPMENT PROJECTS

Marco Antônio Amaral Féris

A thesis submitted for the degree of Doctor of Philosophy of

The Australian National University

(2)

STATEMENT OF ORIGINALITY

I certify that the thesis entitled Enhancing the Quality of Planning of Software Development Projects submitted for the degree of Doctor of Philosophy of the Australian National University is an original work, which is the result of my own independent intellectual effort. Where reference is made to the work of others, due acknowledgment is given.

--- Marco Antônio Amaral Féris

ANU College of Business and Economics The Australian National University

(3)

ACKNOWLEDGEMENTS

I dedicate this thesis and the completion of my PhD degree to the two most important and influential women in my life: my wife and my mother.

Words cannot express how grateful I am to my wife, Solange Braga Manto, who has encouraged and supported me tirelessly in my academic pursuits. She has allowed me to put our life on hold and to sacrifice our time together so I could strive to fulfil my dream.

My mother, Dulce Helena Cabral Amaral, has instilled in me the confidence that I can accomplish anything that I strive for. Although she cannot be physically present to share in the celebration of the completion of this degree, the

memories of her and her love and support will sustain me, as it has in the past, for the remainder of my years.

It is with particular appreciation that I wish to thank my Chief Supervisor,

(4)

Special thanks to Professor Shirley Gregor, my supervisor, who is one of the most respected educators and researchers in the field of Information Systems and Design Science research methods. Her effort and patience in guiding me throughout my research have enabled me to contribute to the academic field and the practitioner world.

I would especially like to acknowledge and extend my appreciation to Dr Vesna Sedoglavich, my supervisor, for encouraging me and for transferring the

technology from her research field, International Business, to the project management literature.

My most sincere appreciation to Dr Liam O’Brien, also a member of my supervisory panel, for his support and brilliant comments during this journey, and for his suggestion to develop research aligned with business needs.

(5)

Jaques Paes, Jairo Avritchir, Joan Angel, João Alexandro Vilela, João Cláudio Ambrosi, John Smyrk, José Eduardo Klippel, Prof Keith Houghton, Prof Kerry Jacobs, Leonardo Velasco, Lorenzo Tadeu, Luciano Kuhn, Luiz Gerbase, Luiz Kruse, Marcos Arend, Mark Lucas, Matthew Buckley, Mauricio Barbieri, Melissa Nortfleet, Michel Silas Guilherme, Prof Moni Behar, Prof Neil Fargher, Newton Costa, Osmar Brune, Pablo Leandro de Lima, Paul Wong, Paulo Ribeiro, Rafael Michel, Rafael Rauter Herescu, Renato de Pádua Moreira, Ricardo Angrisani, Ricardo Beck, Ricardo Moraes, Rick Van Haeften, Saar Paz, Samantha Gilbert, Samuel Fernandes, Saul Bencke Junior, Silvio Luis Leite, Susan O'Neil, A/Prof Taisy Silva Weber, Telma Camargo, A/Prof Thomas Kalliath and Trista Mennen.

Finally, I cannot forget to say thank you to the entities that supported me during my journey: National ICT Australia (NICTA), Elite Editing, which edit my thesis according to Standards D and E of the Australian Standards for Editing Practice, the Division of Information, and especially to the Research School of

(6)

(7)

PREFACE

When I was 10 years old, in 1978, my English teacher asked me to leave the classroom. As I was a very active child, I expected to be reprimanded. However, she asked me to answer an

interview for the school’s magazine, and among other questions, she asked what I would like to be in the future. Without

hesitation, I answered that I would like to be an engineer.

Five years later, I was persuaded to study the vestibular, a very competitive exam that allows students with higher marks to enter university. One month before the exam, my mother, in a tremendous financial effort, gave me my first computer, which had a Z80 processor and 2 KB of RAM! I was fascinated and stopped studying for the exam in order to start developing computer programs and small electronic devices to modify the new computer. Fortunately, I still achieved good exam marks and entered the electronic engineer and computer science courses at university.

(8)

In 1988, I started working at Altus, an organisation that develops, manufactures, integrates and deploys systems that automate other industries. I had the opportunity to develop and manage several projects there, including systems for the automation of the IBM site, an oil platform and a steel-making house. At the end of the 1990s, I obtained a graduate certification in Business

Management at FGV, which is recognised for tradition and quality and is always among those placed first in evaluations by the Brazilian Ministry of Education.

In 2000, I had the opportunity to work in the manufacture of Dell, which involved launching servers, notebooks, desktops,

peripherals and software in the Latin American market. Later, I moved to the information and technology (IT) area where, among other projects, I was responsible for the development, testing and support of Dell’s online stores for Latin America, which was

considered one of the most important projects for Dell in 2006. Moreover, I obtained a Six Sigma Green Belt certification.

(9)

The year of 2009 was critical, as my wife and I decided to leave a very structured life and a consolidated carrier, so I could

undertake a doctorate at one of the most important universities in the world, and supported by one of the most respected academics in the field of project management. This was not an easy decision, as it involved a loss of prestige (from an R&D manager to a

student), lack of financial support (we both left good jobs), distance (Australia and Brazil are on the opposite sides in the world) and family (my mother was very sick and passed away when I was far from home). However, we decided to go and face this tremendous challenge. In Australia, I had the opportunity to learn a lot, improve my English skills, teach classes at two universities, define and establish the project management process, and coach other project managers.

For medical reasons, we decided to return home in the second half of 2011, and after I expressed an interest to my former colleagues, I received a job offer to work at AEL to be a program manager responsible for the development of several high-tech avionics and defence systems for the KC-390, the new military aircraft that Embraer is developing. This is what I am doing in parallel with this research. I cannot complain about monotony.

2008

2009

2010

(10)

(11)

ABSTRACT

As business competition gets tougher, there is much pressure on software development projects to become more productive and efficient. Previous research has shown that quality planning is a key factor in enhancing the success of software development projects. The research method selected for this study was design science research (DSR), and the design science research process (DSRP) model was adopted to conduct the study. This research describes the design and development of the quality of planning (QPLAN) tool and the quality of planning evaluation model (QPEM), which are two innovative artefacts that evaluate the quality of project planning and introduce best planning practices, such as providing references from historical data, suggesting how to manage in an appropriate way and including lessons learnt in the software development process. In particular, the QPEM is based on cognitive maps that represent the project manager’s know-how, project manager’s characteristics and technological expertise, as well as top management support, enterprise environmental factors and the quality of methods and tools in a form that corresponds closely with humans’ perceptions. Data were collected from 66 projects undertaken in 12 organisations from eight types of industries in six countries. The results show that the QPLAN tool has been significantly contributing to enhancing the success rate of projects.

(12)

(13)

LIST OF FIGURES

Figure 2.1: Typical project lifecycle (adapted from PMI, 2013) ... 25

Figure 2.2: Research model for testing the effectiveness of quality of planning in project management success and project ownership success ... 42

Figure 3.1: Cognition in DSR cycle (adapted from Vaishnavi and Kuechler, 2012) ... 48

Figure 3.2: Knowledge contribution framework (adapted from Gregor and Hevner, 2013) ... 57

Figure 3.3: DSRP model applied to this research (adapted from Peffers et al., 2006) ... 59

Figure 3.4: QPEM ... 63

Figure 3.5: QPLAN tool main screen ... 64

Figure 4.1: Design of the QPEM model ... 72

Figure 4.2: Cognitive map (adapted from Stach et al., 2005) ... 75

Figure 4.3: Design of QCM ... 77

Figure 4.4: QCM model ... 80

Figure 4.5: Develop project management plan ... 82

Figure 4.6: Define scope ... 86

Figure 4.7: Create work breakdown structure ... 87

Figure 4.8: Define activities ... 88

Figure 4.9: Sequence Activities... 89

Figure 4.10: Estimate activity resources ... 90

Figure 4.11: Estimate activity durations ... 91

Figure 4.12: Develop schedule ... 92

Figure 4.13: Estimate costs... 93

Figure 4.14: Determine budget ... 94

Figure 4.15: Plan quality ... 95

Figure 4.16: Develop human resource plan ... 97

Figure 4.17: Acquire project team ... 98

Figure 4.18: Plan communications ... 99

Figure 4.19: Plan risk management. ... 101

Figure 4.20: Plan procurements ... 103

Figure 4.21: Project manager characteristics ... 104

Figure 4.22: Technological expertise ... 105

Figure 4.23: Top management support ... 107

Figure 4.24: Enterprise environmental factors ... 109

Figure 4.25: Quality of methods and tools ... 111

Figure 5.1: QPLAN main screen ... 115

Figure 5.2: QPLAN approach for enhancing project success ... 124

Figure 5.3: Example of interview ... 126

Figure 5.4: Example of registering a new project ... 127

Figure 5.5: Example of project classification in planning ... 128

(18)

Figure 5.7: Example of planning factors evaluation at the beginning of

planning ... 130

Figure 5.8: Example of planning factors evaluation at the end of planning ... 131

Figure 5.9: Example of planning products evaluation at the end of planning ... 133

Figure 5.10: Example of QIPlan and QIPlan Org ... 135

Figure 5.11: Example of planning processes quality indices ... 136

Figure 5.12: Example of enterprise environment factors cognitive map ... 137

Figure 5.13: Example of expanded Karnaugh map ... 139

Figure 5.14: Example with suggestions for enhancing the quality of planning ... 141

Figure 5.15: Example of average quality of planning processes ... 142

Figure 5.16: Example of issues reported by project managers ... 142

Figure 5.17: Example of raw data exported by QPLAN ... 143

Figure 5.18: Example of project success valuation at the end of planning ... 144

Figure 5.19: Indication of project success in the main screen ... 145

Figure 5.20: Example of lessons learnt ... 146

Figure 5.21: Example of project classification at the end of project ... 147

Figure 5.22: Differences founded in the project classification made at the beginning of planning and at the end of the project ... 148

Figure 5.23: Project report showing the differences founded in the project classification made in the beginning of planning and at the end of the project ... 149

Figure 5.24: Example of planning factors evaluation at the end of the project ... 149

Figure 5.25: Example of demographic information ... 151

Figure 6.1: IPO modified to test QPLAN (adapted from Zwikael and Smyrk, 2011) ... 155

Figure 6.2: QPM index test ... 156

Figure 6.3: QPLAN project report and the quality indices calculated ... 157

Figure 6.4: Research model for testing the effectiveness of quality of planning in project management success and project ownership success ... 172

(19)

LIST OF TABLES

Table 2.1: Sixteen planning processes used by the PMPQ model... 34

Table 3.1: Philosophical grounding that underpins this study ... 55

Table 3.2: QPLAN testing and evaluation design... 65

Table 3.3: Testing and evaluation steps according to Pries-Heje et al.’s (2008) framework ... 66

Table 4.1: Conversion scale for QPM ... 73

Table 4.2: Conversion scale for QCM (for edges that have a positive causal relationship with quality of planning) ... 83

Table 4.3: Conversion scale for QCM (for edges that have a negative causal relationship with quality of planning) ... 90

Table 5.1: Planning processes code for showing in QPLAN ... 136

Table 6.1: Test scenarios for QPM quality index... 155

Table 6.2: Expected results ... 158

Table 6.3: Questionnaires used for collecting data from current and past projects ... 168

Table 6.4: Data collected by industry type and ongoing and past projects ... 169

Table 6.5: Data collected by industry type and country ... 170

Table 6.6: List of programming languages used ... 170

Table 6.7: Means, standard deviations, reliabilities, and correlations ... 176

Table 6.8: Regression—project management success ... 177

Table 6.9: Regression—project ownership success ... 177

Table 6.10: Paired sample t test of means compared ... 182

Table 6.11: Percentage of projects before and after defining new thresholds ... 182

Table 6.12: Regression—quality of planning over time ... 186

Table B.1: Questionnaire 1—Beginning of Planning (QCM) ... 236

Table B.2: Questionnaire 3—End of Planning (QCM) ... 238

Table B.3: Questionnaire 4—End of Project (QCM) ... 240

Table B.4: Questionnaire 5—End of Project (QCM) ... 241

Table C.1: QCM Test Scenario 1—Answers as ‘Strongly agree’ ... 245

Table C.2: QCM Test Scenario 2—Answers as ‘Agree’ ... 246

Table C.3: QCM Test Scenario 3—Answers as ‘Neutral’ ... 247

Table C.4: QCM Test Scenario 4—Answers as ‘Disagree’ ... 248

(20)

(21)

GLOSSARY

Term Description

QPEM QPEM stands for Quality of Planning Evaluation Model. This model evaluates the quality of planning of software development projects.

QPM QPM stands for Quality of Planning by Manager. This is a measure from the PMPQ model (Zwikael and Sadeh, 2007, p760). The QPEM model uses it to evaluate the quality of planning through a top–down approach.

QCM QCM stands for Quality of planning through Cognitive Maps. This is a measure from the QPEM model that evaluates the quality of planning of software development projects from the evaluation of 55 factors, organised in 21 cognitive maps that affect project planning.

QIPlan QIPlan stands for Planning Quality Index. This index is calculated by QPEM to represent the quality of project planning of software development projects. It is the average of QPM and QCM, and it ranges from 0.0 (lowest) to 1.0 (highest).

(22)

(23)

Chapter 1:

Introduction

This chapter lays the groundwork for this thesis. It begins by showing that, despite the significance of computer software for the world economy, project development has had a low success rate for the last two decades. Section 1.2 deals with the main focus of this research, which is the planning of software development projects. Section 1.3 identifies the knowledge gaps in the project management literature, which indicate a lack of effective evaluation models and tools for determining the quality of planning for software development. Section 1.4 outlines the research questions that aim to decrease this gap, while Section 1.5 presents the research objectives that aim to answer the research questions. Section 1.6 outlines the research method adopted to conduct this applied research: design science research (DSR). Section 1.7 outlines the contributions made by this research. The structure of this thesis is presented in Section 1.8, and Section 1.9 concludes the chapter.

1.1 Introduction

(24)

Software organisations are taking over large slices of the economy from other sectors (Krishnan et al., 2000). For example, Google is the largest direct-marketing platform, and Netflix is the largest video service by number of subscribers (Andreessen, 2011). In the automotive industry, cars have been launched on the market with software to control their engines and safety features, entertain passengers, and guide drivers to their destination. In the oil and gas industry, software has been used for the automation and control of operations that are essential for exploration and refining efforts. The defence industry has planes that do not have human pilots and missiles that achieve their targets guided by software. In some cases, software organisations have become leaders in traditional industries; for instance, Amazon is the world's largest bookseller. More than one decade ago, Borders handed over its online business to Amazon because it thought that online book sales were unimportant (Andreessen, 2011).

(25)

but the success rate was still low; only 39 per cent of projects were completed successfully (Obeidat and North, 2014), leading to estimated annual losses for the United States (US) and European Union (EU) markets of around US$100 billion each (Symons, 2010).

1.2 Planning for Enhancing Project Success

As business competition gets tougher, there is much pressure on software development projects to become more productive and efficient. Many factors affect the success rate of software development projects, including high level of complexity (Wohlin and Andrews, 2005), level of project management knowledge, project manager’s characteristics and level of technical expertise, level of top management support, effective communication, enterprise environment factors, and quality of methods and tools used (Bechor et al., 2010). To complicate matters further, it is usually not obvious how these factors interact (Obiajunwa, 2012; Wohlin and Andrews, 2001).

(26)

The planning involves the establishment of a more formalised set of plans to accomplish the project’s goals (Shepperd and Cartwright, 2001), including an estimation of time, resources and costs, and identification of the critical path (Dawson and Owens, 2008; Dvir and Lechler, 2004), risks (Tesch et al., 2007) and alternative solutions (Alblas and Wortmann, 2012; Bannerman, 2008). This is the phase before the funder makes the major investment, and costs of changes are typically low. However, in this phase, the level of uncertainty regarding planning is at its peak (Howell et al., 2010; Bakker et al., 2010), it is difficult to set realistic limits and goals for projects because of limits set by the available information (Bakker et al., 2010), risks are usually under-analysed and under-managed (Bannerman, 2008; Willcocks and Griffiths, 1994), and when attempting to understand the business context, there can be a lack of awareness of the major relationships between business objectives and project´s goals (Flynn and Arce, 1997). Planning is characterised by the opportunities and risks that may lead to the project’s success or failure; for instance, definition of requirements (Gornik, 2007), estimations of time and cost (Dvir and Lechler, 2004; Napier et al., 2009), and identification (Tesch et al., 2007) and mitigation of the project’s risks (Gornik, 2007).

(27)

1.3 Knowledge Gaps

Given that quality of planning has a demonstrated causal relationship with project success (Zwikael and Globerson, 2004), the project management literature and the software industry offer a myriad of models, methods and tools for evaluating the quality of project planning. Significant examples are the project management planning quality (PMPQ) model (Zwikael and Globerson, 2004); checklists are used for measuring phase completion and readiness (e.g., planning phase exit milestone), guiding reviews (e.g. error prevention), and ensuring adherence to procedures (e.g. quality assurance of software engineering) (Houston, 2004); metrics are considered a vital part of the software industry because of their contribution to improved quality and productivity through the efficient use of the feedback mechanism, based on rationale that one cannot improve something without first measuring it (Gopal et al., 2002); and tools, such as SEER-SEM, a planning tool for software projects (Lagerström et al., 2012).

(28)

Checklists depend on expert knowledge of a process to be effective (Houston, 2004). Most metrics used in the software industry are based only on quantitative data, although others factors must be considered in the planning evaluation, such as pressure from marketing to deliver the product in the shortest timeframe (even with lower quality). The SEER-SEM planning tool focuses on the efficiency of the development process to deliver the project’s output, and not on the perceived benefits of the project for customers.

This leads to the need for the development of a new approach for evaluating the quality of planning of software development projects. That is, there is a knowledge gap in both the project management literature and the software industry with respect to a lack of effective quality of planning evaluation models and tools for software development projects.

1.4 Research Questions

Motivated by the significance of the software industry around the world and the low success rate of software development projects, the following primary research questions have been formulated to guide this research:

RQ1: Does improvement in the quality of planning of software development projects enhance project success rate of these projects?

(29)

1.5 Research Objectives

To answer these two research questions, this research has three main objectives, which aim to contribute to the project management literature and the software industry:

1. examine the effect of quality of planning on the success rate of projects in various types of software projects, organisations, industries and countries 2. develop a model that evaluates the quality of project planning of software

development projects

3. develop a tool that enhances the success rate of projects by evaluating the quality of planning of software development projects and introducing best practices in the software development planning process.

1.6 Research Method

(30)

DSR and a mental model for the research output. The DSRP model has six steps, which are listed below, along with descriptions of how they were applied in this research (see Chapter 3 for more details).

1. Problem identification and motivation: Section 1.1 shows that the software industry is significant for the world economy; however, the low success rate of software development projects has plagued the industry in the last two decades. Section 1.2 shows that, despite researchers’ continuous efforts in relation to planning, results have not been effective over time. The proposal of this thesis is to continue focusing on planning, but to aim at improving the understanding of the effect of planning on software development projects in order to identify opportunities that may lead to an increased success rate. 2. Objectives of a solution: Section 1.5 shows that this research aims to:

(31)

identify strengths and weaknesses of planning (Sedoglavich, 2008), identifying project characteristics for defining proper planning (Shenhar and Dvir, 2007) and implementing a mechanism for planning process improvement (Iversen et al., 2004).

4. Demonstration: Chapter 6 demonstrates the utility of the QPLAN tool in 12 organisations in six countries.

5. Evaluation: Chapter 6 tests and evaluates the QPEM and QPLAN tools using a variety of quantitative and qualitative methods. Statistical analysis will be used for testing hypotheses.

6. Communication: Section 3.4.5 describes the communication of this research to academics and practitioners.

1.7 Research Contributions

(32)

(Appendix B), and the use of cognitive maps for mapping the relations between them. The combination of these concepts and knowledge creates a novel approach for quality planning evaluation and extends the PMPQ model.

(33)

1.8 Structure of the Thesis

The subsequent parts of the thesis are organised according to the DSR publication schema proposed by Gregor and Hevner (2013). An overview of each chapter is presented below.

Chapter 2 reviews the relevant literature related to project success and planning, focusing on software projects.

Chapter 3 outlines DSR as the research method adopted in this thesis, as well as the process model (Peffers et al., 2006) selected to conducting the research.

Chapter 4 describes the development of the QPEM, which evaluates the quality of planning of software development projects and fills the gaps found in the project management literature.

Chapter 5 describes the development of the QPLAN tool for the software industry. This tool evaluates the quality of planning and introduces best practices in the software development planning process.

Chapter 6 evaluates QPLAN in terms of functionality, completeness, accuracy, reliability, usability and fit with the organisation’s needs.

(34)

1.9 Chapter Summary

The objective of this chapter was to introduce the thesis and the need for the research work. It started by explaining that, despite the significant effect of software on the world’s economy, the low success rate of software project development has plagued the IT industry for many years.

The introduction was followed by the proposed solution for reversing this situation— that is, a focus on planning, which is a critical phase of software development projects (Pinto and Slevin, 1988; Johnson et al., 2001; Belout and Gauvreau, 2004; Zwikael and Globerson, 2004) to identify opportunities that may lead to an increase in the success rate of projects.

The chapter then identified gaps in the current knowledge and the lack of effective quality of planning evaluation models and tools for software development projects.

Likewise, two research questions were raised. The first research question aims to test whether the enhancement in the quality of planning of software development projects enhance success rate of these projects. The second research question assumes that this statement is true, and it aims to identify how the effectiveness of the quality of planning of software development projects can be better evaluated and improved.

(35)

quality of project planning of software development projects, and a tool must be developed that enhances the success rate of software development projects.

Next, DSR was presented as a research method, and the DSRP model was identified for use in conducting the research.

The two research contributions were then outlined: a model that evaluates the quality of planning of software development projects (QPEM), and a tool that enhances the success rate of projects by evaluating the quality of planning and introducing best practices into the software development planning process (QPLAN).

(36)

(37)

Chapter 2:

Literature Review and Theory Development

2.1 Introduction

The previous chapter showed that, despite the significant effect of software on the world’s economy, the low success rate of software development has plagued the IT industry for many years. Guided by the two research questions (Section 1.4), this chapter reviews the relevant literature related to project success and planning, from an investigation of 87 articles published in 43 project management, general management and computer science leading journals between 1969 and 2015. Section 2.2 describes the evolution of the project success concept over time and the different points of view of success. It concludes with a recent and more elaborate concept. Section 2.3 presents the characteristics of project planning and how project management approaches deal with it. Section 2.4 discusses the effectiveness of planning on project success and presents a research model and set of hypotheses for testing the effectiveness of quality of planning on project success. Section 2.5 concludes this chapter.

2.2 Project Success

2.2.1 Introduction

(38)

down (Shenhar and Dvir, 2007). Researchers tend to see a crisis regarding software development and conclude that their research will improve the success rate of projects (Eveleens and Verhoef, 2010; Glass, 2005). Among others, Krishnan et al. (2000) claimed that the low success rate of software development projects has plagued the IT industry for many years. Moløkken-Østvold and Jørgensen (2005) stated that software development projects have a bad reputation for exceeding their original estimates. Although these findings have been questioned by Eveleens and Verhoef (2010) and Glass (2005, 2006), the fact is that the project success rate is low (Culmsee and Awati, 2012). Symons (2010) found that the estimated annual losses for the US and EU markets were around US$100 billion for each market. This work aims to increase the success rate of these projects.

For theorists, the definition of project success is ambiguous (Rai et al., 2002). For example, a software project where the customer is satisfied with the software’s functionalities and performance, but that misses the project’s budget or schedule goals by 10 per cent, may not be a successful project. The customer will say that it is a successful project, but the financial manager from the organisation that developed the software may say that it is a failure (Glass, 2005; Schaupp et al., 2009).

(39)

services was created to supply organisations that did not want to spend large amounts of money (Campbell-Kelly and Garcia-Swartz, 2007), and success depended on the technical quality of the system (Petter et al., 2012). In the mid-1960s, as hardware costs dropped, organisations started buying computers with software to run applications to meet their needs and after-sales services. In the late 1970s, with the advent of low-powered and independent personal computers, the mass-market industry was established, and hardware and software were sold in high volumes at low prices (Campbell-Kelly and Garcia-Swartz, 2007). At this time, success meant producing systems that could contribute to decision-making criteria and reduce costs (Petter et al., 2012). From 1980 to 1990, success was reducing the development life cycle, enhancing the system’s performance and obtaining user satisfaction with the systems and quality of the information provided. From 1990 to 2000, success involved the strategic value of IT, team performance, project quality and service quality (Petter et al., 2012). The Internet now connects all types of hardware and software, the industry is internationalised and there are endless opportunities for new businesses that have increased software development to an unprecedented degree (Campbell-Kelly and Garcia-Swartz, 2007). Compared to other eras, the success criteria are broader, and they consider effects on society (Petter et al., 2012).

(40)

2.2.2 Measuring the Development Performance of a Software

Project

The traditional definition of project success was made four decades ago, when Avots (1969, p.78) suggested implicitly that project success is determined based on scope/quality, time and cost: ‘some of the more obvious indications (of project failure) are high costs or schedule overruns, poor-quality products, or, as in the case of sophisticated systems, failure to meet project objectives’. That is, project success is defined as delivering the project on time, within budget and according to specifications. These three success dimensions are also known as the Iron (or Golden) Triangle (Atkinson, 1999; Toor and Ogunlana, 2010; Zwikael and Smyrk, 2011).

(41)

The Iron Triangle has also been criticised by other researchers. Dvir and Lechler (2004) argued that it does not investigate the effect of success on project performance during its lifecycle. Scott-Young and Samson (2008) claimed that it ignores important outcomes such as client satisfaction, longer-term business success and the preparation of the organisation for the future. Bakker et al. (2010) stated that this definition does not fit the context of software projects very well because requirements change during the project lifecycle, thereby influencing time and cost plans. Consequently, it is almost impossible to provide adequate estimations (Bakker et al., 2010).

2.2.3 Project Management Success and Project Ownership

Success

The evolution of the concept of project success over time has demonstrated the need for new dimensions for testing the benefits that the project aims to provide. This leads to a distinction between two success concepts. According to Zwikael and Smyrk (2011):

 Project management success is for testing the efficiency of the development process to deliver the project’s outputs. The Iron Triangle can be applicable.

(42)

2.2.4 Measuring the Benefits of the Software Product

Developed

The view of success for measuring the benefits provided by the project has accompanied the evolution of the IT industry over the years (Petter et al., 2012). Pinto and Slevin (1988) included the effect on the customer as a success dimension—that is, assessment of the usefulness of the project, level of satisfaction and effectiveness for the intend users. Pinto and Mantel (1990) used the same concept in other research.

In the software industry, Atkinson (1999), Wohlin and Andrews (2001) considered long-term properties, such as maintainability and evolvability factors, as additional success dimensions. For Bradley (2008), project success was related to organisational effects and deliverables on time and budget. However, Schaupp et al. (2009) stated that it is not possible to define a common list of factors to assess project success for website development, as the factors vary across website types.

(43)

creation of new technological and operational infrastructures (Shenhar and Dvir, 2007).

In the information systems (ISs) field, a significant research stream is the work of the DeLone and McLean, who developed a model for measuring success in ISs in 1992 (Petter et al., 2012). Named the D&M IS Success Model, it is dependent on the organisational context (DeLone and McLean, 2003) and aims to synthesise different measures of effectiveness. The model has six interdependent dimensions of IS success:

1. system quality: desirable features (e.g., flexibility, reliability and response time)

2. information quality: desirable characteristics (e.g., relevance, understandability, accuracy and usability)

3. system use: degree and manner in which staff and customers utilise the capabilities of the system

4. user satisfaction: level of satisfaction with the outcomes provided by the system (Petter et al., 2008)

5. effects of the system on individuals

6. effects of the system on the organisation (Petter et al., 2008).

(44)

measuring factors such as responsiveness, accuracy, reliability, technical competence and empathy of the personnel staff (Pitt et al., 1995). Second, individual impacts and organisation impacts were collapsed into net benefits, which measure the extent to which ISs are contributing to stakeholders’ success, such as improved productivity, increased sales, cost reductions, improved profits and job creation (DeLone and McLean, 2003).

Later, Lechler and Dvir (2010) published a more detailed view of project success with four distinct success dimensions. Each one is utilised extensively in the literature:

1. efficiency, for measuring the extent to which time and cost plans have been met (Scott-Young and Samson, 2008; Malach-Pines et al., 2008; Zwikael and Sadeh, 2007; Dvir and Lechler, 2004; Dvir et al., 2003)

2. effectiveness, for measuring the extent of benefits that the project brought to its client (Malach-Pines et al., 2008; Scott-Young and Samson, 2008; Zwikael and Sadeh, 2007; Dvir et al., 2003; Shenhar and Dvir, 2007) 3. customer satisfaction, for measuring the extent of satisfaction with the

benefits provided by the project and how it was conducted (Malach-Pines et al., 2008; Scott-Young and Samson, 2008; Zwikael and Sadeh, 2007; Dvir et al., 2003; Shenhar and Dvir, 2007; Atkinson, 1999)

(45)

2.2.5 Conclusion

This section reviewed the literature regarding project success. From the traditional project success criteria defined four decades ago until the present day, it showed the evolution of the success concept over time and across different points of view of success. It started by presenting the definition of traditional project success. The success concept was then refined in two different ways: project management success, for measuring the efficiency of the development process, and project ownership success, for measuring the benefits that the project provides to stakeholders. This section concluded by presenting a recent and more detailed concept of project success.

2.3 Project Planning

2.3.1 Introduction

Some practitioners and organisations consider that all projects are similar, and they suggest that success can be achieved by well-defined methods and a common set of tools and techniques for planning and managing their activities. This misconception has contributed to the low success rate of projects (Krishnan et al., 2000).

(46)

timeframe (even with lower quality) and high degree of novelty (Rodriguez-Repiso, et al., 2007b), and the diversity of projects is continuing to grow (Howell et al., 2010).

Given this context, is it possible to claim that a particular project management framework is the most suitable approach for all types of projects? Different project management approaches should be associated with different types of projects (Shenhar et al., 2005) in order to increase the likelihood of achieving success.

This section deals with project planning, which is a critical success factor for software development projects (Pinto and Slevin, 1988). It starts by presenting project planning characteristics that have opportunities and risks that may lead to project success. This is followed by a description of several project management approaches that can deal with planning, as an improper managerial approach may be considered one cause (Zwikael and Bar-Yoseph, 2004) of disappointing results in the software industry (Krishnan et al., 2000). Finally, based on rationale that one cannot improve something without first measuring it (Gopal et al., 2002), this section presents three methods for evaluating the quality of planning in software development projects.

2.3.2 Project Planning Characteristics

(47)

analyses and project plans are mature enough for conducting the project through the next phases and achieving the desired goals (Gornik, 2007).

[image:47.612.147.508.313.520.2]

This is the project phase before the funder makes the major investment. Here, the level of effort steadily increases, the level of uncertainty remains high but tends to decrease towards the end of the phase, and the costs of changes are typically low, but costs that influence the final characteristics of the project’s product begin to rise (see Figure 2.1).

Figure 2.1: Typical project lifecycle (adapted from PMI, 2013)

(48)

(Alblas and Wortmann, 2012; Bannerman, 2008), and risks (Tesch et al., 2007) and their mitigation (Gornik, 2007).

This is not an easy task. In the planning, a project’s uncertainty peaks (Howell et al., 2010; Bakker et al., 2010). It is difficult to set realistic limits and goals because of limited available information (Bakker et al., 2010). Risks are usually under-analysed and under-managed (Willcocks and Griffiths, 1994). When attempting to understand the business context, there is a lack of awareness of the major relationships between goals and aims to sustain the desired outcomes (Flynn and Arce, 1997). Issues are even more severe when some might conclude that planning is not necessarily helpful or even desirable (Dvir et al., 2003). In summary, project planning is characterised by having opportunities and risks that may lead to project success.

2.3.3 Project Management Approaches to Planning

2.3.3.1 Introduction

(49)

Project management now deals with projects as sets of practices aimed at providing better products to customers through integration considering organisational practices and being effective in terms of resource utilisation (Parast, 2011). Nevertheless, despite continuous efforts, results have not been effective over time (Bakker et al., 2010). These disappointing results call for the need to enhance project management approaches (Zwikael and Bar-Yoseph, 2004; Howell et al., 2010), which are usually variations of the traditional project management approach promulgated by the Project Management Body of Knowledge (PMBOK) (Nicholas and Hidding, 2010).

(50)

2.3.3.2 Stage-Gate Model

Created by Robert Cooper, Stage-Gate is a sequential development process that aims to promote result-oriented thinking by introducing five stages for managing activities, budgets and resources over time, and five gates with acceptance criteria for moving from one phase to the next.

In Stage 2—Build Business Case (planning), the project manager is responsible for analysing the project technically and developing the PMP, which is an input for Gate 3—Go to Development (Cooper et al., 2002).

2.3.3.3 Critical Chain Project Management (CCPM)

Created by Eliyahu Goldratt (Goldratt, 1997), CCPM is based on his TOC. It aims to modify common behaviours of team members by including buffers on the duration of tasks to be safety, which usually leads to delivering fewer features than expected and missing project deadlines (Pinto, 2002). CCPM is focused on the planning and executing phases.

(51)

2.3.3.4 Agile

Published by the Agile Alliance in 2001 (Fowler and Highsmith, 2001), Agile is a flexible methodology (Howell et al., 2010) that focuses on the individual rather than processes in order to promote an iterative and incremental way of thinking to address unavoidable changes (Noor et al., 2008).

The planning is made by sprints rather than project phases, and it tends to be tailored by practitioners for their specific needs. For example, Intel Shannon uses two planning stages—one at the start of the project and one at the start of each sprint—with milestones aligned with sprint completions (Fitzgerald et al., 2006).

2.3.3.5 Microsoft Solution Framework (MSF)

Created by Microsoft in 1994, MSF is a milestone-driven approach (Jenkin and Chan, 2010) for the entire project development lifecycle (Microsoft, 2005). It aims to be a flexible approach to accommodate different types and sizes of projects (Microsoft, 2005) through five phases: initiation, planning, developing, stabilising and deploying.

(52)

2.3.3.6 IBM Rational Unified Process (RUP)

The IBM RUP is a development process aimed at ensuring the development of high-quality software that meets the customer’s needs within a predictable schedule and budget. RUP has guidelines, templates and tools (Karlsson and Wistrand, 2006) for developing software iteratively, managing requirements, verifying quality, controlling changes and visually modelling the structure and behaviour of architectures and components (Gornik, 2007). RUP has four project phases: inception, elaboration, construction and transition (Dahanayake et al., 2003).

In the elaboration phase (planning), the project manager analyses the problem domain, establishes the software architecture and develops the PMP that serves as an input for the stakeholders to decide whether the project should go to the next phase (Jenkin and Chan, 2010).

2.3.3.7 Projects IN Controlled Environments version 2

(PRINCE2)

(53)

1. Seven principles, to determine who should do what, when and why: continued business justification, learn from experience, defined roles and responsibilities, manage by stages, manage by exception, focus on products and tailored to suit the project environment (Tomanek et al., 2014; Kruger and Rudman, 2013; Karamitsos et al., 2010).

2. Seven processes, to define how the jobs get done: starting up a project, directing a project, initiating a project, controlling a stage, managing product, delivery, managing a stage boundary and closing a project (Tomanek et al., 2014; Kruger and Rudman, 2013; Karamitsos et al., 2010). 3. Seven themes, which must be addressed continually throughout the

project: business case, organisation, quality, risk, plans, change and progress (Tomanek et al., 2014; Kruger and Rudman, 2013; Karamitsos et al., 2010).

4. Project Environment, the need to tailor PRINCE2 to a specific context (Kruger and Rudman, 2013).

(54)

2.3.3.8 Conclusion

The section explored several project management approaches that deal with planning in different ways. Stage-Gate is a sequential approach, while Agile is iterative and MSF and RUP are a mix of both. In terms of best practices, CCPM aims to modify common behaviours of team members by including buffers on tasks duration as a safety time. RUP provides more tools and templates related to the development process, PRINCE2 can be tailored to a specific context, whereas MSF deals with fewer details in a more general way (Santos, 2007). This discussion served for helping project managers to select a proper managerial approach for planning (Zwikael and Bar-Yoseph, 2004), which should be according to the project’s characteristics (Shenhar and Dvir, 2007).

2.3.4 Evaluating the Quality of Planning

2.3.4.1 Introduction

(55)

2.3.4.2 PMPQ Model

Zwikael and Globerson (2004) developed the PMPQ model to evaluate the quality of project planning through the evaluation of planning products. The model has been validated and utilised extensively in the literature (e.g., Zwikael and Globerson, 2004, 2006; Masters and Frazier, 2007; Zwikael and Sadeh, 2007; Papke-Shields et al., 2010; Zwikael and Ahn, 2011; Barry and Uys, 2011; Rees-Caldwell and Pennington, 2013; Zwikael et al., 2014).

The overall project planning quality indicator in the model, called the PMPQ index, consists of two subindices: quality of planning by organisation (QPO), which evaluates the organisational support processes, and quality of planning by manager (QPM), which evaluates the project’s know-how processes. QPO represents the means that the organisation places at the disposal of the project manager to enable proper project planning, execution and completion. It is a weighted linear combination of the 17 organisational support-related variables related to organisation systems, cultures, styles, structure and project office (Zwikael and Sadeh, 2007).

(56)

[image:56.612.70.507.212.708.2]

Table 2.1 shows the 16 core planning processes used by the PMPQ model (Zwikael and Globerson, 2004), organised into nine project management knowledge areas defined by PMBOK (PMI, 2013).

Table 2.1: Sixteen planning processes used by the PMPQ model

Knowledge Areas Planning Process Description

Integration Develop project

management plan

Documents actions necessary to define, prepare, integrate and coordinate all subsidiary plans

Scope

Define scope Development of a detailed description of the project and product

Create work breakdown structure

Subdivide project deliverables and project work into smaller, more manageable components

Time

Define activities Identify specific actions to be performed to produce the project deliverables

Sequence activities Identify and document relationships among activities

Estimate activity resources

Estimate type/quantities of material/people/equipment/ supplies required to perform each activity

Estimate activity durations

Approximate the number of work periods needed to complete each activity

Develop schedule Analyse activity sequences, durations, requirements and constraints to create the schedule

Cost

Estimate costs Develop an approximation of the monetary resources needed to complete project activities

Determine budget Aggregate the estimated costs of individual activities to establish an authorised cost baseline

Quality Plan quality Identify quality requirements and documenting how

the project will demonstrate compliance

Human resources

Develop human resource plan

Identify and document roles, responsibilities and required skills, and report relationships

Acquire project team

Confirm human resources (HR) availability and obtaining the team necessary to complete project assignments

Communications Plan communications Determine project stakeholder information needs and define a communication approach

Risk Plan risk management Define how to conduct risk management activities for a project

Procurement Plan procurements Document project purchasing decisions and the

(57)

2.3.4.3 Checklists

Other approaches to assess the quality of planning include checklists. Based on expert knowledge of a process (Houston, 2004), checklists are used for measuring phase completion and readiness (e.g., planning phase exit milestone), guiding reviews (e.g. error prevention), and ensuring adherence to procedures (e.g. quality assurance of software engineering). Checklists are extensively used in organisations that had adopted: the capability maturity model integration (CMMI) model (Barbour, 2001); Six Sigma, which is considered a complementary approach for CMMI because of its characteristic of continuous process improvement (Mahanti and Jiju, 2009); and ISO/IEC standards (Barbour, 2001) dedicated to software, such as ISO/IEC 15939, which defines a measurement process applicable to system and software engineering and management disciplines, and the SQuaRE model, for covering software quality requirements specifications and software quality evaluations.

(58)

For software development projects, checklists are used for measuring phase completion and readiness, guiding reviews, and ensuring adherence to procedures, with a low cost. Examples of checklists for software development

a) Checklist for dealing with cryptography (adapted from Microsoft, 2007, p.34):

[ ] Code uses platform-provided cryptography and does not use custom implementations. [ ] Keys are not held in code.

[ ] Access to persisted keys is restricted. [ ] Keys are cycled periodically.

[ ] Exported private keys are protected

b) Checklist for dealing with sensitive data (adapted from Microsoft, 2007, p.28):

[ ] Secrets are not stored in code.

[ ] Database connections, passwords, keys or other secrets are not stored in plaintext. [ ] Sensitive data is not logged in clear text by the application

[ ] The design identifies protection mechanisms for sensitive data that is sent over the network. [ ] Sensitive data is not stored in persistent cookies.

However, the quality and usefulness of a checklist depends on how it is produced. Checklists are valuable only to the extent that they incorporate expert knowledge of a process, including lessons learnt from past projects (Houston, 2004).

2.3.4.4 Metrics

(59)

implemented and utilised, they should lead the software organisation towards more disciplined processes through the efficient use of feedback mechanisms.

The rationale to use metrics arises from the notion that one cannot improve something without first measuring it (Gopal et al., 2002). In more detail, from better recognition of issues, practitioners can better manage the software development process and make the necessary changes to increase productivity and quality, thereby reducing cycle times and costs in the long run. Examples of metrics are quality of planning index (QIPlan) and the organisation project quality index (QIPlanOrg, which are described further in Sections 4.2 and 5.2.

However, many companies find metrics a complex matter and difficult to undertake. Less than 10 per cent of the industry classifies metrics programs as positive, and most metrics initiatives do not last beyond the second year. To be successful, the implementation of a metrics program should have the support of the organisation and be easy to use (Gopal et al., 2002). In addition, practitioners should understand that metrics are not the goal, but an important tool that highlights problems and gives ideas as to what can be done (Daskalantonakis, 1992).

2.3.4.5 Conclusion

(60)

products from 16 core planning processes defined in the PMBOK (Zwikael and Sadeh, 2007), and checklists and metrics, which are widely used by quality management and process improvement systems such as ISO/IEC standards, CMMI and Six Sigma.

All three methods have limitations. The PMPQ model was not designed specifically for software development projects, it does not evaluate the specific factors that affect the 16 core planning processes, and it does not consider the relationships among them, which are significantly correlated with project success (Ling et al., 2009). Checklists depend on expert knowledge of a process to be effective (Houston, 2004). Metrics are based only on quantitative data, although there are others factors to consider in the planning evaluation, such as pressure from marketing to deliver the product in the shortest timeframe (even with lower quality). This leads to the need to develop a new approach to assess the quality of project planning software development, and to integrate the best of each approach presented and overcome their limitations. This will be described in Chapter 4 (QPEM Model).

2.3.5 Conclusion

(61)

project planning on software development projects in order to identify opportunities that may lead to an increase in the success rate of projects.

2.4 Effectiveness of Planning in Project Success

2.4.1 Introduction

This section discusses the effectiveness of planning on project success. It starts by discussing the existing debate in the literature, where most researchers argue that planning is a critical factor for enhancing the success rate of projects. However, others claim that its importance is overplayed. This is more pronounced in software projects whose characteristics differ from other engineering projects (Rodriguez-Repiso et al., 2007b). For example, volatility of requirements, intangibility of software products and high level of complexity of the system continuously challenge project managers (Napier et al., 2009). Motivated by this debate, this section will then present a research model and the hypotheses to test it.

2.4.2 Planning Debate in the Literature

(62)

contributors to project success. Gornik (2007) from IBM argued that planning is the most critical project phase for software development projects.

However, some researchers have suggested that effectiveness of planning in project success has been overplayed. Dvir and Lechler (2004) recognised that planning is necessary, but it is not a sufficient condition for a successful project because it is difficult to determine precisely which activities—and estimated costs and duration—must be carried out in order to complete the project. This is also valid for software development projects (Rose et al., 2007). Dvir et al. (2003) and Dvir and Lechler (2004) suggested that success is insensitive to the level of implementation of management processes and procedures, but that requirements management—part of the project management plan—has a positive correlation with success. Rodriguez-Repiso et al. (2007a) and Conforto and Amaral (2010) argued that traditional planning approaches present limitations for the development of innovative products because they are characterised by project complexity, unpredictable activities and changes. Ika and Saint-Macary (2012) further claimed that the effect of planning on success is a ‘myth’.

Some researchers have identified planning factors that may lead to project success, such as level of collaboration, level of risk and type of projects:

(63)

b) level of risk, where planning is more effective in high-risk projects than in low-risks projects (Zwikael and Sadeh, 2007)

c) type of project, where the effect of planning of construction projects is higher than in software projects (Zwikael, 2009).

Others have suggested that project managers should focus on subsidiary plans such as cost (Butler and Fitzgerald, 1999), schedule, scope and HR management plans (Linberg, 1999).

Recent studies have indicated that project managers should have appropriate planning for each type of project (Shenhar and Dvir, 2007), reduced to a minimum required level (Dvir and Lechler, 2004), and be able to handle uncertainty (Bakker et al., 2010), constant requirements and goal changes (Karlström and Runeson, 2005; Fitzgerald et al., 2006; Noor et al., 2008; Chow and Cao, 2008). The next section presents a model aimed at helping project managers define the best way to plan and manage projects according to the project’s characteristics.

2.4.3 Research Model and Hypotheses

(64)

[image:64.612.149.425.92.224.2]

Figure 2.2: Research model for testing the effectiveness of quality of planning in project management success and project ownership success

For testing the effectiveness of planning on project management success, two opposing hypotheses were raised: H1 assumes a positive causal relationship

between planning and success, and the null hypothesis (H01) is opposed to this

affirmative:

H1—A higher level of quality of planning is associated with enhancement in project management success.

H01—A higher level of quality of planning is not associated with enhancement in project management success.

Likewise, for testing the effectiveness of planning on project ownership success, two opposing hypotheses were raised: H2 assumes a positive causal relationship

between planning and success, and the null hypothesis (H02) is opposed to this

affirmative:

H2—A higher level of quality of planning is associated with enhancement in project ownership success.

Effectiveness

Customer Satisfaction

Business Results Efficiency

Quality of Planning

Project Management

Success

+

Project Ownership

(65)

H02—A higher level of quality of planning is not associated with enhancement in project ownership success.

2.4.4 Conclusion

This section discussed the debate in the literature about the effectiveness of quality of planning in project management success and project ownership success in relation to software development projects. To contribute to this debate, this thesis developed a research model and a set of hypotheses.

2.5 Chapter Summary

(66)

Enhancing the Quality of Planning of Software Development Projects