Towards a Multi-Dimensional Framework for Assessing the Value of. Software Projects

(1)

Towards a Multi-Dimensional Framework for Assessing the Value of

Software Projects

by

John Richard Kopp

Submitted in partial fulfillment

of the requirements for the Master’s degree in Software Development and Engineering

at

Seidenberg School of Computer Science and Information Systems Pace University

(2)

We hereby certify that this thesis, submitted by John Kopp, satisfies the thesis requirements for the Master’s degree in Software Development and Engineering and has been approved.

_____________________________________________-________________

Dr. Olly Gotel Date

Chairperson of Thesis Committee

_____________________________________________-________________

Dr. Frank Marchese Date

Thesis Committee Member

_____________________________________________-________________

Dr. Sotiris Skevoulis Date

Thesis Committee Member

Seidenberg School of Computer Science and Information Systems Pace University 2009

(3)

Abstract

Towards a Multi-Dimensional

Framework for Assessing the Value of

Software Projects

by

John Richard Kopp Submitted in partial fulfillment

of the requirements for the Master’s degree in Software Development and Engineering

June 2009

As the role of software within organizations and within society has increased, so have the costs of producing and maintaining it. Whether the value delivered by software has proportionally increased is a subject of debate. This phenomena has been coined the IT value paradox. Part of the difficulty in establishing value is that it is dependent on perspective. Value at the level of society, value at an organization level and value at an individual project level need to be evaluated differently. Value also depends on the viewpoint of the recipient. That received by society in general differs from that received by consumers and from that received by the organization. This thesis specifically examines the assessment of value of software projects from the perspective of the organization. Value is seen to be multidimensional. Financial value can be measured using techniques such as net present value calculations, internal rate of return and payback period under the assumption that the world is static. Flexibility by management is quantified by real option techniques and the dynamics of the marketplace by game theory. In addition to financial measures, understanding project alignment with the organization and with IT departments is essential to obtaining and understanding the value of projects. A framework incorporating these multiple dimensions is presented to allow systematic valuation of software projects.

(4)

Acknowledgements

I express my sincere gratitude to Dr. Gotel for her guidance, comments and feedback over the last year which greatly assisted me in developing brief initial thoughts on this topic into a more comprehensive understanding and framework. I thank her for teaching me how to perform academic research and how to frame a large topic, such as the one addressed in this thesis, in a manner so that it can be understood and communicated. Learning how to find and define relevant questions and how to fully and properly answer those questions was as valuable as learning the specific content of my thesis.

I also thank Dr. Marchese and Dr. Skevoulis for their efforts and feedback over the last year.

(5)

v

List of Tables

Table 1 Measures Used in Earned Value Techniques ... 13

Table 2 Cost and Schedule Variance (Based on [8]) ... 15

Table 3 Summary of Cost and Schedule Performance Indexes (Based on [8]) ... 16

Table 4 Cost Estimates at Completion (based on [8]) ... 17

Table 5 Time Estimates at Completion (based on [8]) ... 18

Table 6 Evaluation of Critical Success Factors for an Example Project ... 35

Table 7 Example of NPV Sensitivity Analysis ... 45

Table 8 Payback Period Example ... 48

Table 9 Comparison of NPV and IRR on Mutually Exclusive Projects (from [36]) ... 51

Table 10 Comparison of IRR and NPV for Three Projects ... 52

Table 11 Comparison of NPV, IRR and Payback Period ... 53

Table 12 November 11th Google Option Prices [47] ... 66

Table 13 Factors Impacting Option Prices [51] ... 79

Table 14 Analogies between Real and Financial Options for Option to Defer [54]... 87

Table 15 Present Values of Cash Flows for Movie Delivery Example ... 91

Table 16 Cash Flows Years 2 through 10 ... 92

Table 17 Classic Prisoner's Dilemma ... 100

Table 18 Analogies between Prisoners and Firms in Classic Prisoners Dilemma Game 103 Table 19 Prisoner’s Dilemma Game between Firms ... 105

Table 20 Analogies between Prisoners and Firms in Classic Grab the Dollar Game... 107

Table 21 Grab the Dollar Example ... 108

Table 22 Generic Process Goals ... 111

(11)

xi

Table 24 Causal Relationships Supporting Strategic Objective of Increased Sales ... 134

Table 25 Additional Measure Supporting Strategic Objective of Increased Sales ... 134

Table 26 Supplier-Customer Type Summary ... 154

Table 27 Assumptions, Measures and Uncertainties of Shrink Wrap Projects (Supplier's Perspective) ... 156

Table 28 Assumptions, Measures and Uncertainties of Shrink Wrap Projects (Customer's Perspective) ... 157

Table 29 Assumptions, Measures and Uncertainties of Bespoke Projects (Supplier's Perspective) ... 158

Table 30 Assumptions, Measures and Uncertainties of Bespoke Projects (Customer's Perspective) ... 159

Table 31 Assumptions, Measures and Uncertainties of Internal Projects ... 160

Table 32 Usefulness of Process Measures by Project Type ... 168

Table 33 Relevant Financial Measures by Project Type ... 170

Table 34 Value in Different Firm Types... 171

Table 35 Project Type Emphasized by Firm Type ... 172

(12)

xii

List of Figures

Figure 1 Relationship between Project and Product Life Cycles (based on [5]) ... 10

Figure 2 Illustration of PV, BAC, AC and EV ... 14

Figure 3 Tracking Project Performance using Earned Value Techniques ... 20

Figure 4 Earned Value Measures as Feedback (based on [9]) ... 21

Figure 5 Modeling Elements for Benefit Realizations Approach (from [10]) ... 26

Figure 6 Example of Benefit Realizations Approach ... 27

Figure 7 MBASE Model Integration Framework [14] ... 29

Figure 8 Plot of NPV vs. Discount Rate ... 46

Figure 9 Decision Tree for Option to Expand ... 59

Figure 10 Decision Tree Example ... 63

Figure 11 Profit from Nov Call Option with 320 Strike Price ... 69

Figure 12 Profit from Nov Call Option with 320 Strike Price ... 70

Figure 13 Profit from Nov Put Option with 320 Strike Price ... 71

Figure 14 Upper and Lower Bounds to Value of Call Option ... 72

Figure 15 Stock Prices of Hypothetical Security ... 76

Figure 16 Option Prices on Hypothetical Security ... 76

Figure 17 Price of Equivalent Portfolio ... 77

Figure 18 NPV Analysis for Option to Defer Example ... 84

Figure 19 Option Analysis for Option to Defer Example ... 86

Figure 20 Option Value Calculation - Option to Expand Example ... 89

Figure 21 Cash Flows for Internet Based Movie Delivery Example ... 90

Figure 22 Possible Changes in Demand (Movie Delivery Example) ... 93

Figure 23 Working Backwards in Movie Example ... 94

(13)

xiii

Figure 25 Organizational Balanced Scorecard Example ... 135

Figure 26 Balanced Scorecard Cascade [75] ... 137

Figure 27 Project Evaluation Framework ... 148

Figure 28 Supplier Types ... 151

Figure 29 Customer Types ... 153

Figure 30 Shrink Wrap Projects Financial Value Calculation (Supplier's Perspective) . 155 Figure 31 Shrink Wrap Projects Financial Value Calculation (Customer's Perspective) 157 Figure 32 Bespoke Projects Financial Value Calculation (Supplier's Perspective) ... 158

Figure 33 Bespoke Projects Financial Value Calculation (Customer's Perspective) ... 159

Figure 34 Internal Projects Financial Value Calculation ... 160

Figure 35 Project Evaluation against Firm's Objectives ... 161

Figure 36 Project Evaluation against IT Goals and Architecture ... 161

Figure 37 Project Evaluation against Project Success Factors ... 161

Figure 38 Returns from the Four Project Categories (based on [84]) ... 164

Figure 39 Project Classifications (from [85]) ... 165

Figure 40 Proposed Project Classification Scheme ... 168

Figure 41 Investment Boundary: ROI vs. Chance of Loss ... 173

Figure 42 Investment Boundary: NPV vs. Chance of Loss ... 174

Figure 43 Project Evaluation with respect to Organization and IT Goals ... 178

Figure 44 Project Evaluation with respect to Organization Goals and Project Success Factors ... 179

(14)

xiv

List of Equations

Equation 1 Future Value Of $100, One Period At 5% ... 40

Equation 2 Future Value After One Period... 40

Equation 3 Present Value After One Period ... 40

Equation 4 Present Value of $110 ... 40

Equation 5 Future Value After N Periods ... 41

Equation 6 Present Value Of An Amount To Be Received in N Periods ... 41

Equation 7 NPV of Word Processor Project Example Assuming 5% Discount Rate ... 42

Equation 8 NPV of Word Processor Project Example Assuming 12% Discount Rate .... 44

Equation 9 Rate of Return... 49

Equation 10 Single Period NPV Set to Zero ... 50

Equation 11 Discount Rate ... 50

Equation 12 Calculation of NPV for Example from Section 6.1.2 ... 50

Equation 13 Variance of Portfolio of Two Assets [39] ... 55

Equation 14 Capital Asset Pricing Model (Risk Premium) ... 56

Equation 15 Calculation of Beta ... 56

Equation 16 NPV Calculation at Start of Project for Full System ... 64

Equation 17 NPV Calculation at Year 1 Assuming System Upgrade ... 64

Equation 18 NPV Calculation at Year 1 Assuming No Upgrade ... 64

Equation 19 NPV Calculation at Start of Project for Partial System ... 65

Equation 20 NPV Calculation at Start of Project for Partial System with No Option to Expand ... 65

Equation 21 Relationship of Call Price to Hypothetical Portfolio at Start of Period ... 77

Equation 22 Relationship between Call Price and Hypothetical Portfolio with Increase in Stock Price ... 77

Equation 23 Relationship between Call Price and Hypothetical Portfolio with Decrease in Stock Price ... 77

(15)

xv

Equation 24 Number of Shares ... 78

Equation 25 Size of Loan in Portfolio ... 78

Equation 26 Value of Call... 78

Equation 27 Risk Neutral Probability ... 79

Equation 28 Black-Scholes Formula ... 81

Equation 29 NPV Analysis using Risk Free Discount Rate - Option to Defer Example . 85 Equation 30 NPV Analysis using Risk Adjusted Discount Rate - Option to Defer Example ... 85

Equation 31 Static NPV of Operating System Example ... 85

Equation 32 Option Value - Option to Defer Example ... 87

Equation 33 Project Value at Year 1 - Option to Expand Example... 88

Equation 34 Project Value at Year 1 with High Demand - Option to Expand Example .. 88

Equation 35 Project Value at Year 1 with Low Demand - Option to Expand Example... 88

Equation 36 Project Value at Start of Project - Option to Expand Example ... 89

Equation 37 Option Value Calculation - Option to Expand Example ... 89

Equation 38 Risk Free Probability (Movie Delivery Example)... 94

(16)

Chapter 1 Introduction

1.1 Overview

Over the last few decades, the cost of software to firms has grown significantly both in absolute terms and as a percentage of the budgets of firms. Given this increase in spending, one would expect to see a similar corresponding increase in value delivered to the firm and to society. This linkage has been difficult to demonstrate, leading to what has been described as the IT value paradox. Despite becoming ubiquitous within firms and society in general, software’s impact and that of software projects has been difficult to elusive to quantify. Part of the difficulty in establishing value is that it is dependent on perspective. Value at the level of society, value at an organization level and value at an individual project level need to be evaluated differently. Value also depends on the recipient. That received by society in general differs from that received by consumers and from that received by the firm. This thesis specifically examines the evaluation of value of software projects from the perspective of the firm.

The need to deliver value to the users of our products increases as the role of software in the success of enterprises and in society in general continues to grow. The value of software projects can be evaluated in multiple dimensions including the degree of satisfaction of stakeholder value propositions, financial terms and in terms of support for larger organizational goals or strategies. Value considerations can be used to focus

(17)

resources and to evaluate the usefulness of software processes toward the goal of delivering user needs.

Current project tracking and control methods, earned value techniques, as prescribed by professional organizations describe project tracking and control as being done against the plan of execution of the project. Completion of a project plan does not necessarily correlate with the delivery of value. Explicit consideration of value is required. Project tracking and control against a value baseline will lead to the better achievement of stakeholder goals, the organizations strategies and to financial benefits. The components required to establish such a value baseline are examined within this work.

There are several techniques used to quantify the financial benefits or returns expected from a software project. The Net Present Value (NPV) of a project is calculated by discounting future revenues and expenses by appropriate rates to calculate the total present value of the project. The internal rate of return and the payback period provide alternate views into the same numbers supporting the NPV calculation. Real options and the use of decision trees are used to quantify the value of flexibility within projects. Results from game theory can quantify the effects of market conditions, competition and market timing. The inputs into these financial techniques vary from sales and market forecasts for products sold in the marketplace to the quantification of business process changes for software products used internally by the firm. The applicability of these techniques and their use are examined.

Many techniques for eliciting, reconciling and measuring satisfaction of stakeholder value propositions have been developed and are prerequisites for determining and

(18)

obtaining value. Identifying all key stakeholders and understanding the role of the intended software within larger systems which include other software applications and human actors are significant to identifying and reconciling value propositions. The applicability and the use of techniques such as the Benefits Realization Approach, Model Based (System) Architecting and Software Engineering (MBASE) and goal oriented requirements engineering are examined within this thesis.

Projects exist within a larger framework of portfolios and also to varying degrees support or conflict with overall corporate strategies and with IT strategies. The degree to which an individual project complements other projects within a portfolio or supports corporate or strategy is another important measure of its value. The satisfaction of stakeholder value propositions refers to the degree that a project fulfils its own set of requirements. A project has additional value when viewed from a wider perspective. The alignment of projects within portfolios and with overall corporate and IT goals and strategies can be an important criterion in project selection and delivery of value. Balanced scorecard techniques are examined as a means of achieving this alignment.

Other factors play a role in successful delivery of value from a project and must be identified to develop more comprehensive project evaluation techniques. For instance, the level of innovation in a project and its level of complexity have impact on its chance for success and influence the project’s chance of delivering value. Factors such as the skills of the project team and the organizational environment are also crucial. The projects evaluation against these factors can be related to its chances for successful completion.

(19)

Ultimately, multiple value based techniques, such as those measuring the financial returns from a project and those measuring alignment with organizational goals need to be combined in order to establish comprehensive measures for a project. This thesis will examine ways to combine these techniques in a useful way for practitioners. A systematic framework for the evaluation of projects is presented.

1.2 IT Value Paradox

Robert Solow, a Nobel price winning economist is frequently quoted as saying “You can see the computer age everywhere but in the productivity statistics.” [1] In “IT Doesn’t Matter” [2], Carr provides a comparison that shows that the growth in spending on IT has paralleled growth in spending of railroads and electricity. He argues that each follows the same development pattern. Early adopters gain significant strategic advantages. As usage becomes more widespread, each no longer offers advantage, only parity. He argues that just as electricity and railroads became commodities, so too has IT. If IT is a commodity, then firms should seek to minimize spending, seek off the shelf solutions rather than firm specific distinct IT solutions and seek to ensure only that they have sufficient IT resources to have parity with competitors. If IT can offer strategic advantages to the firm, then increased spending on IT and development of firm specific applications and IT solutions is valuable. It would offer competitive advantages. Carr’s postulation and other similar ones cause considerable debate. By extension, if IT’s contribution to value is limited, then what about the contributions of individual academics and practitioners? [3]

(20)

The responses to arguments about IT’s lack of value, which predate Carr, can be divided into two groups. One attempts to understand sources of IT value and to improve value delivered through the use of requirements techniques. Any lacking in delivered value is seen as a failure to understand business needs and in the business changes necessary to effectively utilize IT solutions. These are examined in section 3.2, Stakeholder Value Proposition Elicitation, Analysis and Reconciliation. The second group attempts to understand IT value generated, both quantitatively and qualitatively. The value of IT can be studied at three levels.

At a national or society level. What has IT brought to the population?

At a firm level. What role does IT bring in allowing the firm to achieve greater wealth generation? Does IT bring competitive advantage to the firm? How can this be maximized?

At a project level. Simply, what is the value of individual projects? How should projects be selected, prioritized and monitored?

This thesis primarily studies the question at the project level. It examines techniques to determine the value of individual projects and of their selection. As part of understanding value, the scope and requirements of the project must also be well understood.

1.3 Basic Question Addressed in This Work

The basic question addressed in this thesis is the following. How can the value of a software project be understood and assessed? Consider the following two projects:

Project A was estimated to cost 200,000 and last for 6 months. It was completed for 180,000 in 5 months.

Project B was estimated to cost 300,000 and last for 9 months. It was completed for 800,000 in 18 months.

(21)

Which project had more value?

The correct answer is that you can’t tell from the information provided, yet it is not uncommon for practitioners and organizations to make assessments using the very type of incomplete information presented above. Only project cost and possibly the successfulness of the job done by the project manager are described. The business value of the project is not. It’s quite possible that project B delivered more value to the organization. This thesis is directed toward developing a systematic framework to answer questions of this nature, that is, to understand project value, both quantitatively and qualitatively.

1.4 Research Process

A survey and review of available literature from multiple fields including computer science and software engineering, general management, finance and project management was performed. Relevant approaches and concepts from these sources were combined and synthesized into a framework with the intended goals of producing a summary and guide useful to practitioners desiring to understand the sources and assessment of project value and of providing a step by step process that could be used for assessment of project value. Empirical validation of the proposed framework was outside the scope of this thesis given time and resource constraints, but would be a direction for future work.

1.5 Roadmap

(22)

Current State of Practice in Project Valuation: A discussion of what is commonly meant by project success and what should be meant is given. The benefits and lackings of earned value techniques, commonly used to track and measure project progress, are described.

Prerequisites for Project Valuation: Certain core competencies are required before systematic valuation of projects is possible. An overview of several requirements techniques, including the Benefits Realization Approach and goal based requirements engineering are presented. Good requirements techniques allow the costs and benefits of projects to be better understood and measured. Critical success factors, factors associated with a better chance of a project successfully completing are also presented.

Quantitative Approaches to the Measurement of Value: Financial measures of value,

calculated from estimated costs and revenues, are presented. Static techniques, such as net present value, internal rate of return and payback period, assume the course of the project will not change. Real options is presented as a technique to allow the benefits of managerial flexibility and choices to be quantified. Game theory is presented as a way to model and quantify competition. Process measures are presented as a means of quantifying the benefits of internal projects, that is, those that address internal business needs and do not produce a product for sale.

Qualitative Approaches to the Measurement of Value: Project value is

multidimensional. Quantitative (financial) techniques do not fully capture project value. Alignment with the strategic goals of the organization and with IT goals and architecture is considered.

(23)

A Proposed Multi-Dimensional Framework for Project Valuation: A proposed process for evaluating project value is presented. Both quantitative and qualitative techniques are combined to allow better assessment of value.

Application of Framework: An example of the use of the framework is presented. Due

to time and resource constraints, the example is fictional, but is designed to emphasize various aspects of the proposed process.

Conclusions and Future Work: Concluding remarks and possible future research

(24)

Chapter 2 Current State of Practice in Project Valuation

A clear distinction must be made to distinguish between the project being conducted and the product being designed and constructed. Many project management techniques and measures of success examine only the former while the later is the only source of value. Current practice utilizes techniques such as the earned value technique, which while useful as a means of controlling and measuring progress, provide no information on the value generated by a software project. There is no “value” in the earned value technique; it is actually a cost tracking technique.

2.2 Definition of Project Success

An understanding of the meaning of project success is essential in developing frameworks that can be used to value projects and to track success in the achievement of that value. The definition of success is a key element in understanding what elements need to be measured and tracked to evaluate project value. Success can be measured along several dimensions and evaluation of the project in each of these dimensions can be crucial for successful project delivery and to evaluate project value.

The distinction between the project and product and between the project life cycle and the product life cycle is important. “A project is a temporary endeavor undertaken to create a unique product, service or result” [4]. The product is the result of the project and has a

(25)

lifespan well past the end of the project. The relationship between the project and product cycles is illustrated in the figure below from [5].

Business Plan Product Life Cycle ID E A

Initial Intermediate Final

Operations Divestment Project Life Cycle P ro d u c t Upgrades

Figure 1 Relationship between Project and Product Life Cycles (based on [5])

Of note, it is during the potentially long operations period, when the product is in active use, when much of the business value promised by the project is delivered. During the divestment or end of life phase some of the cost of the product is incurred or potentially through salvage value, some of the cost of the product is recovered. Also significant is that the operations phase of the products life is typically much longer than the formal software project.

It is important to consider the difference between the project life cycle and the product life cycle. Traditional measures of project success, such as the earned value technique, discussed in section 2.3, have focused on time, cost and scope measures during the life of the project. This narrow focus on the project life cycle leads to only operational measures and misses the strategic value of the project. [6] It casts project success in terms of internal project measures rather than in terms of what the project delivers to the organization. It measures project delivery against the original plan, but fails to consider the value delivered to stakeholders. The project life cycle is typically considered to end

(26)

with the start of the operational life of the product, that is, when the product is delivered or implemented. By taking a limited view, looking only at the project life cycle rather than the whole product life, much of the value of the project is not accounted for.

Judgev [6] suggests that a more holistic understanding of project success can be achieved “by measuring success during operations and decommissioning when effectiveness measures are taken into account and involve input from different stakeholders”. By considering the whole product life, rather than just the project, we can measure both the project’s value to the organization and obtain feedback from stakeholders. Measuring the value during the product’s operational life is also critical feedback that allows assessment of valuation techniques done earlier in the project.

Another consideration in measuring project success is the difference between project efficiency and effectiveness. Efficiency looks at time, scope and cost. Effectiveness focuses on satisfying stakeholder requirements and goals. It is a wider view of project success. Efficiency is operational in focus measuring the success of the project against its plan. Effectiveness is more strategic measuring the value the project produced. An alternate description of this is the difference between project management success and project success. Cooke-Davies [7] makes the following distinction.

Project management success, being measured against the traditional gauges of performance (i.e., time, cost and quality), and

Project success, being measured against the overall objectives of the project. Many traditional measures of project success are actually measures of project management success rather than project success. There are many examples of successive projects that failed from an evaluation with respect to time and cost, and more

(27)

significantly, projects that succeeded on cost and time measures but failed overall. In the latter case, traditional project measures such as the earned value technique may declare the project a success and the project manager successful despite a lack of satisfaction of stakeholder goals. Jugdev describes this phenomena succinctly as an example of "the operation was a success, but the patient died” [6]. Project management success helps to complete the project and deliver a product. Project success is the measure of the business or strategic value delivered.

Jugdev [6] traces the evolution of measures of project success. Initially (1960 - 1980), success was defined mainly in terms of cost, time and scope measures. As noted earlier, these measures fail to consider stakeholder satisfaction or overall product success and as such fail to be good measures of project success and do not in themselves lead to effective means of measurement of value delivered. Value must consider the financial worth of the product and include an assessment of the project in terms of the stakeholder needs or satisfaction or in terms of the wider organization.

2.3 Earned Value Techniques

2.3.1 Introduction

The earned value technique allows tracking and control based on time and scope estimates and measurements. It allows for calculation of schedule and cost variances and provides a means to calculate estimated cost at completion and estimated time to completion as a project progresses. As shown later in this section, earned value, EV, techniques can serve a role in early detection of time and schedule variances and in

(28)

understanding the effectiveness of corrective actions. The earned value technique is in common use in practice and is frequently a main measure of project success even though it only it has no measures of value, only costs.

EV techniques are based on four measures or parameters [8]:

Table 1 Measures Used in Earned Value Techniques

Parameter Definition Example

Planned value (PV) or Budgeted Cost of Work Scheduled (BCWS)

This is the budget allocated for performing a task, work package or project itself. It is a time phased, meaning it is dispersed or spent as the task or project proceeds.

Assuming linear progress, a task estimated to take two months and cost 10000, would have a planned value of 5000 at the end of month one. Budget at Completion

(BAC):

This is the total budget allocated to a task, work package or project.

A task estimated to cost 10000 has a budget at completion of 10000. Actual Cost (AC) or

Actual Cost of Work Performed (ACWP)

This is the actual cumulative cost at an interim point in time spent on a task, work package or the project itself.

Suppose that at the end of month one, 6000 has been spent working on a task. Its actual cost is 6000 regardless of how much of the task is completed or what was estimated to be done.

Earned Value (EV) or Budgeted Cost of Work Performed (BCWP)

This is the cumulative (earned) value of work completed on a task, work package or the project itself at an interim point in time.

Assume a task is estimated to cost 10000. If it is 45% complete, its earned value is 4500.

PV, BAC, AC and EV as of the end of the first month for the example presented in the above table are illustrated below in Figure 2.

(29)

C u m m u la ti v e V a lu e Time (Months) 1 2 5000 10000 BAC PV at Month 1 EV at Month 1 AC at Month 1

Figure 2 Illustration of PV, BAC, AC and EV

2.3.2 Calculation of Earned Value (Example)

Earned value is best understood with an example. Suppose a project consists of two sequential tasks. Assume that the estimates have been reviewed and accepted and the budget is approved. The budgets at completion of the tasks are 20000 and 30000, respectively.

Task A: Estimated cost 20000, estimated duration 2 months Task B: Estimated cost 30000, estimated duration 3 months Total Project: Estimated cost 50000, estimated duration 5 months

Suppose after one month, 12000 dollars have been spent on task A and it is 45% completed.

The planned value, PV, of task A and the project at one month is 10000. This assumes the progress expected and spending are linear. The task was expected to take two months, so at the end of the first month, it was planned to have it half done.

(30)

The earned value, EV, of task A and the project at one month is:

EV = percent complete * budgeted cost = 0.45 * 20000 = 9000. From these measures, it is possible to calculate the cost performance and schedule performance of the project at one month and to make predictions about the cost and duration for project completion.

Table 2 Cost and Schedule Variance (Based on [8])

Term Calculation Meaning

Cost Variance, CV CV = EV –AC Absolute variance in cost. EV worth of value was generated and it cost AC Cost Performance Index,

CPI

CPI = EV/AC Ratio of value generated (EV) to actual cost (AC) Schedule Variance, SV SV = EV – PV Variance in value

generation against plan. EV worth of value was generated, but we had planned (scheduled) to accomplish PV worth of value

Schedule Performance Index, SPI

SPI = EV/PV Ratio of value generated (EV) to planned (scheduled) value (PV)

Critical Ratio (CR) CR = CPI * SPI A measure of project health [8]. Further described in Table 3 Summary of Cost and Schedule Performance Indexes.

Continuing with the example:

Cost variance (CV) = EV – AC = 9000 – 12000 = -3000. The interpretation of this is that 12000 dollars has been spent to create 9000 worth of value. Money is being spent than expected for the value produced.

(31)

Cost Performance Index (CPI) = EV/AC= 9000/12000 = 0.75. If money was spent exactly as planned CPI would equal 1.0. A CPI of less than one indicates that less value per dollar is being produced than that planned.

Schedule Variance (SV) = EV – PV = 9000 – 10000 = -1000. The interpretation of this is that 10000 dollars worth of value were planned to be created during the first month, but only 9000 worth of value were actually created. Progress is slower than planned.

Schedule Performance Index (SPI) = EV/PV = 9000/10000 = 0.9. If the project was exactly on schedule, SPI would equal 1. A SPI of less than one indicates that the project is behind schedule.

Critical Ratio (CR) = CPI *SPI = 0.75 * 0.9 = 0.675. The critical ratio is a measure of project health [8]. A critical ratio greater than one is desirable. Less that one indicates that the project is either behind on cost or schedule or on both. It can be seen that CPI and SPI individually are insufficient measures. Suppose it is costing more than estimated to create value (AC > EV or CPI < 0) and the project is ahead of schedule (EV >PV or SPI > 0). The project may cost more but may be completed sooner. Or suppose that it is costing less that estimated to create value (EV > AC or CPI > 0) and the project is behind schedule (EV < PV or SPI < 0). The project may cost less but may be completed later. Whether either of these two scenarios should be a cause of concern is very dependent on project particulars, but CR is a measure of the severity of the potential problem.

Table 3 Summary of Cost and Schedule Performance Indexes (Based on [8])

CPI SPI CR Cost Schedule Interpretation

> 1, EV > AC

> 1, EV > PV

>1 Ahead Ahead Positive

> 1, EV > AC

< 1, EV < PV

Depends Ahead Behind Mixed

< 1, EV < AC

> 1, EV > PV

Depends Behind Ahead Mixed

< 1, EV < AC

< 1, EV < PV

(32)

The example project is behind on both schedule and cost, corrective action may be indicated and the cause of this deviation must at least be understood. Was the initial estimate incorrect? Were the resources less skilled or less motivated than expected? Was there a steeper than expected learning curve? Certainly, the corrective actions taken depend on the cause, but equally significantly, the predicted cost at completion and predicted project duration that can be made at this point as well based on the cause of the deviation.

2.3.3 Predicting Cost at Completion

Earned value techniques can be used to estimate the cost of a task or project at completion. Two parameters of interest are

Estimate at Completion (EAC): Estimated cost of the project at completion Estimate to Completion (ETC): Estimated cost for the remainder of the task or project

The estimate to completion and estimate at completion at dependent on whether there is any variance in cost and the possible cause of the variance as summarized below.

Table 4 Cost Estimates at Completion (based on [8])

Case ETC EAC Comments

No cost variance BAC – AC = BAC- EV

BAC Original cost estimates correct, project on budget. The earned value (EV) is exactly equal to the actual cost (AC). For example, $1000 was spent to create $1000 worth of value. Original estimate incorrect Must create estimate for remaining work

AC + ETC New estimate needed for remaining work. Potential causes for error in estimate are ill defined scope, incorrect understanding of task or project

(33)

complexity or of skills of resources Over cost, but

original estimate valid BAC – EV AC + BAC – EV = BAC +AC – EV = BAC + CV

One explanation is that the learning curve was greater than expected. Once trained, resources can be expected to perform as by the original cost

estimate. EAC is the cost so far (AC) plus the original estimate for the remaining work (BAC – EV). Past performance is a good indicator of future work (BAC – EV)/CPI AC + (BAC – EV)/CPI = BAC/CPI

The original estimate for the remaining work (BAC – EV) is scaled by the performance to date (1/CPI) and added to the cost so far (AC).

Cost variance but no

correction

BAC – AC = BAC- EV

BAC In this case, there is cost variance, but it is assumed that the variance will

somehow be made up. This is usually not very realistic and not a good approach.

The value of the above table is that it provides a way to estimate the cost to complete a project or task and the total cost at completion given its current state. It also provides guidance on when a new estimation for the project is required.

2.3.4 Predicting Task or Project Duration

Analogous to ETC and EAC, it is possible to predict the time estimate at completion (TEAC) and time estimate to completion (TETC) using the following parameters:

Schedule at Completion (SAC): This is original estimated duration for a task or project.

Actual Time (AT): The actual duration of a task or project to date.

(34)

Case TETC TEAC Comments

No schedule variance

SAC – AT SAC Original time estimates correct, project on schedule Original time estimate incorrect Must create new time estimate for remaining work

AT + TETC New time estimate needed for remaining work. Potential causes for error in estimate are ill defined scope, incorrect understanding of task or project complexity or of skills of resources Behind schedule, but original estimate valid Original time estimate for remaining work.

SAC - TV One explanation is that the learning curve was greater than expected. Once training, resources can be expected to perform as by the original cost estimate.

Past performance is a good indicator of future work (SAC – AT)/SPI

SAC/SPI The original estimated duration for the remaining work (SAC – AT) is scaled by the performance to date (1/SPI) and added to the time spent so far (AT).

Cost variance

but no

correction

SAC – AT SAC In this case, there is time variance, but it is assumed that the variance will somehow be made up. This is not very realistic.

The value of the above table is that it provides a way to estimate the time required to complete a project and the total time the project will take.

2.3.5 Use of Earned Value Techniques in Project Control

Cost and time variances can be monitored over time as a project progresses as a means to track progress and as a means to evaluate the effectiveness of project controls. Below is an example of how CPI, SPI and CR are plotted and monitored as a project progresses.

(35)

Figure 3 Tracking Project Performance using Earned Value Techniques

The project is performing according to plan or better with respect to cost when CPI >= 1. The project is performing according to plan or better with respect to schedule when SPI >= 1. When either measure is less than one, the project may be in need of corrective measures. Earned value is used as a form of feedback to control the project against its original plan. The process is illustrated below.

Develop/Update

Plans Perform to Plan

Measure actual cost and completion status Is CPI > 1 && SPI > 1 Determine Corrective Action Yes No 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 R a t i o Time

Project Performance

CPI SPI CR

(36)

Figure 4 Earned Value Measures as Feedback (based on [9])

2.3.6 Discussion

Earned value techniques perform a useful role in tracking and controlling cost and schedule in software projects. They can be effective in controlling a project against an initial plan. The technique itself is not lacking, but the real problem is described in section 2.2. Earned value techniques measure project management success rather than project success. They measure conformity to a plan. There is no explicit consideration of whether the plan delivers value in financial terms or in terms of stakeholder value propositions.

Interestingly, all of the “value” measures in earned value techniques are actually cost measures. For instance, planned value, PV, is actually the estimated cost to complete the project.

Successful project control requires attention to progress against plans, possibly via earned value techniques, monitoring of the critical success factors described in section 3.3 and explicit consideration of the measures of value described in Chapter 4, Chapter 5 and Chapter 6 of this thesis.

(37)

Chapter 3 Prerequisites for Project Valuation

All project valuation techniques have an implicit assumption that the scope and requirements of the project are sufficiently understood to allow estimation of costs and of benefits. Without this level of understanding of the requirements of the project, accurate and meaningful estimates are not possible and thus, accurate assessment of value is not possible. Most software projects consists of both the design and construction of the product and an associated set of changes within the business or within processes to successfully adopt the product and adapt to use it well. Good requirements techniques are needed to identify the changes needed to successfully use a new or modified product and to obtain value from that product. A high level review of several requirements techniques is presented.

Achievement of value from a software project requires the successful completion of the project. Critical success factors are factors associated with higher chances of successful project completion. They can be evaluated to establish necessary prerequisite conditions before a project should be started and also can be used as a discriminator for project selection. Critical success factors for projects are described and a simple method for evaluation of projects against these factors is presented.

(38)

3.2 Stakeholder Value Proposition Elicitation, Analysis and Reconciliation

3.2.1 Overview

Good requirements techniques can be seen as a prerequisite for the valuation of projects. In all techniques to measure the value of projects, there is a critical implicit assumption that the scope of the project and the requirements for the project are known sufficiently well to support value estimations appropriate for the current stage in the project’s life cycle. For instance, when projects are initially requested or proposed, sufficient knowledge of their scope is required to make high level cost and benefit estimates. As requirements are elicited, analyzed and validated, cost and benefit estimates should be refined to reflect the better understanding of the project. Good requirements techniques can be seen as necessary to obtain the understanding and details of the project required to support valuation techniques.

IT expenditures represent a significant percentage of the expenditures of many organizations. A natural question is, what value is returned by these expenditures. Can this value be determined and produced by IT groups alone? Thorp [10] correctly states that IT efforts or projects are typically only a small portion of a larger business effort and it is necessary to consider the larger effort to both achieve and measure value to the business. In project management terms, value needs to be established from an overall program of business and IT projects. The IT project alone can neither deliver value nor be the measurement of value. Thorp [10] refers to this as “silver bullet thinking”. The IT project, by itself, does not deliver value and is not a silver bullet solution to the underlying business need.

(39)

3.2.2 Benefits Realization Approach 3.2.2.1 Introduction

The Benefits Realization Approach [10] is a framework that allows consideration of the overall business program, of which an IT effort or project is a part. It includes a simple modeling technique that supports reasoning about the associated efforts needed to achieve business value. There is an attempt to look at the entire set of conditions needed for success; not just the IT efforts alone. The entire product life is examined rather than just limited aspects of the software project. A concept to cash approach, rather than a design to development approach, is needed to achieve value.

3.2.2.2 Approach

Key to an understanding of this approach is the notion that any IT solution, regardless of its own merits, will not achieve its full value or benefit without wider and coordinated changes in the business. Thorp [11] notes that even for IT efforts that would appear initially to be primarily IT, when the product is fully integrated into the business much of the changes and investments are actually on the business side. Organizations do not always analyze, understand or track these related business costs. Thorp labels his approach the “Benefits Realization Approach”. Three key premises underlie the approach [11].

“Benefits do not just happen”

There is an adoption period for any new product and a learning curve as people learn how to best utilize its functionality. Related to this is the notion that successful realization of value (benefit) requires training and a migration plan from the current state to a new one utilizing the software developed.

(40)

“Benefits rarely happen according to plan” “Benefits realization is a continuous process”

These last two points noted by Thorp, really emphasize points noted in other parts of this thesis. The achievement of value (or benefit) requires several steps.

An understanding of the measures of value is needed. From the realms of finance and engineering economics, means of understanding the financial value of

projects (or of wider programs) are needed. Means of assessing the project in relation to organizational and IT strategies and goals are also needed.

A wide understanding of means to elicit and understand the benefits and values sought by stakeholders is needed. Modeling techniques such as the results chain and goal based methods can elicit the requirements that will yield the value sought by stakeholders. Getting this full set of requirements and goals and understanding both the business and IT components needed are keys to obtaining value.

The results of an initial analysis, an estimation of the value of the effort, a business case, sets of goals, requirements and initiatives must be taken as what they are: rough initial estimates of what needs to be done and what can be achieved.

Progress must be measured, financial calculations must be redone and achievements against models of stakeholder and organization value must be monitored as reality unfolds. Some of the initial estimates will be proven correct and some incorrect. Without measurement, the project or program is proceeding blindly. It’s unlikely to achieve the desired results given that it is quite unlikely that initial estimates and plans are perfect. Evaluation of financial models, requirements and goals, and business cases needs to be done early and often. 3.2.2.3 Results Chain Modeling

Results chain modeling is fully described in [12]. The interested reader is directed there. A very brief example is presented below as an illustration. The key point of this technique is to capture and understand all related business changes needed to accompany software changes to achieve value.

(41)

OUTCOME

INITIATIVE

ASSUMPTION CONTRIBUTION

Outcome: The results sought, including either intermediate outcomes in the chain, those outcomes that necessary but not sufficient to achieve the end benefit, or ultimate outcomes, the end benefit to be harvested.

Initiative: actions that contribute to one or more outcomes.

Contributions: the roles played by elements of the Result’s Chain, either initiatives or intermediate outcomes, in contributing to other initiatives or outcomes.

Assumptions: hypothesis regarding conditions necessary to the realization of outcomes or initiatives but over which the organization has little or no control. Changes to assumptions require updates in the Result Chain

Figure 5 Modeling Elements for Benefit Realizations Approach (from [10])

3.2.2.4 Results Chain Example

The following example is an enhanced version of an example from Thorp [13]. The enhancements include a consideration of training, of motivating factors and of associated business process changes.

A firm is experiencing a drop in sales. Feedback from customers includes complaints that order fulfillment is taking too long. The current order entry system is paper based, manually intensive and time consuming. An automated order entry system is proposed.

(42)

Initiative: Implement an automated order entry system. Outcome: Reduced order entry time (Intermediate outcome) Assumption: The time to fulfillment of an order is an important buying criterion Initiative: Train staff

to use new system.

Outcome: Increased sales

Initiative: Create financial incentives

for sales staff to adopt new system

(commisions). Decreased time to process order Allows staff to efficiently use new system

Reduces time for and resistance to adoption Reduced time to order fulfillment Initiative: Improve distribution system to reduce delivery time by storing products at multiple locations near customers. Reduced time to order fulfillment

Figure 6 Example of Benefit Realizations Approach

There are several key observations from this model. First, only one of the four initiatives shown involves software development. An initiative can be considered the equivalent of a project in standard project management terms. Achievement of the desired outcome of increased sales requires a series of related projects that involve both training and changes in business process. In project management terms, this collection of projects would be considered a program. A key assumption is that the time to order fulfillment is an important criterion for buyers when they are choosing whether to make a purchase. If this is false, for instance, if potential customers are interested only in price, then the benefits (value) desired from the program will not be achieved. Tracking, measurement

(43)

against both financial estimates and satisfaction of goals/requirements, and control, making modifications in the overall plan based on these measurements, also play a key role in achievement of value.

3.2.3 Model Based (System) Architecting and Software Engineering (MBASE) 3.2.3.1 Introduction

MBASE [14] [15] , Model Based (System) Architecting and Software Engineering, is another technique that attempts to identify all conditions that are needed to achieve value. A narrow focus on IT is again seen as too limited to achieve success. There is an attempt to look along four dimensions to maximize value. An iterative process is used to ensure that models representing each dimension are mutually consistent. The dimensions are as follows [14]:

Product: The system being developed. Models in this dimension represent requirements, code and architecture.

Process: The processes used to develop the system. Examples of process models include waterfall development models, spiral models, iterative approaches. Properties: These are models of the properties of the product or process. These include project management models, such as cost estimates and schedules, and models of the product that model performances or reliability.

Success: What each class of stakeholders needs to be satisfied. Success models include the financial models presented in Chapter 4 and stakeholder satisfaction models such as Win-Win or IKIWISI (I’ll know it when I see it).

(44)

Success Models

Win-Win; IKIWISI; Business-Case; Mission Models

Process Models Life-Cycle Waterfall Evolutionary Incremental Win-Win Spiral Anchor Points Risk Management Activities CMM KPA’s Product Models Domain Artifacts Requirements Architecture Code Documentation Packaging Embedded Shrink Wrapped Turn Key Product Line Property Models

Cost/Schedule; Performance; Assurance;Usability

Ent ry/E xit C riter ia V a lid a tio_n a_n d V e rif_ic a tio_n C rit_e ria

Product Development and Evolution Process Milestone Content; Planning and Contol

Evaluation and Analysis

Figure 7 MBASE Model Integration Framework [14]

As can be seen in the above figure, models are formed in each dimension. These models have interactions and there is a need for mutual consistency to achieve value. An iterative “spiral” approach is used to achieve consistency between these models.

3.2.4 Win-Win Model / Theory W 3.2.4.1 Introduction

At the core of techniques to elicit and deliver stakeholder value are models to understand stakeholder value and to reconcile the values of varied stakeholders. Theory W is a theory that states that project success can be obtained by satisfying the value propositions of all critical stakeholders. Any IT effort will have multiple stakeholders including end

(45)

users, customers, line of business management, IT management, project manager, analysts, developers, maintainers of the future system among others. Each stakeholder group will have their own goals which may conflict with other groups. Resolving these conflicts can be a key to delivering value.

3.2.4.2 Theory W

Theory W is simply, “make everyone a winner” [16].

Or more formally from [17]

“Making winners of the system’s key stakeholders is a necessary and sufficient condition for project success”

3.2.4.3 Is Win-Win possible?

Many situations appear at a first pass to be zero-sum; that is, they appear to be a win-lose situation where one parties will gain at the expense of another. A critical question is whether these situations can be converted to win-win through negotiation and expectation management. Theory W holds that unless everyone wins, success for a project is less likely.

The key to creating Win-Win situations is negotiation. In the book, Getting to Yes [18], five principles are established toward creating win-win situations. The reader is referred to that text for more details.

Don’t bargain over positions

A position is where someone stands on an argument. Arguing or bargaining over positions can only result in one party losing.

(46)

Negotiators are people. Their emotions and relationships can impact their positions and negotiations over those positions. It is important to separate those factors from the substantive problem at hand.

Focus on interests, not positions

It is critical to focus on the interests and needs of each stakeholder and not just their current position.

Invent options for mutual gain

Expanding or inventing options can turn a win-lose situation into a win-win. Insist on using objective criteria

The techniques described in Getting to Yes and from Theory W can enhance the identification and creation of Win-Win conditions. Under certain conditions win-win is possible. The examination of the conditions required is beyond the scope of this work, but reconciling stake holder value propositions is a significant step in achieving value from a project.

3.2.5 Goal Based Techniques 3.2.5.1 Introduction

A key part of obtaining value from a software endeavor is eliciting requirements. Fully understanding and developing requirements is crucial to determining stakeholder value propositions, that, simply, what they value. Goal modeling is a technique that can be used to establish requirements. The establishment and definition of goals is recognized as an important component in many aspects of requirements engineering. The creation of a hierarchy of related goals at varied levels of abstraction can lead to better identification, elicitation and elaboration of requirements. Analysis of goals by refinement, asking “how”, and by abstraction, asking “why”, can lead to a more complete set of requirements. Common GORE approaches include GBRAM [19] [20], i* [21] [22], the NFR framework [23] and KAOS [24] [25]. Detailed study of these approaches is outside

(47)

the scope of this thesis, but the interested reader is directed to the references noted. General GORE concepts are described below.

3.2.5.2 Goal-Oriented Requirements Engineering Concepts

Goals are fundamental to the requirements process. Yu noted [22]: “Perhaps the elucidation and manipulation of goals is a natural and inherent part of doing RE, even though earlier RE methods have not made this explicit, and have not provided the associated support. This interpretation is plausible since requirements by its very nature represents a target to be reached, a wish to be fulfilled, a vision to be materialized”.

“Goal-oriented requirements engineering (GORE) is concerned with the use of goals for eliciting, elaborating, structuring, specifying, analyzing, negotiating, documenting, and modifying requirements” [24]. Traditional requirements practices focus on what the system is intended to do, that is, on its functionality. Users and other sources of requirements typically express their needs as list of features. In contrast, GORE techniques focus on why, leading to a more thorough understanding of the proposed system within its environmental context and to a more complete, consistent set of requirements, and on how, which allows generation and comparison of alternative implementations. A better understanding of the project can lead to better achievement of value.

The relationship between goals and requirements is analogous to the relationship between design and code. Requirements implement the goals. Alternate sets of requirements can be evaluated to determine how well they satisfy an identified hierarchy of goals.

(48)

Gore techniques can play a role across many aspects of requirements engineering, but they are particularly relevant early in the process. Goals are fundamentally at a high level and fully understanding stakeholder intent and motivations first can lead to successful requirements development. Two fundamental approaches to GORE are:

Begin with an initial set of user goals at a high level and derive sub goals. Successive and iterative refinement of sub goals leads to requirements.

Begin with a system description, scenarios or functional specification. From these, determine an initial set of goals. Successive and iterative refinement of these goals leads to a more complete set of requirements.

The concept of a goal is fundamental. A goal is an objective or intended purpose for a system. Goals exist at varied levels of abstraction. At the highest level they may express the overall strategy of the organization, for instance, "offer accounting services to clients". At the lowest level they may be satisfied by the actions of a single agent and map directly into a requirement or assumption. Goals can be concerned with either functional or non-functional aspects of a system. Functional goals represent the individual operations or services that the system supports. Non-functional goals are system qualities. Goal oriented techniques provide a means for the qualitative and quantitative systematic evaluation of both functional and non-functional goals. Requirements can be linked back to non-functional goals and the satisfaction of these goals by the proposed set of requirements can be evaluated.

Many non-functional goals are soft goals. A soft goal is one that can be satisfied to varied extent. For instance, usability is a soft goal. The final system usability will be supported to different extents by different sets of requirements, designs and implementations.

Towards a Multi-Dimensional Framework for Assessing the Value of. Software Projects

Towards a Multi-Dimensional Framework for Assessing the Value of

Software Projects

Abstract

Towards a Multi-Dimensional

Framework for Assessing the Value of

Software Projects

Acknowledgements

Table of Contents

List of Tables

List of Figures

List of Equations

Chapter 1

Introduction

Chapter 2

Current State of Practice in Project Valuation

Project Performance

Chapter 3

Prerequisites for Project Valuation