Science of Computer Programming

Contents lists available atSciVerse ScienceDirect

Science of Computer Programming

journal homepage:www.elsevier.com/locate/scico

Quantifying forecast quality of IT business value

J.L. Eveleens

a,∗

, M. van der Pas

b

, C. Verhoef

a

a_{VU University Amsterdam, Department of Computer Science, De Boelelaan 1081a, 1081 HV Amsterdam, The Netherlands} b_{Vodafone, Portfolio & Innovation Management, Am Seestern 1, D-40547 Dusseldorf, Germany}

a r t i c l e i n f o

Article history:

Received 24 August 2010

Received in revised form 31 March 2011 Accepted 6 July 2011

Available online 28 July 2011

Keywords:

IT business value Forecast Estimation Reference cone Forecast to actual ratio Estimating Quality Factor EQF

IT

a b s t r a c t

This article discusses how to quantify the forecasting quality of IT business value. We address a common economic indicator often used to determine the business value of project proposals, the Net Present Value (NPV). To quantify the forecasting quality of IT business value, we develop a generalized method that is able to account for asymptotic cases and negative valued entities. We assess the generalization with real-world data of four organizations together consisting of 1435 IT assets with a total investment cost of 1232+million Euro for which 6328 forecasts were made. Using the generalized method, we determine the forecasting quality of the NPV, along with the benefits and cost using real-world data of another 102 IT assets with a total business value of 1812 million Euro. For the real-world case study, we will find that the quality of the forecasted NPVs is lower than the forecasted benefits, which is again lower than the forecasting quality of the cost. Also, we perform a sensitivity analysis to investigate the impact on the quality of an asset’s forecasted NPV when the forecasting quality of benefits or cost improves. Counterintuitively, it turned out in this case study that if the quality of cost forecasts would improve, the overall quality of its NPV predictions would degrade. This underlines the importance of both accurate cost and benefit predictions. Finally, we show how to use the quantified forecast information to enhance decision information using two simulation examples.

©2011 Elsevier B.V. All rights reserved. 1. Introduction

Organizations need to undertake IT projects every year to consolidate and expand their business. These IT projects often play a profound role in modern companies due to their size and impact. This stresses the importance of their adequate management by the Chief Information Officer (CIO).

More and more, voices are raised [43,20] that the CIO must manage IT projects to maximize their business value and return, instead of controlling their cost. For instance, in 1998 a research project known as Beyond Budgeting was started to change the current management model [23]. Where previously the management was focussed on planning and control of cost and the technical aspect of IT, the research emphasizes the importance of the business value.

Pisello et al. [48] argue that the CIO needs to become the Chief Financial Officer (CFO) of IT to improve the organization’s value. But, what does managing the business value of IT entail? An article by Lorie et al. [40] states that executives face three tasks in achieving good financial management, among which the correct forecasting of the expected cash flows. In this article, we focus on that particular task.

In information technology, the META group [20] showed that forecasting, especially of benefits, is far from common practice. Of the organizations that were surveyed, 84% indicated that no business cases were made for their IT projects or only for a select few projects. And if forecasts of the cash flows are made, the question is how to assess their validity.

∗_{Corresponding author. Tel.: +31 20 5987760.}

E-mail addresses:[email protected](J.L. Eveleens),[email protected](M. van der Pas),[email protected](C. Verhoef). 0167-6423/$ – see front matter©2011 Elsevier B.V. All rights reserved.

That proper forecasting is indeed a challenging task, is shown, for instance, by a survey of the Kellogg School of Management [36]. The survey found that 82% of the responding CIO’s regarded forecasting of IT benefits as a major challenge. On top of that, the survey showed that 68% do not track project benefits at all and only 26% track actual financial metrics after having made an investment decision. So, how do we know that forecasts of the IT business value are accurate, unbiased and reliable enough to support the decision making process? And are organizations able to adequately predict the business value of IT investments?

In this article we will address these questions. First, we discuss common economic indicators often used to determine the business value of proposals. We elaborate on one of these indicators, the Net Present Value (NPV). For this indicator, we will assess the forecasting quality. Since the overall forecasting quality of the NPV is determined by the forecasting quality of its components, such as benefits and cost, their forecasting quality is assessed as well.

We assess the quality of NPV forecasts, benefits and cost using real-world data of 102 IT assets. We obtained this data from a telecommunication organization, Z, that structurally makes business cases for its IT investments. Moreover, this organization uses procedures to evaluate completed investments. The data represents an NPV value of 1812 million Euro, with discounted benefits of 4714 million Euro and an investment value of 173 million Euro. Combined with data from other organizations, this article discusses 1620 IT assets with an investment cost of 1232

+

million Euro for which in total 6513 forecasts are made.

For the data of organization Z, we determine the accuracy of the forecasts and check for potential biases. We found that the quality of the forecasted NPVs is lower than the forecasted benefits, which is again lower than the forecasting quality of the cost. Also, there turned out to be a significant difference in forecasting quality between assets classified as Cost Reduction or New Product Development. The NPV, benefits and cost forecasts of Cost Reduction assets were more accurately predicted than that of the New Product Development assets. Moreover, whereas the forecasts of Cost Reduction assets showed no biases, the New Product Development forecasts of benefits and cost were generally overestimated in the real-world case study.

We also performed a sensitivity analysis to investigate the impact on the quality of an asset’s forecasted NPV when the forecasting quality of benefits or cost improves. Counterintuitively, it turned out that if the quality of the cost forecasts would improve, the overall quality of the NPV predictions would degrade. This is caused by the bias toward overestimation of both benefits and cost forecasts. The overestimation of the benefits is compensated for by an overestimation of the cost. This illustrates that increasing control over the cost without measures to ensure the quality of the benefits, may yield a lower forecasting quality of the overall NPV.

Finally, we will illustrate how it is possible to use the quantified forecasting quality further to enhance decision information. This is done by describing two basic simulation examples. The first example shows how to acquire additional information for rationing the capital budget over various project proposals. The second example provides insight in the forecasted business value that will be generated from project proposals when accounting for the forecasting quality and bias.

1.0.0.1. Generalized method. This article is self-contained, but also forms the final article in a triptych. To assess the forecasting quality of the NPV in this article, we make use of an existing method to quantify and visualize the quality of IT-forecasts [14]. That first article or left pane covered a method that assumes positive values, like budgets, durations, and size in function points or lines of code.

That research led to a second article or middle pane in IEEE Software [15] that allowed us to question existing research on the forecasting quality of important key performance indicators for IT-projects (cost, time, amount of functionality). In particular, we were able to question the validity of the Standish Chaos Report Figures.

This third and final article, the right pane, covers the case for forecasts and actuals that can take any value. By applying the method of our first article to real-world data of NPV calculations, it turned out the model was inadequate to deal with zero’s and negative values. Indeed, it is natural for an NPV to be zero valued or negative.

Next to that, the actuals of all entities are only known with certainty after some time. For instance, the actual project cost are only known after the project is completed. Or, the actual NPV of an asset is known after the economic life span of an investment, for instance, 5 years after the initial forecast. However, to support decision making, we prefer recent information about the current forecasting practice; something that the quality of 5-year old forecasts will not provide for. Moreover, in case of some entities, for instance, the NPV, it is possible the actual will never be objectively measured. Therefore, we cannot always compare forecasts with actuals, making other reference points necessary. For these reasons, we need adaptations of the forecast assessment method of Eveleens et al. [14]. Note that the existing method is perfectly fit for positively valued indicators, so that method is not at all outdated by this article. But for business value a more sophisticated method is necessary.

Therefore, to assess the quality of NPV forecasts, in this article we extend and generalize the existing method that is intuitive for positively valued forecasts and actuals. We will develop a method that incorporates other reference points than the actual and allows for negative valued reference points or forecasts. Moreover, we resolve visual limitations in asymptotic cases.

At first sight, these issues appear trivial, but they are not. Adjusting the model to accommodate for these issues, entails an extensive and detailed discussion. For readers interested in the use of the model rather than the elaborate discussion of the adjustments, it is possible to skim the generalization of the method.

1.0.0.2. Data. In this article, we will make use of extensive data of four organizations. In total, we obtained data of 1620 IT assets with an investment cost of 1232

+

million Euro for which 6513 forecasts were made.

Of a large financial service provider, Y, we use data consisting of 667 forecasts of 140 project costs and 83 functionality forecasts of 83 assets. A multinational organization, X, provided us with 3767 forecasts for 867 project costs. From Landmark Graphics, LGC, we obtained data containing 923 forecasts of 121 project durations. Finally, a large telecommunication organization Z provided 971 forecasts made for 307 project costs.

For the purpose of evaluating the forecasting quality of IT business value, we analyze another data set from the large telecommunication organization, Z. This data consists of 102 NPV forecasts made for 102 IT assets that together represent discounted benefits of 4714 million Euro and an investment value of 173 million Euro.

1.0.0.3. Related work. In statistical mathematics, assessing the quality of estimation methods is a well-discussed topic [16]. There are well-defined criteria that determine the quality of these methods. The generalized method we will develop makes use of such criteria and does not provide new statistical ways to determine forecasting quality. The generalized method is a conservative extension of the existing method, as it is in parts identical, yet extends the possibilities.

However, the statistical methods and metrics are often not accessible to IT governors. Therefore, in this article, we discuss how to present, summarize and visualize the IT forecasting accuracy in such a way that executives are able to assess their quality and use it to enhance decision information. With the generalized method we aim to make quantifying the forecasting quality more readily available to governors. Furthermore, the method allows executives to acquire knowledge on the forecasting quality of their organization.

IT forecasting methods and their accuracy are frequently discussed in the literature to achieve correct forecasting of project proposals. For example, many books [4,27,42,9,8,34] have been written describing issues and guidelines to achieve accurate estimates. Moreover, numerous estimation tools exist and are used in practice, among others COCOMO [4], SLIM [49], SEER, SPQR/20 and KnowledgePlan [28]. These tools assist in forecasting relevant project values, such as cost, effort and durations.

Numerous articles [6,30,32,54] compare different estimation methods to determine which of them are most accurate under certain circumstances. An article by Eveleens et al. [14] proposes a method to quantify the forecasting quality by assessing the accuracy and potential bias of predictions.

Yet, these articles do not address business value forecasts, but forecasts such as cost, size, functionality or duration. We are unaware of articles that quantify the quality of business value forecasts of IT investments. A book by Bower [35] did quantify the quality of NPV predictions for 50 assets in another industry.

In this article, we analyze forecasted and re-estimated NPVs of 102 IT assets to assess the accuracy of the initial forecasts of IT business value. We will compare our case study to the one described by Bower.

1.0.0.4. Organization of this article. In Section2, we introduce terms and notations that will be used throughout the article. In Section3we discuss economic indicators that are used to quantify the business value of project proposals. More particularly, we cover in detail the well-known Net Present Value. For the interested reader, in Section4we extend an existing method to assess IT forecasting quality and make it fit to assess economic indicators. We generalize the method to allow for different reference points and negatively valued entities. We assess the impact of the generalization using data of four organizations. Section5explores the real-world NPV data we obtained from the telecommunication organization, Z. We compare this data to benchmark data from the literature. We also investigate the data for possible heterogeneity. After the exploration, we commence with the assessment of the forecasting quality of the NPV, benefits and cost in Section6. Moreover, we perform a sensitivity analysis to investigate the impact on the quality of NPV forecasts when the forecasting quality of benefits or cost improve. Section7illustrates how the quantified forecast information is used to enhance decision information. This is illustrated by using two simulation examples. In Section8we discuss limitations of our research. Finally, Section9concludes the article.

2. Terminology

In this section, we introduce terms and notations that we use throughout the article. We recall some terms and notation that were elaborately discussed in another article [14]. Since we will make extensive use of these definitions and methods, for the sake of completeness, we summarize them here. Moreover, we introduce new terms that will often occur in this article. InFig. 1, some of the terms and their relations are depicted. Although most terms are used frequently in common English, the meaning and understanding of them differs among people. Therefore, we clearly state how we comprehend them.

2.0.0.5. Asset. An asset is defined by the Oxford dictionary [55] as an item of property owned by a person or company, regarded as having value and available to meet debts, commitments, or legacies. In this article, we assume that projects need to be executed to create these assets. Even if a project modifies an existing asset, we will consider the altered asset as being a new asset. The assets can be both tangible and non-tangible.

2.0.0.6. Entity. In this article, we will use the wordentityto denote any quantifiable aspect of an asset that is of interest. For instance, entities that we will consider are the project cost or benefits. Other examples are economic indicators, such as the Net Present Value or Internal Rate of Return.

asset life span asset cost (A=C+I)

Start of project End of project _{End of asset's}

economical life span

x x

Initial forecast date Re-estimation date

benefits (B) asset usage cost (C) project cost / investment cost (I)

project duration

ts tf te tr ta

Fig. 1.Relevant terms in the life span of an IT asset.

2.0.0.7. Asset usage cost and project cost. We will use the term cost in different ways. The total asset cost, denoted byAis the full range of cost that is made for an asset. These cost consist of the project cost, denoted byIfor investment cost, and the asset usage cost, denoted byC. We will consider project cost to constitute merely the cost for executing the project. The asset usage cost are all cost excluding the project cost. This entails, for instance, marketing, network usage and maintenance cost. Note, that in this article we will only discuss cost that are computed in their present value. When we refer toA,IorC, we refer to the discounted asset cost, discounted project cost or discounted asset usage cost.

We make a distinction in cost as most organizations try to manage, control and contain project cost. This is done mainly due to the relative ease with which the project cost can be measured. In this article, we investigated to which extent control of this subset of the cost is useful in the context of the entire benefits and cost of an asset.

Note that for this article, it is not important how the asset cost are derived. We consider the asset usage cost as given. We will not question how the cost are computed, what precisely should be quantified and how it should be quantified. We assume that the estimators within an organization use equal definitions of how and what to incorporate in determining the cost.

2.0.0.8. Benefit. In the Oxford dictionary [55], a benefit is defined as an advantage or profit gained from something. In the context of this article, we consider a benefit, denoted byB, as aquantified monetaryadvantage or profit gained from something. Thus, we are only discussing benefits that have been quantified and represent a monetary gain.

Note, that in this article we will only discuss benefits that are computed in their present value. When we refer toB, we refer to the discounted benefits.

In this article, we will not dive into the question how the benefits have been quantified. Similar to the asset cost, we do not consider what precisely should be quantified and how it should be quantified. We assume that estimators within an organization apply the same definitions to compute the benefits.

2.0.0.9. Cash flow. In this article, a cash flowCFof a certain time periodpis equal to the benefitsBminus the asset costAin that period. Discounting the cash flows of all time periods to the present time and summing them, leads to the Net Present Value, which will be discussed later on.

2.0.0.10. Duration and life span. As with cost, there are a number of durations we will consider. The different durations are displayed inFig. 1. The first is the project duration, given byte

−

ts. The second duration that is of interest is the economical life span of an asset, given byta

−

ts. We define this life span of an asset as the period over which benefits and asset cost are forecasted.

2.0.0.11. Forecast. We define a forecast (or forecasting) of a certain entity as the prediction of the value that entity will have in the future. A forecast consists of the ex-post and the ex-ante part. The ex-post part is the part of the forecast that is already known—it is what has been done thus far. The ex-ante part is a prediction of what lies in the future.

Since we will address forecasts of different entities, we introduce a notation for forecasts. When we discuss forecasts, we will denote forecastfof entityeasf_e. If there is no ambiguity about the entity in question in a particular paragraph or it is of no relevance, we will simply usef.

2.0.0.12. Point forecast. The forecasts we discuss in this article are point forecasts. A point forecast is a single prediction that is often a summary of a large range of possible outcomes.

When a prediction is made, an estimator considers multiple scenarios that may occur and all relevant risks for the entity in question. For example, consider an estimator that needs to estimate the business value of an IT asset. The estimator should consider the risk that the project required to create the IT asset gets canceled. A study by Capers Jones found that of software applications in the 10

.

000 function point size range, about 36% are canceled and never completed [28]. The cancelation of a project significantly impacts the project outcome up to a swap to negative business value. This risk should be accounted for in the range of possible outcomes when forecasting the business value early on.

Table 1

Ten potential risks that can influence the outcome of software projects [29].

Potential risk

Risk of dilution of ownership due to multiple funding rounds Risks of difficult data migration from legacy applications Risk of significant layoffs of project team

Risk of inadequate warranties for quality and security Risk of security flaws in application

Risk of late start in deploying risk solutions

Risk of estimates being rejected due to lack of benchmarks Risk of software raising hardware warranty costs Risks from disconnected ‘‘stove pipe’’ applications Risk that requirements are not kept updated after release

Another example is the possibility of litigation when the asset is developed under a contract. In 2001, Capers Jones and his colleagues at SPR observed that 5% of projects within the United States that were outsourced, were probable to result in litigation or had litigation in progress [26]. Capers Jones states that an average lawsuit in the US costs both the plaintiff and the defendant so much money that all applications ending up in court have negative values. If an IT application is going to be developed under contract, a formal risk assessment is needed plus very strong contracts with penalties for non performance. If one third of large applications are canceled, and 5% of outsourced projects may result in litigation, the CIO needs more certainty than exists today that applications receiving funds will be developed using best practices. This implies that an early risk analysis should be part of the funding equation.

A risk analysis considers the likelihood the risk will occur and its impact on the entity to be forecasted. Through personal communication from Capers Jones, we received a list containing 200 potential risks that can influence the outcome of software projects [29].Table 1describes 10 risks from that list.

These scenarios and their chance of occurrence lead to a range of possible outcomes. This range or interval of possibilities is the prediction of the value of interest and provides information on the risks related to the project. The interval allows the management to set adequate targets and make commitments based on their risk averseness or appetite.

However, in practice this interval is rarely given to the management. In many cases, the interval is summarized to a single point forecast, for instance, the most likely scenario to occur. As the management is confronted with these point forecasts, we assess their quality in this article. Next to that, we discuss ways to recreate the interval based on historic point forecasts in Section7.

2.0.0.13. Actual. We define an actual of a certain entity to be the final realization of that entity. That is, the actual is the true value of that which has been forecasted. The notation that we will use for actuals is the same as with the forecasts,aewith aan actual of entitye. Again, we also use the shorter versionawhen it is clear which entity is referred to.

We make two assumptions about the actual. First, we assume that IT governors want estimators to provide a prediction of the final realization. That is, the executives are interested in the true value. Second, we assume the actual is objectively measurable and thus that manipulation of the final realization is not possible.

2.0.0.14. Reference point. We define a reference point as the reference to which we compare the value of forecasts. An example of a reference point is the actual itself. In this article, we also use other reference points. For instance, we will use re-estimations of the benefits and the asset usage cost made a year after project completion as reference point. In Section4, we will discuss the implications of using different reference points.

2.0.0.15. Bias. A bias is defined by the Oxford dictionary [55] as a systematic distortion of a statistical result due to a factor not allowed for in its derivation. Since we assume that the statistical result or forecast should predict the reference point, any systematic distortion, for instance, general overestimation, is considered a bias. There are several reasons why systematic distortions occur, both consciously and unconsciously. For instance, if an estimator wants to present an IT project proposal positively in order to get it approved, the estimator may underestimate the cost or duration of the project. Or, with a forecasting tool, parameter settings can unintentionally be inadequately set. In this article, we will not elaborate on the reasons for biases.

2.0.0.16. Forecast to actual ratio. A forecast to actual ratio orf

/

aratio is a measure to assess the quality of forecasting. In the IT-context, this ratio was introduced by Barry Boehm [4]. It measures the forecasting quality by dividing the forecast by its actual. We will analyzef

/

aratios of several entities. When necessary, we denotefe

/

aealso as

(

f

/

a

)

efor forecastf and actualaof entitye.

2.0.0.17. Forecast to actual plot. To assess the quality of forecasts, we plot thef

/

aratios in what is known as the forecast to actual plot, orf

/

aplot. Thef

/

aplot depictsf

/

aratios against the relative time at which the forecasts are made. The

0.0 0.2 0.4 0.6 0.8 1.0 project completion forecast / actual 1 overpessimistic pessimistic unbiased overperfect optimistic overoptimistic

Fig. 2.Typical patterns in anf/aplot.

We recallFig. 2[14] that depicts several typicalf

/

apatterns. The horizontal axis of the figure depicts the percentage of project completion at which the forecast is made. The vertical axis shows thef

/

aratios on a logarithmic scale.

In the figure, a number of patterns are illustrated that may be found when plottingf

/

aratios. If forecasts mostly overestimate the actual, thef

/

aplot will reveal the optimistic or over-optimistic pattern, or variations thereof. In case forecasts are aimed at predicting the actual value, one will find thef

/

aratios equally above and below the value 1, indicating the unbiased pattern. If one finds many forecasts extremely close to 1, the data may be manipulated or fixed price agreements are present, leading to the overperfect pattern. When forecasts are mostly underestimated, thef

/

aplot will resemble the pessimistic or overpessimistic pattern. These patterns illustrate the usefulness of thef

/

aplot as it allows to obtain an impression of potential biases in the forecasts made.

Note that the naming conventions of the different patterns are ambiguous. For instance, for entities as project cost or project durations, a forecast larger than the actual is a pessimistic forecast. That is, the forecast is a pessimistic projection of what really happened. However, in case of entities such as benefits, a forecast larger than the actual is an optimistic forecast. Although the labels are ambiguous, the patterns remain the same.

2.0.0.18. Estimating Quality Factor. Thef

/

aplot provides a means to distinguish potential biases in the forecasts made. Yet, it does not allow for quantifying and benchmarking their quality with others. To this end, we use the Estimating Quality Factor, or EQF, developed by Tom DeMarco [11]. The EQF is a measure of the deviation between the forecast and actual. It is computed with the following equation.

EQF

=

Area under actual value

Area between forecast and actual value

=



ta ts a dt



ta ts

|a

−

e

(

t

)

|

dt (1)

=



ta ts 1dt



ta ts

|

1

−

e

(

t

)/

a|dt

.

(2)

In this formula,ais the actual value,tsthe start date of the asset,tathe end date of the asset ande

(

t

)

the value of the forecast at timet(ts

≤

t

≤

ta). In Section4, we will generalize the EQF for other reference points than the actual and discuss the impact of this generalization.

An assumption is thate

(

t

)

is known for the range

[t

s

,

ta

]

. That is, we know at all times what the value of the most recent forecast is. However, in some circumstances the initial forecast is not made at the beginning. In this case, we assume the first forecast made at time

v

is actually made at the start. Mathematically, this means we assume thate

(

t

)

on range

[t

s

, v)

equalse

(v)

.

In statistics, other measures are known that quantify the quality of forecasts, such as the MSE, MAPE and MRE [10,16,24]. It is possible to use these measures instead of the EQF. However, the benefit of the EQF is that it is defined as a time-weighted average deviation to the actual. In our analyses, we assess the forecasting quality of assets that can have multiple forecasts. For decision making, it is important for these forecasts to be as quickly as accurately as possible. Therefore, it is important to account for the timing of subsequent forecasts. The EQF is defined to incorporate this effect.

2.0.0.19. Reference cone. The reference cone is a tool that comparesf

/

aratios against a benchmark forecasting quality. Given certain assumptions it is possible to compute lines that represent this forecasting quality. Consider the following assumptions.

– Ex-post inclusion: We assume that each consecutive forecast incorporates the ex-post part and we assume this part is known with certainty.

– Ex-post growth: The growth of the ex-post part is assumed to be described by a constant function.

– Ex-ante accuracy: The accuracy of the ex-ante part is assumed to remain constant as the project progresses.

– Goal: The goal of the forecast is to predict without bias and quickly as accurately as possible the actual value of interest for the project.

Recall that the ex-post part is the part of the forecast that is what has been done thus far. The ex-ante part is the prediction of what still lies in the future. The above assumptions are captured, assuming infinitely many forecasts are made, in the following formulas taken from an article by Eveleens et al. [14].

l

(

x

)

=

x

+



1

−

2 EQFl



·

(

1

−

x

)

(3) u

(

x

)

=

x

+



1

+

2 EQF_u



·

(

1

−

x

).

(4)

In this formula,xis the project’s progression relative to its project duration withx

=

(

tf

−

ts

)/(

ta

−

ts

)

∈ [

0

,

1

]

. The lower reference linel

(

x

)

is defined by the value EQFlthat represents the EQF quality of the lower line. The upper reference lineu

(

x

)

is given by the value EQFu, which determines the EQF quality of the upper line. These reference lines are defined under the assumption that EQF_l

≥

2 and EQF_u

>

0.

When we draw reference lines with these formulas, we use the notationc

(

l

,

u

)

for reference conecwith lower bound of EQF quality EQF_land a quality of EQF_ufor the upper bound. If EQF_l

=

EQF_uwe also use the shorter notationc

(

l

)

.

Using these terms and notations, we commence with our investigation of the forecasting quality of IT asset business value.

3. Asset business value

Pisello et al. [48] state that the Chief Information Officer (CIO) must become the Chief Financial Officer (CFO) of IT. To do so, the CIO must manage IT projects in such a way that their business value is maximized. But how to determine the business value of IT assets?

To specify the business value of IT assets, in their decision making executives are provided with information about the advantages, disadvantages and risks of each project proposal. Project proposals contain two kinds of information: qualitative and quantitative. Qualitative arguments are, for instance, corporate social responsibility or strategic alignment. According to multiple surveys [2,17,47], many organizations consider these arguments as important criteria for project selection.

The contribution of this article is to the quantitative kind of information. This part of the available information consists, among others, of forecasts of quantified benefits and cost. Often, these predictions are summarized using economic indicators. Well-known examples of such indicators are the Net Present Value, the Internal Rate of Return, the Return on Investment, and the Payback Period. We describe these indicators briefly below.

– Net Present Value (NPV). The Net Present Value is a summation of the predicted monetary benefits and cost of a project discounted to current value. If the NPV is positive this indicates the project is estimated to provide for monetary gain given the discount rate. If the NPV is zero, the project is a neutral investment: it generates enough discounted benefits to cover the discounted cost. If the NPV is negative the project proposal is expected to result in a monetary loss given the rate used. In the next subsection we will discuss the NPV in more detail.

– Internal Rate of Return (IRR). The Internal Rate of Return is the discount rate for which the NPV is equal to 0. The IRR is compared to the rate we would receive for similar investments, which is also known as the opportunity cost of holding capital. If the IRR is lower than the opportunity cost of capital, the investment should not be funded from a quantitative point of view. If the IRR is high compared to other rates, the project will generate more yield than other similar projects in the market.

– Return on Investment (ROI). In a book by Bierman [21], the Return on Investment is defined as an average income after depreciation divided by its investment. The ratio is also known as rate of return and there are many different ways of computing it. In all cases the higher the ratio the more profitable the project.

– Payback Period (PBP). The Payback Period is the amount of time needed after project completion to generate an equal amount of cumulated cash flows to cover the initial investment. Often these calculations are not discounted to today’s currency. The shorter the Payback Period, the sooner the initial investment is paid back.

These economic indicators give IT executives an indication of the business value of the project proposals. It is possible to use them to make predictions and to evaluate the final realization. Therefore, we are able to use these indicators to answer our question: how do we know that the quantitative business value forecasts are accurate, unbiased and reliable enough to support the decision making process?

We will answer this question by assessing the forecasting quality of one of the indicators, the NPV. We will only use the NPV, simply because our case study of organization Z uses this indicator to support its decision making. However, the generalized method we will propose later on, is applicable to the other indicators as well.

But, before answering the question, we need to better understand the NPV. First, we discuss the limitations of the different indicators with respect to each other. Each indicator has its theoretical and/or practical disadvantages. Then, we will identify the different components of the NPV and discuss each of them.

3.1. Indicator limitations

We briefly discussed the NPV, IRR, ROI and PBP. Brealey et al. [5] state that theoretically the use of the NPV leads to better investment decisions than these other well-known indicators. They argue that in some situations the IRR leads to different results than the NPV. We note that although both the IRR and NPV are derived from the same formula, the way they are derived can causes different outcomes. One situation in which the IRR is ineffective, is when there is a mixture of multiple positive and negative cash flows. For instance, consider an asset that, besides the initial investment, has some final fee to be paid at the end of its life span. In that case it may occur that there are two realistic IRR ratios for which the NPV is 0. This is also observed by, among others, Lorie et al. [40]. Moreover, Brealey et al. discuss that the IRR does not discern between borrowing or lending money, and has problems when the opportunity cost of capital is different over several years.

Brealey et al. also discuss that the Return on Investment leads to worse decisions, since it does not account for the timing of the cash flows. Since the cash flows are generally averaged, the indicator places no importance on whether the cash flows are earned in the first year or the last year. Yet, in reality this is a crucial aspect of the investment decision.

Finally, Brealey et al. argue that the Payback Period ignores all cash flows generated after the initial investment is paid back. However, these cash flows can make an investment highly lucrative or not. Therefore, Brealey et al. suggest to use the NPV to justify investment decisions.

However, the NPV is also not without limitations. A practical disadvantage of the NPV is that it does not consider the scarceness of the available resources. For instance, an asset with a predicted NPV value of 100 Euro is considered a better investment than one with a predicted 80 Euro, even though the former asset may involve investment cost of 10 million Euro and the latter 0

.

1 million Euro. Note that this aspect is accounted for by the IRR as well as in the ROI indicator by dividing by the investment cost.

Another disadvantage of the NPV is that the determination of the discount rate, which is needed in the calculations, is difficult. Later on in this section, we discuss this discount rate in more detail. It is possible to derive organization-specific discount rates. However, determining the required project-specific discount rate is not trivial.

Moreover, one should consider that a discount rate that is applicable now, may not be applicable next month as is pointed out by Ross [51]. A project proposal with a negative NPV given the discount rate this month, can be a very interesting opportunity some time later. Ingersoll et al. [25] developed a method to account for the value of optionality with respect to the uncertainty of interest rates. An investment should only be made when the NPV is sufficiently positive to forego the option to delay the investment.

3.1.0.20. Indicators in practice. A few decades ago, numerous surveys [17,19,45,47,53] found that the NPV was not the method of choice of most Chief Financial Officers (CFO). In 2002, an article by Ryan et al. [52] shows that till 1996 studies generally indicated the method most used by organizations was the IRR. Both Ryan et al. [52] and Arnold et al. [2] find that only just after the year 2000, organizations have adopted the NPV as preferable indicator. The survey of Ryan et al. [52] found that 85% of the respondents of the survey indicated to use the NPV often.

Still, all these surveys indicate that the organizations frequently use multiple indicators to support decision making. Although theory suggests the NPV should suffice to make investment decisions, in practice a combination of the NPV and other common indicators are used. Ryan et al. [52] also found that there is a correlation between the capital budget and the use of NPV and IRR. The larger the budget the more likely the use of either one of these methods. The Payback Period is found to be more frequently used by organizations with smaller capital budgets. These surveys show that many economic indicators are used for decision making.

3.2. Net Present Value

In this section we elaborate on the Net Present Value. First, we show how to compute the NPV. Then, we discuss its components, assumptions and interpretation.

Informally stated, the NPV determines the monetary value an asset adds to an organization. The cornerstones of this economic indicator are the predicted benefits, cost and economic life span. Simply put, the NPV determines whether the benefits outweigh the cost, both of which are computed in todays worth. If the NPV is positive, it means the asset will generate value for the organization. If it is negative, creating the asset will result in an overall loss.

Formally, the NPV is described by Brealey et al. [5] as follows. DenoteCFpas the cash flow predicted for time periodp andrpthe discount rate of time periodp. LetNbe the total amount of time units that are used. Then, the NPV is calculated in the following way:

NPV

=

N

−

p=1 CFp

(

1

+

rp

)

p−1

.

Below we discuss the elements of the formula in more detail.

3.2.1. Discount rate

The discount rater_pis also known as the opportunity cost of capital. Often, the discount rate is chosen identical for each time unit. That is,r

=

rp,

∀p

.

The purpose of the discount rate is two-fold. First, it accounts for the time value of money. It is better to have 100 Euro today than it is to have 100 Euro tomorrow, since the former can be invested immediately to generate additional income. By discounting the future cash flows, we acknowledge this time value of money.

Second, any investment must be funded with capital. The providers of this capital require compensation for making their capital available. This is the cost of the capital that the organization intends to use. Any investment should aim to have a higher return than the cost of capital. If not, the organization would waste the capital of the investors. They would have done better to return the money to the investors and let them invest it otherwise.

But how to determine this cost of capital or discount rate? A number of books and articles [1,5,21,50,53] explain methods, among others the weighted average cost of capital or WACC, to find the discount rate. A survey [7] found that the WACC is used most often in practice. The WACC is organization-specific.

However, the discount rate used in the calculations of the NPV is investment-specific. Not all assets of an organization will have the same risk as the entire organization. Some assets will have higher risk and other assets will have lower risk. The organization-specific WACC should be changed to account for the particular asset risk. For instance, Dewan et al. [12] and Verhoef [56] suggest to increase the WACC in case of IT assets.

Determining this correction to the WACC is not a trivial task. Moreover, it is often difficult to establish which other assets are equivalently risky. It is therefore not surprising that the survey by Bruner et al. [7] found that many organizations do not adjust the WACC for individual investments. Petty et al. [45] contended that the use of sophisticated risk-adjustment techniques would be limited until risk can be measured more precisely and one can show the impact of additional risk upon the firm’s cost of capital.

3.2.1.1. Discount rate forecast. The discount rate that is used to compute a particular NPV, is based on an assessment of the risk of that investment. In most cases the risk involved is not objectively measurable and is thus only a prediction of the actual risk.

The accuracy of this forecasted discount rate directly impacts the forecasting accuracy of the NPV. However, in this article, we will not assess the forecasting quality of the discount rate. We consider the cost of capital as given and will not question its derivation. In our case study of organization Z, all calculated NPVs of a particular asset are based on the same discount rate.

3.2.2. Cash flow

Another crucial element of the NPV calculations are the forecasts of the cash flows. The cash flowsCFpin the equation are the expected cash flows for each time periodp. These predictions should account for the likelihood and impact of the risk of different scenarios on the benefits and cost, such as cost overrun, project failure and/or late delivery. These scenarios lead to a probability distribution of possible cash flows.

Surveys of Gitman et al. [19] and Fremgen [17] found that estimating the cash flows is considered the most critical stage of the capital budgeting process. In 1978, Schall et al. [53] surveyed that individual project risk is assessed by means of a probability distribution of cash flows by 25% of the respondents and another 10% using sensitivity analysis. In most cases, the risks were assessed implicitly. That is, the distribution is not made explicit, but is implicitly incorporated in the predictions of the cash flows by the estimator. In 2000, Arnold et al. [2] found that 94% of the organizations required a formal risk evaluation. This was done in 85% of the cases using sensitivity analysis, often in conjunction with a subjective assessment. A probability analysis was performed by 31% of the organizations.

In stark contrast are the findings in the information technology sector. In 2002, the Meta Group [20] surveyed that 84% of the organizations do not use business cases at all for their IT-projects or only for selected projects. The Kellogg School of Management [36] found that 68% do not track benefits. The numbers show that organizations have difficulty determining the benefits of IT assets, let alone to formally account for the risks in the forecasts.

3.2.2.1. Unbiased. A critical assumption of the cash flow predictions is that they are unbiased. However, forecasts of, for instance, duration and cost can be biased [3,4,14,46]. It is conceivable that the same applies to forecasts of cash flows. The decision to invest in a project is highly dependent on these forecasts. Those making the proposal and those with interests in executing it, may be inclined to overestimate the cash flows to make the proposal more appealing. It is therefore crucial to check whether the assumption of unbiased cash flow forecasts holds.

To investigate this assumption, we have to consider what the cash flow is composed of. A cash flow is the resultant of the projected benefits minus the forecasted cost. If we find the cash flows to be unbiased, this may imply both benefits and cost are unbiased. But it could also mean that they are both highly biased, but counter each other’s effect. Although the effect is overall the same, the latter situation is unwanted. In that case, it is mere luck and not good forecasting practice. Luck may change any instant, a good forecasting practice not. Therefore, to assess whether cash flows are unbiased, we should not only analyze the cash flows, but consider the components that it consists of.

To better illustrate the different components of the cash flow, we reformulate the above equation of the NPV. We define

bpto be the predicted benefits of periodp,cpthe predicted asset usage cost andipthe project cost, withCFp

=

bp

−

cp

−

ip. LetNbe the total number of periods in which the economical asset life span is divided andrpthe discount rate of time period p. Then, we write the NPV using the following equation:

NPV

=

N

−

p=1 bp

−

cp

−

ip

(

1

+

rp

)

p−1

=

N

−

p=1 bp

(

1

+

rp

)

p−1

−

N

−

p=1 cp

(

1

+

rp

)

p−1

−

N

−

p=1 ip

(

1

+

rp

)

p−1

=

B

−

C

−

I

.

(5)

In Formula (5),Bamounts to the cumulated discounted benefits,Cthe cumulated discounted asset usage cost andIthe cumulated discounted project cost. Recall that inFig. 1in Section2, we illustrated that the benefits and asset usage cost are not present during project execution. That is, the summation of the benefits and asset usage cost usually only have values over the intervalp

∈

y

,

y

+

1

, . . . ,

N, whereyis the period in which the end of the project,te, falls. The summation of project cost usually has values over the intervalp

∈

1

,

2

, . . . ,

y.

To assess whether the cash flows and NPV are unbiased, we have to investigate each of these components. We make a distinction between project cost and asset usage cost, since we wish to know whether the quality of their forecasts are different. Many organizations record and have insight in the project cost. However, these cost often amount to only a relative small portion of the entire asset cost. We want to see whether accurate forecasts of the project cost pays off, or that it is wise to put more effort in the correct prediction of the asset usage cost.

Note that it remains helpful to analyze the NPV directly. Assessing the quality of the forecasted NPVs shows the impact of the interactions of the individual elements. If we find both the benefits and the cost to be overestimated, we do not know their combined effect on the accuracy of the NPV. Therefore, a combination of the individual analyses and the overall quality of the NPV is most insightful. In this situation, the interdependences are contained in the NPV analysis and the individual analyses provide answers as to where the variance comes from.

In the case study of organization Z, we will assess the forecasting quality of the NPV. Above, we discussed that it is useful to increase the depth of the analysis by also investigating the components of the NPV, that is,B,CandI. It is possible to further increase the depth of the analysis by also considering the components ofB,CandI. For instance,Bis the sum ofbp over all time periodsp. Therefore, it is possible to investigate the forecasting quality of eachbpseparately.

The level of detail that is required depends on the goal of the analysis. In this article, the primary goal is the forecasting quality of the NPV. Moreover, we wish to determine whether control of the project cost is useful without sufficient control over the other elements. For these purposes, we will only analyze the NPV,B,C, andI.

3.2.3. Time

Besides the discount rate and the predicted cash flows, time is another variable in the equation. Time is captured in the predicted total number of time periods denoted byN. This total amount is a forecast of the asset’s economical life span. The time periods are commonly expressed in years, but can also be different, for instance, in months.

An accurate prediction of this variable is important for the resulting NPV. If the life span is too long, we may unjustly attribute additional benefits and cost to the asset. On the other hand, if the predicted life span is too short, we will ignore future cash flows of the asset in our calculations. For instance, suppose the initial forecast predicted the economic life span to be 5 years. It may occur that when the NPV is re-estimated, it is estimated the economic life span is 5

.

5 years. When in this half year a positive cash flow is generated, the initial NPV forecast will be underestimated. These cash flows can make the difference between a positive or negative NPV.

However, in this article, we will not assess the forecasting quality of the asset’s economic life span. We consider the life span,N, as given and will not question its derivation. In our case study of organization Z, the two estimated NPVs of a particular asset used the same economic life span. Moreover, all estimated NPVs of a particular asset are discounted to the same present moment.

3.2.4. Indirect influences

Besides the described elements in the NPV equation, there are other factors that influence the outcome of the NPV indirectly.

3.2.4.1. Indirect time effect. Apart from the direct impact, time also influences the NPV calculation in indirect ways. For example, consider an asset that is delivered three months later than expected. In the first three months no benefits or asset usage cost occur, making their forecasts overestimations. Moreover, all predicted cash flows will become different. Namely, because the cash flows occur later, they will be discounted differently. Also, the timing of the forecasts can be crucial, for example, due to growing competition on the market. Therefore, delays can severely influence the NPV, benefits

0.5 1.0 2.0 0.2 0.5 1.0 2.0 5.0 10.0

f/a ratio of initial functionality forecasts

f/a ratio of initial project cost forecasts

Fig. 3.No evidence of correlation between forecasting accuracy of initial project cost and functionality forecasts.

and cost, making their accuracy forecasting difficult. These inaccuracies in the forecasts are also discussed in an article by Peters et al. [44].

Moreover, in an article by Putnam [49] and a book by Boehm [4, p. 472] it was found that shortening the project duration beyond its optimum can significantly increase the project cost. Boehm observed a similar effect when the project duration was stretched beyond its optimum.

Therefore, the accuracy of the prediction of the project duration is reflected in the accuracy of the forecasts of benefits, asset usage cost and project cost. Any inaccuracy we find in the analysis of the other variables may partly be caused by the inaccurate forecast of the project duration.

3.2.4.2. Functionality. Although functionality is not mentioned in the formula of the NPV, it has an indirect impact on the variables in the equation. For instance, if a project is executed and delivers less functionality than anticipated, this may prevent certain forecasted benefits to materialize (solution underdelivery). Similarly, if the resulting IT program has more functionality, it is possible additional benefits are generated as a result. An increase in requirements is also known as requirements creep [27,8]. On the other hand, an increase in functionality may also result in higher project and asset usage cost.

These effects should be weighted in the overall forecasts of the benefits, asset usage cost and project cost. This was done, for instance, in an article by Peters et al. [44] by considering requirement creep scenarios. Due to these effects, it is possible the forecasts of functionality are correlated with the forecasts of the other components. For instance, if the functionality is underestimated, the benefits and cost may be underestimated as well.

To investigate a possible correlation between the accuracy of the project cost forecasts and the functionality forecasts, we analyzed data from a large financial organization Y. InFig. 3, we depicted the initial functionality, denoted byF,

(

f

/

a

)

Fratios against the initial project cost

(

f

/

a

)

Iratios of 55 projects of which we had all the relevant data. The functionality forecasts were measured using function point countings [13,18].

The correlation coefficient of the two data sets is

−

0

.

05. Since values of

−

1 or 1 represent perfect correlation and 0 depicts no correlation, the coefficient shows that there is no correlation. This result reveals that the relation between functionality and project cost may only be marginal with respect to their forecasting quality. That is, underestimating the functionality does not directly cause an underestimation of the project cost.

Note that Boehm’s cone of uncertainty [4] showed that the time at which the forecasts are made is relevant for their quality. Although the forecasts in our real-world data set of project cost and functionality are not made at the same time, the differences between the moments they are made are small. The median of the difference between the moment the project cost forecast was made and the functionality forecast was made, divided by their project duration is 0 and the average difference is 0

.

08. Therefore, there is no indication that the results of this analysis are influenced by the different timing of the forecasts.

3.2.4.3. Software quality. Software quality is highly relevant to value prediction of both the benefits and the cost. The benefits derived from an asset are based on the perceived value of the customer. If the software contains many defects, the customer may value the asset less, thereby generating less benefits.

Capers Jones [28] found that the US average for IT software quality is about 5.0 defects per function point combined with 85% defect removal efficiency. This results in delivery of about 0.75 bugs or defects per function point. Best in class IT software quality combines less than 3.0 defects per function point combined with more than 95% defect removal efficiency. This results in delivery of about 0.15 defects per function point.

Jones argues that increasing the defect removal efficiency from 85% to 95% saves money and shortens development schedules. This is supported by others [8]. Applications using state of the art software quality methods have development

cost about 15% lower than average projects, and maintenance cost about 55% lower than average projects. Total cost of ownership is about 40% lower than average projects [29]. Therefore, it is beneficial to produce assets with a relative high quality.

This illustrates that the forecasts of benefits and cost are influenced by the delivered software quality of the asset. However, in this article, we will not consider the software quality of the assets. In this article, we focus on assessing the resulting forecasting quality of the NPV, benefits and cost.

3.3. Summary

To support decision making, IT executives have access to numerous economic indicators that summarize the expected business value of project proposals. Despite its limitations, theory suggests that the NPV method is superior compared to other methods. However, our discussion of the indicator shows that the NPV is far from certain. Each of its components, needs to be predicted. Therefore, it is not surprising that in practice many organizations use multiple economic indicators to gain insight in the value of an asset proposal.

Like the NPV, any economic indicator is highly dependent on the accurate forecasting of its elements. There are numerous components that influence the final NPV either directly, such as the benefits or cost, or indirectly, such as project duration or functionality. Therefore, in this article, we assess the forecasting quality of the NPV, and its components: the benefits, asset usage cost and project cost.

In the next section, we discuss a method with which it is possible to assess the forecasting quality of the NPV and its components. Those not interested in the mathematical elaborations, may skip the next section and continue with Section5.

4. Generalized method

In this article, we wish to determine whether organizations are able to adequately assess the business value of project proposals. In the previous section, we discussed the NPV and other economic indicators, which represent the business value of assets. To assess whether organizations are capable to accurately forecast the business value of IT proposals, we need to investigate the forecasting quality of the NPV and its components. Recall that the Net Present Value is a summation of the predicted monetary benefits and cost of a project discounted to current value.

A method to assess forecasting quality is described in an article by Eveleens et al. [14]. As discussed for the sake of completeness in Section2, that method uses thef

/

aplot, the reference cone and the EQF. Recall that the EQF is a measure of the deviation between forecast and actual. The method is applicable for entities that are positively valued, that is,f

,

a

>

0. We wish to evaluate the quality of the forecasts of the NPV, benefits and cost in the same manner. However, if we want to do so, a number of problems arise.

4.0.0.4. Asymptotic behavior. The first problem is a visual complication in case of asymptotic behavior. With asymptotic behavior we refer to situations in which forecast and/or actual are zero. For project cost, usually the forecasts and actuals are greater than zero. In practice, project proposals that cost nothing or assets that cost nothing hardly ever occur. However, in the context of economic indicators, a forecast of zero is important. For instance, a NPV of zero is the turning point between an asset being yield or loss generating. Also, it is not unlikely to find assets that have no benefits. For example, consider an asset that was developed, yet on completion it turned out there was no longer a market for it.

Zero actual benefits cause visual problems for thef

/

aplot. Let us explain. Thef

/

aplot visualizes potential biases by plottingf

/

aratios on a logarithmic scale. If the actual is zero, the ratio becomes infinite, making the logarithm also infinite. Therefore, in thef

/

aplot this point cannot be visualized in a normal way.

This problem also arises if the forecast is zero. In this case, thef

/

aratio is zero. However, the logarithm of zero is not defined. Thus, in that case the ratio can also not be depicted in the plot.

Normally, not many of such zero forecasts will be made. In most cases, a forecast of zero indicates no forecast is made at all. If this is the case, it is best to remove these forecasts from the analysis all together, as they will not reveal information on the quality of the forecasting process. However, sometimes an entity is truly forecasted to be zero, or remains interesting for analysis in conjunction with other forecasts. For instance, consider the case of a forecast of benefits and no forecast made for the cost of the project. In that case, it may be interesting to incorporate the data point in the analysis, to assess the quality of the resulting NPV forecast.

Why is it a problem that thef

/

aplot does not visualize ratios with a forecast or actual of zero? If there are many such ratios in a data set, thef

/

aplot may point to a potential bias in the data that does not exist. For example, consider an extreme situation in which 51% of the data in a data set consists of zero actuals and the remainder of the forecasts are underestimations, that isf

/

a

<

1. Thef

/

aplot would only depict the latter half of the data, which will show a bias toward underestimation. In reality, there is no particular bias given the data, since 51% of the data, the zero actuals, are overestimations. Clearly, the ability of thef

/

aplot to detect biases is hampered by zero forecasts and/or actuals.

4.0.0.5. Reference point. A second technical problem is caused by the reference point with which we compare the quality of the forecasts. With thef

/

aratio this reference point is the actual. However, it is questionable whether the actual is useful to support decision making. For instance, suppose one finds that 2-year old project cost forecasts or 5-year-old forecasted benefits were generally overestimated. Although this sheds light on the forecasting quality of 2 or 5 years ago, in most cases

1 0 f/a x overestimation f > a, a > 0 underestimation a > f, a > 0 overestimation f > a, a < 0 2 –1

Fig. 4.Illustration of anf/aplot with a linear vertical axis, positive forecasts and allowing for positive and negative actuals.

this is hardly information that is useful to apply to today’s forecasts. By the time the actuals are known, the forecasting practice may already have been changed. To support decision making, we prefer more recent information about the current forecasting practice. Moreover, in case of, for instance, the NPV, the actual may never be objectively measured.

A solution is to re-estimate, for instance, the benefits before the end of the economic life span. For instance, it is possible to re-estimate the benefits a year after project completion. At that time, it is more clear to the estimator which of the many possible scenarios is unfolding. The ex-post part is considerably larger than in the previous forecasts and the ex-ante part becomes smaller and smaller. This way we are able to approximate the actual, which allows us to derive more recent forecast information that we are able to use for today’s project proposals.

However, in this case the value of the re-estimation is no longer objectively measurable. In fact, it is a forecast in itself. If we compare the forecasting quality of earlier forecasts with this approximation of the actual, we use a different reference point than the actual. We will investigate how the assessment of forecasting quality is affected by such alternative reference points.

4.0.0.6. Negative values. Finally, the NPV can be both positive and negative. In practice, most negative NPVs will occur in re-estimations. For example, consider an asset with a positive forecasted NPV that grossly overestimated the benefits. Afterward, the asset was recomputed and the cost turned out to be greater than its benefits, resulting in a negative NPV.

It is also possible to have negative forecasted NPVs. For example, mandatory assets or assets with a negative NPV that are interesting for their qualitative features. Note that mandatory assets can have a positive value as well. If the asset would not be performed, the organization risks a fine or other sanctions, which potentially make such assets beneficial to undertake. Even in case of negatively forecasted NPVs, their forecasting quality remains interesting to investigate to contain the predicted losses.

The negative values cause two problems for thef

/

aplot. The first problem is that thef

/

aplot uses a logarithmic axis to depictf

/

aratios. However, the logarithm is not defined for negative values. Therefore, we are unable to depict them. We could abandon the logarithmic scale and use a linear axis, allowing us to depict negative ratios as well. However, a linear axis does not allow for easy distinction of biases.

Let us explain. Assume we have positive forecasts and both positive and negative actuals, that isf

>

0 anda

∈

_R. Suppose we would use a linear axis for thef

/

aplot and allow for negative actuals. In this case, all the negative ratios would be depicted below the linef

/

a

=

0. This situation is illustrated byFig. 4. These negative ratios indicate that, since forecast

f is positive, the forecast is larger than the actual and is thus an overestimation. For allf

/

aratios in the interval

[

0

,

1

]

, the

f

/

aratio indicates that the forecasts are smaller than their actuals, or equivalently, they are underestimations. Finally, the

f

/

aratios above the linef

/

a

=

1 depict forecasts that are larger than the actuals, which are again overestimations. Thus, when we use a linear axis and assume non-negative forecasts, and both positive and negative actuals, thef

/

aplot consists of three sections of which the middle one shows underestimations and the remainder overestimations.

A similar explanation applies when we assume non-negative actuals, and both positive and negative forecasts. Thus, such figures make it difficult to adequately distinguish biases, not to mention obtaining a visual impression of the quality of the forecasts.

The second problem of the negative values is that thef

/

aratio becomes ambiguous. For instance, consider anf

/

aratio of

−

1. Such a ratio is possible whenf

<

0

<

a, but also whena

<

0

<

f. Thus, a ratio of

−

1 can both mean the forecast is larger than the actual as well as the actual is larger than the forecast.

A similar problem arises with positivef

/

aratios. Consider anf

/

aratio of 2. This occurs whenf

>

awithf

,

a

>

0 and whenf

<

awithf

,

a

<

0. Thus, it is no longer possible to determine whether the forecast is smaller or larger than the actual value. To cope with these problems, we will propose other ratios than Boehm’sf

/

aratio that deal satisfactorily with negatively valued entities.

In the remainder of this section, we address each of these issues separately. This results in a generally applicable method to assess forecasting quality of entities that range over the entire real numbers.