Determine Program Cost Drivers and Associated Risks

In chapter 13, we noted that one of the benefits of a sensitivity analysis is a list of the program cost drivers. Since numerous risks can influence the estimate, they should be examined for their sources of uncertainty and potential effect, and they should be modeled to determine how they can affect the uncertainty of the cost estimate. For example, undefined or unknown technical information, uncertain economic conditions, unexpected schedule problems, requirements growth, security level changes, and political issues are

often encountered during a program’s acquisition. Each of these risks can negatively or positively affect a program’s cost. This means that uncertainty can cause the actual cost or schedule to differ from any current plan either in a positive or beneficial direction or in a negative or harmful direction. In addition, new technologies may be proposed that can fail outright, causing rework and unexpected cost growth. Risks are also associated with the estimating process itself. For instance, historical data from which to make a credible estimate can be lacking. When this happens, a cost estimator has no choice but to extrapolate with existing methods or develop a new estimating approach. No matter the method, some error will be introduced into the estimate.

Accounting for all possible risks is necessary to adequately capture the uncertainty associated with a program’s point estimate. Far from exhaustive, table 21 describes some of the many sources of risk. It is only a starting point, since each program is unique.

Table 21: Potential Sources of Program Cost Estimate Uncertainty

Uncertainty Definition Example

Business or

economic Variations from change in business or economic assumptions

Changes in labor rate assumptions—e.g., wages, overhead, general and administrative cost—supplier viability, inflation indexes, multiyear savings assumptions, market conditions, and competitive environment for future procurements Cost estimating Variations in the cost estimate

despite a fixed configuration baseline

Errors in historical data and cost estimating relationships, variation associated with input parameters, errors with analogies and data limitations, data extrapolation errors, optimistic learning and rate curve assumptions, using the wrong estimating technique, omission or lack of data, misinterpretation of data, incorrect escalation factors, overoptimism in contractor capabilities, optimistic savings associated with new ways of doing business, inadequate time to develop a cost estimate

Program Risks outside the program

office control Program decisions made at higher levels of authority, indirect events that adversely affect a program, directed funding cuts, multiple contractor teams, conflicting schedules and workload, lack of resources, organizational interface issues, lack of user input when developing

requirements, personnel management issues, organization’s ability to accept change, other program dependencies Requirements Variations in the cost estimate

caused by change in the configuration baseline from unforeseen design shifts

Changes in system architecture (especially for system of systems programs), specifications, hardware and software requirements, deployment strategy, critical assumptions, program threat levels, procurement quantities, network security, data confidentiality

Schedule Any event that changes the schedule: stretching it out may increase funding requirements, delay delivery, and reduce mission benefits

Amount of concurrent development, changes in

configuration, delayed milestone approval, testing failures requiring rework, infeasible schedule with no margin, overly optimistic task durations, unnecessary activities, omission of critical reviews

Software Cost growth from overly optimistic assumptions about software development

Underestimated software sizing, overly optimistic software productivity, optimistic savings associated with using commercial off-the-shelf software, underestimated integration effort, lack of commercial software documentation, underestimating the amount of glue code needed, configuration changes required to support commercial software upgrades, changes in licensing fees, lack of support for older software versions, lack of interface specification, lack of software metrics, low staff capability with development language and platform, underestimating software defects

Technology Variations from problems associated with technology maturity or availability

Uncertainty associated with unproven technology, obsolete parts, optimistic hardware or software heritage assumptions, feasibility of producing large technology leaps, relying on lower reliability components, design errors or omissions

Collecting high-quality risk data is key to a successful analysis of risk. Often there are no historical data from which to derive the information needed as inputs to a risk analysis of cost or schedule. Usually most risk data are derived from in-depth interviews or in risk workshops. In other words, the data used in program risk analyses are often based on individuals’ expert judgment, which depends on the experience of the interviewees and may be biased. The success of data collection depends also on the risk maturity of the organization’s culture. It is difficult to collect useful risk analysis data when the organization is indifferent or even hostile to expressing risk in the program. Obtaining risk information from staff outside the acquisition program office can help balance potential optimism.

After identifying all possible risks, a cost estimator needs to define each one in a way that facilitates determining the probability of each risk occurring, along with the cost effect. To do this, the estimator needs to identify a range of values and their respective probabilities—either based on specific statistics or expressed as best case, worst case, and most likely—and the rationale for choosing the variability discussed. While the best practice is to rely on historical data, if these data are not available, how

qualitative judgment was applied should be explained (e.g., not planning for first time success in testing). Because the quality and availability of the data affect the cost estimate’s uncertainty, these should be well documented and understood. For example, a cost estimate based on detailed actual data in significant quantities will yield a more confident estimate than one based on an analogy using only a few data points. Since collecting all this information can be formidable, it should be done when the data are collected to develop the estimate. Interviews with experts familiar with the program are good sources of how varied the risks are for a particular cost element. However, experts do not always think in extremes. They tend instead to estimate probability ranges that represent only 60 percent to 85 percent of the possible outcomes, so adjustments may have to be made to consider a wider universe of risks. In addition, the technical baseline description should address the minimum and maximum range, as well as the most likely value for critical program parameters.

Several approaches, ranging from subjective judgment to complex statistical techniques, are available for dealing with uncertainty. Here we describe different ways of determining the uncertainty of a cost estimate.

Cost Growth Factor

Using the cost growth factor, the cost estimator reflects on assumptions and judgments from the development of the cost estimate and then makes a final adjustment to the estimate. This is usually a percentage increase, based on historical data from similar programs, or an adjustment solicited from expert opinion and based on experience. This yields a revised cost estimate that explicitly recognizes the existence of uncertainty. It can be applied at the total program level or for one or more WBS elements. The advantages of this approach are that it is easy to implement, takes little time to perform, and requires minimal detail. Its several problems are that it requires access to a credible historical database, the selection of comparative projects and adjustment factors that can be subjective, and new technologies or lessons learned that may cause historical data to be less relevant.

Expert Opinion

An independent panel of experts can be gathered to review, understand, and discuss the system and its costs, in order to quantify the estimate’s uncertainty and adjust the point estimate. This approach is often

used in conjunction with the Delphi technique, in which several experts provide opinions independently and anonymously. The results are summarized and returned to the experts, who are then given the opportunity to change or modify their opinions, based on the opinions of the group as a whole. If successful, after several such iterations, the expert opinions converge.

The strengths of this approach are directly related to the diversity and experience of the panel members. The major weaknesses are that it can be time consuming and experts can present biased opinions. For example, some of the largest risks facing a program may stem from a new technology for which there is little previous experience. If the risk distributions rest on the beliefs of the same experts who may be stakeholders, it could be difficult to truly capture the program risks. A typical rule of thumb is that lower and upper bounds estimated by experts should be interpreted as representing the 15 percent and 85 percent levels, respectively, of all possible outcomes. Therefore, the cost estimator will need to adjust the distribution bounds to account for skew (see step 2 for more on this issue). Cost estimators can also mitigate bias by avoiding leading questions and by questioning all assumptions to see if they are backed by historical data.

The analytic hierarchy process, like the Delphi technique, is another approach to making the best of expert opinion. It can be applied to the opinion of either an individual or a panel of experts and mitigates the problems of bias that result from group think or dominating personalities. The analytic hierarchy process provides a structured way to address complicated decisions: it relies on a framework for quantifying decision elements and evaluating various alternatives. This process allows for effective decision making because it captures both subjective and objective evaluation parameters, which can lead to less bias and help determine the best overall decision. The approach relies on mathematics to organize pair-wise comparisons of decision components and prioritizes the results to arrive at a stable outcome.

Mathematical Approaches

Mathematical approaches rely on statistics to describe the variance associated with an analogy or a cost estimating relationship. The most common approach is to collect data on the optimistic, most likely, and pessimistic range (the “3-point estimate”) for the risk or the cost element schedule activity duration. Statistics like the standard error of the estimate and confidence intervals are more difficult to collect from program participants and are not commonly used. Some distributions use more exotic inputs such as “shape parameters” that are often difficult to collect, even in the most in-depth interviews. Therefore, the 3-point estimate and an idea about the distribution shape can be used to define the probability distribution to be used in a simulation. Probability distributions are used either to characterize risks that are assigned to cost elements or activity durations or as estimates of uncertainty in costs or durations that may be affected by several risks. With either of these approaches, in the simulation the lower-level WBS element cost probabilities are combined to develop an overall program cost estimate probability distribution.

A benefit of this approach is that it complements the decomposition approach to cost estimating. In addition, the emergence of commercial software models means that Monte Carlo simulation can be implemented quickly and easily, once all the data have been collected. Some drawbacks to the approach include that input distributions can be various, correlation between cost elements needs to be included, and decision makers may not always accept the output. In addition, high-quality risk data are sometimes difficult and may be expensive to collect.

Technology Readiness Levels

NASA and the Air Force Space Command, among other organizations, address uncertainty by applying readiness levels, which capture the risk associated with developing state-of-the-art technology. They have historically developed technology readiness levels to indicate how close a given technology is to being available. Technology readiness levels are rated on a scale from 1 to 9, with 1 representing paper studies of a technology’s feasibility and 9 representing technology completely integrated into a finished product. In appendix XII, we list and describe nine technology readiness levels.

Knowing a technology’s readiness level allows a cost estimator to judge the risk inherent in assuming it will be available for a given program. For example, GAO has determined that level 7—demonstration of a prototype in an operational environment—is the level of technological maturity that constitutes low risk for starting a product development program. One needs to be cautious, since programs can inflate the level. There should be specific evidence that a program has achieved the claimed technology readiness level, such as that physical and functional interfaces are clearly defined, raw materials are available, and manufacturing procedures are set up and undergoing testing for proof of concept before accepting a claim as true.

Software Engineering Institute Maturity Models

SEI has developed a variety of models that provide a logical framework for assessing whether an organization has the necessary process discipline to repeat earlier successes on similar projects.

Organizations that do not satisfy the requirements for the “repeatable” level are by default judged to be at the initial level of maturity—meaning that their processes are ad hoc, sometimes even chaotic, with few of the processes defined and success dependent mainly on the heroic efforts of individuals. The lower the maturity, the higher the risk that a program will incur cost overruns.

In addition to evaluating software risks, SEI’s risk evaluation method can be tailored to address hardware and organizational risks with a program. This method includes identifying and quantifying risk using a repeatable process for eliciting risks from experts. Furthermore, using SEI’s taxonomy, the risk evaluation method provides a consistent framework for employing risk management methods and mitigation techniques.

Schedule Risk Analysis

Schedule risk analysis captures the risk that schedule durations may increase from technical challenges, lack of qualified personnel, and too few staff to do the work. Schedule risk analysis examines the effect of activities and events slipping on a program’s critical path or the longest path through the network schedule. A program schedule delay will have cost effects for all aspects of a program, including systems engineering and program management. It also analyzes how various activities affect one another because of precedence relationships—activity C cannot begin until activities A and B are finished—and how a slip in one activity affects the duration of other activities when concurrence is high among tasks. By applying probabilistic distributions to capture the uncertainty with traditional early start–late start and early finish– late finish schedule durations, using optimistic, pessimistic, and most likely values, a cost estimator can draw a better picture of the true critical path and any cost effects to the program. In addition, this analysis addresses the feasibility of the program plan as well as the effect of not meeting the anticipated finish date.

Risk Cube (Probability Impact Matrix) Method

The risk cube method prioritizes uncertainties that could jeopardize program cost, schedule, performance, and quality objectives in terms of probability of occurrence and cost effect. Subject matter experts,

typically engineers and others familiar with the program, define the risk factors, probabilities, and cost effect for each identified risk. Using these data, the cost estimator develops the expected cost overrun by multiplying the cost impact by each risk factor’s probability of occurrence. A common technique for engaging those knowledgeable about the program is creating a two-dimensional matrix like the one in figure 17.

Figure 17: A Risk Cube Two-Dimensional Matrix

Risk

Consequence The program penalty incurred if the objective is not obtained

Probability The likelihood that an objective will not be met if the current plan is used Source: GAO.

LO

W

MEDIUM

HIGH

In the risk cube (P-I matrix) method, risks are mapped onto the matrix, based on the severity of the consequence—ranging from low risk = 1 to high risk = 5—and the likelihood of their occurring— ranging from low likelihood = 1 to high = 5. Risks that fall in the upper right quadrant are the most likely to occur and have the greatest consequences for the program, compared to risks that fall into the lower left quadrant.

When risks are plotted together, management can quickly determine which ones have top priority. For a risk cube (P-I matrix) analysis to be accurate, complete lists of all risks are needed, as well as accurate probabilities of occurrence and cost impacts. Determining the cost impact will vary by program and WBS element, but a cost impact could, for example, be categorized as “60 percent more funding is required to resolve a technical shortfall that has no acceptable workarounds.” Once the cost impacts are identified, they are mapped to the appropriate WBS elements to help identify risk mitigation steps that would be most beneficial.

The advantages of using this approach are that those knowledgeable about the program can readily understand and relate to risks presented in this manner and that decision makers can understand the link between specific risks and consequences. A disadvantage is that engineers may not always know the

cost impacts and may not account for the full spectrum of possible outcomes. Moreover, this method can underestimate total risk by omitting the correlation between technical risk and level of effort in activities like program management.

Risk Scoring

Risk scoring quantifies and translates risks into cost impacts. Risk scoring is used to determine the amount of risk, preferably using an objective method in which the intervals between a score have meaning—a score of 1 = low risk, a score of 5 = medium risk, and a score of 10 = high risk. This method is used most often to determine technical risk associated with hardware and software. The following categories are used for hardware: technology advancement (level of maturity), engineering development (current stage of development), reliability (operating time without failure), producibility (ease to manufacture), alternative item (availability of back-up item), and schedule (amount of aggressiveness). Table 22 is an example of the hardware risk scoring matrix.55

Table 22: A Hardware Risk Scoring Matrix

Risk category

Risk score: 0 = low, 5 = medium, 10 = high

0 1–2 3–5 6–8 9–10

1. Technology

advancement Completed, state of the art Minimum advancement required Modest advancement required Significant advancement required New technology 2. Engineering

development Completed, fully tested Prototype Hardware and software development

Detailed design Concept defined 3.

Reliability Historically high for same system Historically high on similar systems

Modest problems

known Serious problems known Unknown 4. Producibility Production and

yield shown on same system Production and yield shown on similar system Production and

yield feasible Production feasible and yield problems No known production experience 5. Alternative item Exists or availability on other items not important Exists or availability on other items somewhat important Potential alternative in development Potential alternative in design Alternative does not exist and is required

6.

Schedule Easily achieved Achievable Somewhat challenging Challenging Very challenging