Sensitivity Analysis

chAPter 13

As a best practice, sensitivity analysis should be included in all cost estimates because it examines the effects of changing assumptions and ground rules. Since uncertainty cannot be avoided, it is necessary to identify the cost elements that represent the most risk and, if possible, cost estimators should quantify the risk. This can be done through both a sensitivity analysis and an uncertainty analysis (discussed in the next chapter).

Sensitivity analysis helps decision makers choose the alternative. For example, it could allow a program manager to determine how sensitive a program is to changes in gasoline prices and at what gasoline price a program alternative is no longer attractive. By using information from a sensitivity analysis, a program manager can take certain risk mitigation steps, such as assigning someone to monitor gasoline price changes, deploying more vehicles with smaller payloads, or decreasing the number of patrols. For a sensitivity analysis to be useful in making informed decisions, however, carefully assessing the underlying risks and supporting data is necessary. In addition, the sources of the variation should be well documented and traceable. Simply varying the cost drivers by applying a subjective plus or minus percentage is not useful and does not constitute a valid sensitivity analysis. This is the case when the subjective percentage does not have a valid basis or is not based on historical data.

In order for sensitivity analysis to reveal how the cost estimate is affected by a change in a single assumption, the cost estimator must examine the effect of changing one assumption or cost driver at a time while holding all other variables constant. By doing so, it is easier to understand which variable most affects the cost estimate. In some cases, a sensitivity analysis can be conducted to examine the effect of multiple assumptions changing in relation to a specific scenario.

Regardless of whether the analysis is performed on only one cost driver or several within a single scenario, the difference between sensitivity analysis and risk or uncertainty analysis is that sensitivity analysis tries to isolate the effects of changing one variable at a time, while risk or uncertainty analysis examines the effects of many variables changing all at once.

Typically performed on high-cost elements, sensitivity analysis examines how the cost estimate is affected by a change in a cost driver’s value. For example, it might evaluate how the number of maintenance staff varies with different assumptions about system reliability values or how system manufacturing labor and material costs vary in response to additional system weight growth.

Sensitivity analysis involves recalculating the cost estimate with different quantitative values for selected input values, or parameters, in order to compare the results with the original estimate. If a small change in the value of a cost element’s parameter or assumption yields a large change in the overall cost estimate,

the results are considered sensitive to that parameter or assumption. Therefore, a sensitivity analysis can provide helpful information for the system designer because it highlights elements that are cost sensitive. In this way, sensitivity analysis can be useful for identifying areas where more design research could result in less production cost or where increased performance could be implemented without substantially increasing cost. This type of analysis is typically called a what-if analysis and is often used for optimizing cost estimate parameters.

s

f

Uncertainty about the values of some, if not most, of the technical parameters is common early in a program’s design and development. Many assumptions made at the start of a program turn out to be inaccurate. Therefore, once the point estimate has been developed, it is important to determine how sensitive the total cost estimate is to changes in the cost drivers.

Some factors that are often varied in a sensitivity analysis are a shorter or longer economic life;

■

the volume, mix, or pattern of workload; ■

potential requirements changes; ■

configuration changes in hardware, software, or facilities; ■

alternative assumptions about program operations, fielding strategy, inflation rate, technology ■

heritage savings, and development time; higher or lower learning curves; ■

changes in performance characteristics; ■

testing requirements; ■

acquisition strategy, whether multiyear procurement, dual sourcing, or the like; ■

labor rates; ■

growth in software size or amount of software reuse; and ■

down-scoping the program. ■

These are just some examples of potential cost drivers. Many factors that should be tested are determined by the assumptions and performance characteristics outlined in the technical baseline description and GR&As. Therefore, auditors should look for a link between the technical baseline parameters and the GR&As to see if the cost estimator examined those that had the greatest effect on the overall sensitivity of the cost estimate.

In addition, the cost estimator should always include in a sensitivity analysis the assumptions that are most likely to change, such as an assumption that was made for lack of knowledge or one that is outside the control of the program office. Case study 38 shows some assumptions that can affect the cost of building a ship.

Case Study 38: Sensitivity Analysis, from Defense Acquisitions,

GAO-05-183

Given the uncertainties inherent in ship acquisitions, such as introducing new technologies and volatile overhead rates over time, cost analysts face a significant challenge in

developing credible initial cost estimates. The Navy must develop cost estimates as long as 10 years before ship construction begins, before many program details are known. Cost analysts therefore have to make a number of assumptions about ship parameters like weight, performance, and software and about market conditions, such as inflation rates, workforce attrition, and supplier base.

In the eight case study ships we examined, other unknowns led to uncertain estimates. Labor hour and material costs were based on data from previous ships and on unproven efficiencies in ship construction. GAO found that analysts often factored in savings based on expected efficiencies that never materialized. For example, they anticipated savings from implementing computer-assisted design and computer-assisted manufacturing for the San Antonio class transport LPD 17, but the contractor had not made the requisite research investments to achieve the proposed savings. Similar unproven or unsupported efficiencies were estimated for the Arleigh Burke class destroyer DDG 92 and Nimitz class aircraft carrier CVN 76. Changes in the shipbuilders’ supplier base also created uncertainties in their overhead costs.

Despite these uncertainties, the Navy did not test the validity of the cost analysts’ assumptions in estimating construction costs for the eight case study ships and did not identify a confidence level for estimates.

GAO, Defense Acquisitions: Improved Management Practices Could Help Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005).

s

p

s

A

A sensitivity analysis addresses some of the estimating uncertainty by testing discrete cases of assumptions and other factors that could change. By examining each assumption or factor independently, while holding all others constant, the cost estimator can evaluate the results to discover which assumptions or factors most influence the estimate. A sensitivity analysis also requires estimating the high and low uncertainty ranges for significant cost driver input factors. To determine what the key cost drivers are, a cost estimator needs to determine the percentage of total cost that each cost element represents. The major contributing variables within the highest percentage cost elements are the key cost drivers that should be varied in a sensitivity analysis. A credible sensitivity analysis typically has five steps:

identify key cost drivers, ground rules, and assumptions for sensitivity testing; 1.

reestimate the total cost by choosing one of these cost drivers to vary between two set amounts— 2.

for example, maximum and minimum or performance thresholds;49

document the results; 3.

repeat 2 and 3 until all factors identified in step 1 have been tested independently; 4.

evaluate the results to determine which drivers affect the cost estimate most. 5.

Sensitivity analysis also provides important information for economic analyses that can end in the choice of a different alternative from the original recommendation. This can happen because, like a cost estimate, an economic analysis is based on assumptions and constraints that may change. Thus, before choosing an alternative, it is essential to test how sensitive the ranking of alternatives is to changes in assumptions. In an economic analysis, sensitivity is determined by how much an assumption must change to result in an alternative that differs from the one recommended. For example, an assumption is considered sensitive if a 10–50 percent change yields a different alternative, very sensitive if the change is less than 10 percent. Assumptions and cost drivers that have the most effect on the cost estimate warrant further study to ensure that the best possible value is used for that parameter. If the cost estimate is found to be sensitive to several parameters, all the GR&As should be reviewed, to assure decision makers that sensitive parameters have been carefully investigated and the best possible values have been used in the final point estimate.

s

A

b

L

A sensitivity analysis provides a range of costs that span a best and worst case spread. In general, it is better for decision makers to know the range of potential costs that surround a point estimate and the reasons behind what drives that range than to just have a point estimate to make a decision from. Sensitivity analysis can provide a clear picture of both the high and low costs that can be expected, with discrete reasons for what drives them. Figure 14 shows how sensitivity analysis can give decision makers insight.

Figure 14: A Sensitivity Analysis That Creates a Range around a Point Estimate

$10 billion $11 billion $13 billion $9 billion $12 billion Life-cycle cost (FY 07 constant dollars) $10.040 $9.940 $10.090 $9.89 $11.099 $9.79 $11.121 $9.75 $12.789 $9.36 Increase the number of cost penalties in airframe development CER Double the development testing Increase airframe weight Eliminate concurrent production quantities Increase quality of materials in aircraft Use 88% learning curve instead of 91% Eliminate integration and assembly cost add-on Reduce airframe weight Improve aircraft maintainability Reduce peacetime flying hours +$40.0 million (0.4%) +$50.0 million (0.5%) +$1,009 million (10.0%) +$22.0 million (0.2%) +$1,668 million (15.0%)

-$60.0 million (0.6%) Increase estimate:

Decrease estimate: Description:

Description:

-$50.0 million (0.5%) -$100.0 million (1.0%) -$40.0 million (0.4%) -$390.0 million (4.0%) Increase in life-cycle cost estimate Decrease in life-cycle cost estimate Source: GAO. (Point estimate)

In figure 14, it is very apparent how certain assumptions affect the estimate. For example, increasing the quality of materials in the aircraft has the biggest effect on the highest cost estimate—adding $1,668 million to the point estimate—while reducing the number of flying hours is the biggest driver for reducing the cost estimate—reducing the flying hours saves $390 million. Using visuals like this can quickly display what-if analyses that can help management make informed decisions.

A sensitivity analysis also reveals critical assumptions and program cost drivers that most affect the results and can sometimes yield surprises. Therefore, the value of sensitivity analysis to decision makers lies in the additional information and understanding it brings to the final decision. Sensitivity analysis can also make for a more traceable estimate by providing ranges around the point estimate, accompanied by specific reasons for why the estimate could vary. This insight allows the cost estimator and program manager to further examine potential sources of risk and develop ways to mitigate them early. Sensitivity analysis permits decisions that influence the design, production, and operation of a system to focus on the elements that have the greatest effect on cost.

Sensitivity analysis examines only the effect of changing one assumption or factor at a time. But the risk of several assumptions or factors varying simultaneously, and its effect on the overall point estimate, should be understood. In the next chapter, we discuss risks and uncertainty analyses.

Best Practices Checklist: Sensitivity Analysis 10.

The cost estimate was accompanied by a sensitivity analysis that identified the …

effects of changing key cost driver assumption and factors.

Well-documented sources supported the assumption or factor ranges. 3

The sensitivity analysis was part of a quantitative risk assessment and not based 3

on arbitrary plus or minus percentages.

Cost-sensitive assumptions and factors were further examined to see whether 3

design changes should be implemented to mitigate risk.

Sensitivity analysis was used to create a range of best and worst case costs. 3

Assumptions and performance characteristics listed in the technical baseline 3

description and GR&As were tested for sensitivity, especially those least understood or at risk of changing.

Results were well documented and presented to management for decisions. 3

The following steps were taken during the sensitivity analysis: …

Key cost drivers were identified. 3

Cost elements representing the highest percentage of cost were determined 3

and their parameters and assumptions were examined.

The total cost was reestimated by varying each parameter between its 3

minimum and maximum range.

Results were documented and the reestimate was repeated for each parameter 3

that was a key cost driver.

Outcomes were evaluated for parameters most sensitive to change. 3

The sensitivity analysis provided a range of possible costs, a point estimate, and a …