validating the estimate

chAPter 15

It is important that cost estimators and organizations independent of the program office validate that all cost elements are credible and can be justified by acceptable estimating methods, adequate data, and detailed documentation. This crucial step ensures that a high-quality cost estimate is developed, presented, and defended to management. This process verifies that the cost estimate adequately reflects the program baseline and provides a reasonable estimate of how much it will cost to accomplish all tasks. It also confirms that the program cost estimate is traceable and accurate and reflects realistic assumptions. Validating the point estimate is considered a best practice. One reason for this is that independent cost estimators typically rely on historical data and therefore tend to estimate more realistic program schedules and costs for state-of-the-art technologies. Moreover, independent cost estimators are less likely to automatically accept unproven assumptions associated with anticipated savings. That is, they bring more objectivity to their analyses, resulting in estimates that are less optimistic and higher in cost. An independent view provides a reality check of the point estimate and helps reduce the odds that management will invest in an unrealistic program that is bound to fail.

t

C

e

C

’

b

p

v

e

OMB’s Circular No. A-94 and best practices established by professional cost analysis organizations, such as SCEA, identify four characteristics of a high-quality, reliable cost estimate.59 _{It is well-documented,} comprehensive, accurate, and credible.

By well documented is meant that an estimate is thoroughly documented, including source data and significance, clearly detailed calculations and results, and explanations of why particular methods and references were chosen. Data can be traced to their source documents.

An estimate is comprehensive if it has enough detail to ensure that cost elements are neither omitted nor double counted. All cost-influencing ground rules and assumptions are detailed in the estimate’s documentation.

An estimate that is accurate is unbiased, the work is not overly conservative or overly optimistic, and is based on an assessment of most likely costs. Few, if any, mathematical mistakes are present those that are are minor. As for credibility, any limitations of the analysis because of uncertainty or bias surrounding data or assumptions are discussed. Major assumptions are varied, and other outcomes are recomputed to determine how sensitive they are to changes in the assumptions. Risk and uncertainty analysis is 59 _{For the OMB guidelines, see Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, Circular No. A-94}

(Washington, D.C.: Oct. 29, 1992), and Director, OMB, “2009 Discount Rates for OMB Circular No. A-94,” memorandum for the heads of departments and agencies, Executive Office of the President, OMB, Washington, D.C., Dec. 12, 2008.

performed to determine the level of risk associated with the estimate. The estimate’s results are cross- checked, and an independent cost estimate (ICE) conducted by a group outside the acquiring organization is developed to determine whether other estimating methods produce similar results.

Table 25 shows how the 12 steps of a high-quality cost estimating process, described in table 2, can be mapped to these four characteristics of a high-quality, reliable cost estimate.

Table 25: The Twelve Steps of High-Quality Cost Estimating, Mapped to the Characteristics of a High- Quality Cost Estimate

Cost estimate characteristic Cost estimating step

Well documented

The estimate is thoroughly documented, including source data and significance, clearly detailed calculations and results, and explanations for choosing a particular method or reference

Data are traced back to the source documentation

■

Includes a technical baseline description

■

Documents all steps in developing the estimate so that a cost analyst

■

unfamiliar with the program can recreate it quickly with the same result

Documents all data sources for how the data were normalized

■

Describes in detail the estimating methodology and rationale used to

■

derive each WBS element’s cost

1. Define the estimate’s purpose 3. Define the program

5. Identify ground rules and assumptions

6. Obtain the data 10. Document the estimate 11. Present the estimate to

management

Comprehensive

The estimate’s level of detail ensures that cost elements are neither omitted nor double counted

Details all cost-influencing ground rules and assumptions

■

Defines the WBS and describes each element in a WBS dictionary

■

A major automated information system program may have only a

■

cost element structure

2. Develop the estimating plan 4. Determine the estimating

approach

Accurate

The estimate is unbiased, not overly conservative or overly optimistic, and based on an assessment of most likely costs

It has few, if any, mathematical mistakes; its mistakes are minor

■

It has been validated for errors like double counting and omitted

■

costs

Cost drivers have been cross-checked to see if results are similar

■

It is timely

■

It is updated to reflect changes in technical or program assumptions

■

and new phases or milestones

Estimates are replaced with EVM EAC and the independent EAC from

■

the integrated EVM system

7. Develop the point estimate and compare it to an independent cost estimate

12. Update the estimate to reflect actual costs and changes

Credible

Discusses any limitations of the analysis from uncertainty or biases surrounding data or assumptions

Major assumptions are varied and other outcomes recomputed to

■

determine their sensitivity to changes in assumptions

Risk and uncertainty analysis is performed to determine the level of

■

risk associated with the estimate

An independent cost estimate is developed to determine if other

■

estimating methods produce similar results

7. Develop the point estimate and compare it to an independent cost estimate

8. Conduct sensitivity analysis 9. Conduct risk and uncertainty

It is important that cost estimates be validated, because lessons learned have shown that cost estimates tend to be deficient in this area (see case study 41).

Case Study 41: Validating the Estimate, from Chemical Demilitarization,

GAO-07-240R

GAO reviewed and evaluated the cost analyses that the U.S. Army used to prepare its cost-benefit report on the DuPont plan of treatment and disposal options for the VX nerve agent stockpile at the Newport, Indiana, depot. GAO also interviewed Army and contractor officials on the data and assumptions they had used to prepare their analyses. To determine the accuracy of the underlying data, GAO independently calculated values based on provided assumptions to compare with values in the supporting spreadsheets. GAO compared values from the supporting spreadsheets with summary data in the supporting posttreatment estimate report that the Shaw Environmental Group had prepared, Shaw being the contractor that helped perform the analysis for the U.S. Army Chemical Materials Agency report.

GAO found, based on OMB criteria and criteria approved by the cost estimating

community, that the underlying cost estimates in the Army’s report were not reliable and that the effect of this on the Army’s finding that the DuPont plan had “significant cost savings over the three considered alternatives” was uncertain. GAO’s finding of unreliable cost estimates included (1) the quantity and magnitude of errors, (2) quality control weaknesses, (3) questionable or inadequate supporting source data and documentation, and (4) the undetermined sensitivity of key assumptions. Neither the Army nor the contractor had a system for cross-checking costs, underlying assumptions, or technical parameters that went into the estimates.

Moreover, GAO determined that the results from the Army’s program risk analysis were unreliable because they had been generated from previously discussed, unreliable cost estimates and because the Army attributed no risk to potential permit, legal, or other challenges to the DuPont plan. It was unclear whether the program risks of other alternatives were understated or overstated.

Overall, GAO could not determine the cumulative effect of these problems on the outcome or results of the Army’s analysis, largely because GAO had no confidence in much of the supporting data, given these problems. Without reliable underlying cost estimates, the Army, the Congress, and the public could not have confidence that the most cost-effective solution had been selected.

GAO’s recommendations were that the Army conduct its cost-benefit analysis again, using best practices, so that its data and conclusions would be comprehensive, traceable, accurate, and credible; that it correct any technical and mathematical errors in the cost estimate; that it establish quality control and independent review processes to check data sources, calculations, and assumptions; and that it perform a sensitivity analysis of key assumptions.

GAO, Chemical Demilitarization: Actions Needed to Improve the Reliability of the Army’s Cost Comparison Analysis for Treatment and Disposal Options for Newport’s VX Hydrolysate, GAO-07-240R (Washington, D.C.: Jan. 6, 2007).

Too often, we have reported that program cost estimates are unrealistic and that, as a result, they cost more than originally promised. One way to avoid this predicament is to ensure that program cost estimates are both internally and externally validated—that is, that they are comprehensive,

well documented, accurate, and credible. This increases the confidence that an estimate is reasonable and as accurate as possible. A detailed review of these characteristics follows.

Determine That the Estimate Is Well Documented