developing them
chAPter 2
Cost estimates are necessary for government acquisition programs for many reasons: to support decisions about funding one program over another, to develop annual budget requests, to evaluate resource requirements at key decision points, and to develop performance measurement baselines. Moreover, having a realistic estimate of projected costs makes for effective resource allocation, and it increases the probability of a program’s success. Government programs, as identified here, include both in-house and contract efforts.
For capital acquisitions, OMB’s Capital Programming Guide helps agencies use funds wisely in achieving their missions and serving the public. The Capital Programming Guide stresses the need for agencies to develop processes for making investment decisions that deliver the right amount of funds to the right projects. It also highlights the need for agencies to identify risks associated with acquiring capital assets that can lead to cost overruns, schedule delays, and assets that fail to perform as expected.
OMB’s guide has made developing accurate life-cycle cost estimates a priority for agencies in properly managing their portfolios of capital assets that have an estimated life of 2 years or more. Examples of capital assets are land; structures such as office buildings, laboratories, dams, and power plants; equipment like motor vehicles, airplanes, ships, satellites, and information technology hardware; and intellectual property, including software.
Developing reliable cost estimates has been difficult for agencies across the federal government. Too often, programs cost more than expected and deliver results that do not satisfy all requirements. According to the 2002 President’s Management Agenda:
Everyone agrees that scarce federal resources should be allocated to programs and managers that deliver results. Yet in practice, this is seldom done because agencies rarely offer convincing accounts of the results their allocations will purchase. There is little reward, in budgets or in compensation, for running programs efficiently. And once money is allocated to a program, there is no requirement to revisit the question of whether the results obtained are solving problems the American people care about. 19
The need for reliable cost estimates is at the heart of two of the five governmentwide initiatives in that agenda: improved financial performance and budget and performance integration. These initiatives are 19 President George W. Bush, The President’s Management Agenda: Fiscal Year 2002 (Washington, D.C.: Executive Office of the
aimed at ensuring that federal financial systems produce accurate and timely information to support operating, budget, and policy decisions and that budgets are based on performance. With respect to these initiatives, President Bush called for changes to the budget process to better measure the real cost and performance of programs.
In response to the 2002 President’s Management Agenda, OMB’s Capital Programming Guide requires agencies to have a disciplined capital programming process that sets priorities between new and existing assets.20 It also requires agencies to perform risk management and develop cost estimates to improve the accuracy of cost, schedule, and performance management. These activities should help mitigate difficult challenges associated with asset management and acquisition. In addition, the Capital Programming Guide
requires an agency to develop a baseline assessment for each major program it plans to acquire. As part of this baseline, a full accounting of life-cycle cost estimates, including all direct and indirect costs for planning, procurement, operations and maintenance, and disposal, is expected.
The capital programming process, as promulgated in OMB’s Capital Programming Guide, outlines how agencies should use long-range planning and a disciplined budget process to effectively manage a portfolio of capital assets that achieves program goals with the least life-cycle costs and risks. It outlines three phases: (1) planning and budgeting, (2) acquisition, and (3) management in use, often referred to as operations and maintenance. For each phase, reliable cost estimates are necessary to establish realistic baselines from which to measure future progress.
Regarding the planning and budgeting phase, the federal budget process is a cyclical event. Each year in January or early February, the president submits budget proposals for the year that begins October 1. They include data for the most recently completed year, the current year, the budget year, and at least the 4 years following the budget year. The budget process has four phases:
executive budget formulation, 1.
congressional budget process, 2.
budget execution and control, and 3.
audit and evaluation. 4.
Budget cycles overlap—the formulation of one budget begins before action has been completed on the previous one. (Appendix IV gives an overview of the federal budget process, describing its phases and the major steps and time periods for each phase.)
For the acquisition and management in use phases, reliable cost estimates are also important for program approval and for the continued receipt of annual funding. However, cost estimating is difficult. To develop a sound cost estimate, estimators must possess a variety of skills and have access to high-quality data. Moreover, credible cost estimates take time to develop; they cannot be rushed. Their many
challenges increase the possibility that estimates will fall short of cost, schedule, and performance goals. If cost analysts recognize these challenges and plan for them early, this can help organizations mitigate these risks.
20 OMB first issued the Capital Programming Guide as a supplement to the 1997 version of Circular A-11, Part 3. We refer to the
C
oste
stimAtingC
hALLengesDeveloping a good cost estimate requires stable program requirements, access to detailed documentation and historical data, well-trained and experienced cost analysts, a risk and uncertainty analysis, the
identification of a range of confidence levels, and adequate contingency and management reserves.21 Even with the best of these circumstances, cost estimating is difficult. It requires both science and judgment. And, since answers are seldom if ever precise, the goal is to find a “reasonable” answer. However, the cost estimator typically faces many challenges. These challenges often lead to bad estimates—that is, estimates that contain poorly defined assumptions, have no supporting documentation, are accompanied by no comparisons to similar programs, are characterized by inadequate data collection and inappropriate estimating methodologies, are sustained by irrelevant or out-of-date data, provide no basis or rationale for the estimate, and can show no defined process for generating the estimate. Figure 2 illustrates some of the challenges a cost estimator faces and some of the ways to mitigate them.
Figure 2: Challenges Cost Estimators Typically Face
Program instability Adequate
budget Historicaldata available Well trained and experienced analysts Detailed documentation available Well
defined Risk analysisconducted Stable program Adequate cost reserve Unrealistic assumptions Unreliable data Inexperienced analyst Overoptimism Unreasonable program baselines Data not normalized Historical cost databases not available Obtaining data Cutting edge technology First-time integration Complex technology New processes Unrealistic projected savings Diminishing industrial base Source: GAO.
Some cost estimating challenges are widespread. Deriving high-quality cost estimates depends on the quality of, for example, historical databases. It is often not possible for the cost analyst to collect the kinds of data needed to develop cost estimating relationships (CER), analysis of development software cost, engineering build-up, and many other practices. In most cases, the better the data are, the better the resulting estimate will be. Since much of a cost analyst’s time is spent obtaining and normalizing data, experienced and well-trained cost analysts are necessary. Too often, individuals without these skills are thrown into performing a cost analysis to meet a pressing need (see case study 4). In addition, limited program resources (funds and time) often constrain broad participation in cost estimation processes and force the analyst (or cost team) to reduce the extent to which trade-off, sensitivity, and even uncertainty analyses are performed.
21 For our purposes in this Cost Guide, contingency reserve represents funds held at or above the government program office for
“unknown unknowns” that are outside a contractor’s control. In this context, contingency funding is added to an estimate to allow for items, conditions, or events for which the state, occurrence, or effect is uncertain and experience shows are likely to result in additional costs. Management reserve funds, in contrast, are for “known unknowns” that are tied to the contract’s scope and managed at the contractor level. Unlike contingency reserve, which is funding related, management reserve is budget related. The value of the contract includes these known unknowns in the budget base, and the contractor decides how much money to set aside. We recognize that other organizations may use the terms differently.
Case Study 4: Cost Analysts’ Skills, from NASA, GAO-04-642
GAO found that NASA’s efforts to improve its cost estimating processes were undermined by ineffective use of its limited number of cost estimating analysts. For example,
headquarters officials stated that as projects entered the formulation phase, they typically relied on program control and budget specialists—not cost analysts—to provide the financial services to manage projects. Yet budget specialists were generally responsible for obligating and spending funds—not for conducting cost analyses that underlay the budget or ensuring that budgets were based on reasonable cost estimates—and, therefore, they tended to assume that the budget was realistic.
GAO, NASA: Lack of Disciplined Cost-Estimating Processes Hinders Effective Program Management, GAO-04-642 (Washington, D.C.: May 28, 2004).
Many cost estimating challenges can be traced to overoptimism. Cost analysts typically develop their estimates from technical baselines that program offices provide. Since program technical baselines come with uncertainty, recognizing this uncertainty can help form a better understanding of where problems will occur in the execution phase. For example, if a program baseline states that its total source lines of code will be 100,000 but the eventual total is 200,000, the cost will be underestimated. Or if the baseline states that the new program will reuse 80,000 from a legacy system but can eventually reuse only 10,000, the cost will be underestimated. This is illustrated in case study 5.
Case Study 5: Recognizing Uncertainty, from Customs Service Modernization, GAO/AIMD-99-41
Software and systems development experts agree that early project estimates are
imprecise by definition and that their inherent imprecision decreases during a project’s life cycle as more information becomes known. The experts emphasize that to be useful, each cost estimate should indicate its degree of uncertainty, possibly as an estimated range or qualified by some factor of confidence. The U.S. Customs Service did not reveal the degree of uncertainty of its cost estimate for the Automated Commercial Environment (ACE) program to managers involved in investment decisions. For example, Customs did not disclose that it made the estimate before fully defining ACE functionality. Instead, Customs presented its $1.05 billion ACE life-cycle cost estimate as an unqualified point estimate. This suggests an element of precision that cannot exist for such an undefined system, and it obscures the investment risk remaining in the project.
GAO, Customs Service Modernization: Serious Management and Technical Weaknesses Must Be Corrected, GAO/AIMD-99-41 (Washington, D.C.: Feb. 26, 1999).
Program proponents often postulate the availability of a new technology, only to discover that it is not ready when needed and program costs have increased. Proponents also often make assumptions about the complexity or difficulty of new processes, such as first-time integration efforts, which may end up to be unrealistic. More time and effort lead directly to greater costs, as case study 6 demonstrates.
Case Study 6: Using Realistic Assumptions, from Space Acquisitions,
GAO-07-96
In five of six space system acquisition programs GAO reviewed, program officials and cost estimators assumed when cost estimates were developed that critical technologies would be mature and available. They made this assumption even though the programs had begun without complete understanding of how long they would run or how much it would cost to ensure that the technologies could work as intended. After the programs began, and as their development continued, the technology issues ended up being more complex than initially believed.
For example, for the National Polar-orbiting Operational Satellite System (NPOESS), DOD and the U.S. Department of Commerce committed funds for developing and producing satellites before the technology was mature. Only 1 of 14 critical technologies was mature at program initiation, and it was found that 1 technology was less mature after the contractor conducted more verification testing.
GAO found that the program was later beset by significant cost increases and schedule delays, partly because of technical problems such as the development of key sensors.
GAO, Space Acquisitions: DOD Needs to Take More Action to Address Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, D.C.: Nov. 17, 2006).
Collecting historical data and dedicating the time needed to do this continuously is another challenge facing cost estimators. Certain acquisition policy changes and pressured scheduling have had the unintended consequence of curtailing the generation of a great deal of historical data used for cost estimating. Outside of highly specific technology areas, it is often difficult for the cost analyst to collect the kinds of data needed to develop software cost estimates, valid CERs, and detailed engineering build-up estimates.
In addition, limited program resources in terms of both funds and time often constrain broad
participation in cost estimation processes and force the analyst or cost team to reduce the extent to which trade-off, sensitivity, and even uncertainty analyses are performed. Addressing these critical shortfalls is important and requires policy and cultural adjustments to fix.
Program stability presents another serious challenge to cost analysts. A risk to the program also arises when the contractor knows the program’s budget. The contractor is pressured into presenting a cost estimate that fits the budget instead of a realistic estimate. Budget decisions drive program schedules and procurement quantities. If development funding is reduced, the schedule can stretch and costs can increase; if production funding is reduced, the number of quantities to be bought will typically decrease, causing unit procurement costs to increase. For example, projected savings from initiatives such as multiyear procurement—contracting for purchase of supplies or services for more than one program year—may disappear, as can be seen in case study 7.
Case Study 7: Program Stability Issues, from Combating Nuclear Smuggling, GAO-06-389
According to officials of Customs and Border Protection (CBP) and the Pacific Northwest National Laboratory (PNNL), recurrent difficulties with project funding were the most important explanations of schedule delays. Specifically, according to Department of Homeland Security and PNNL officials, CBP had been chronically late in providing appropriated funds to PNNL, hindering its ability to meet program deployment goals. For example, PNNL did not receive its fiscal year 2005 funding until September 2005, the last month of the fiscal year. According to PNNL officials, because of this delay, some contracting activities in all deployment phases had had to be delayed or halted; the adverse effects on seaports were especially severe. For example, PNNL reported in August 2005 that site preparation work at 13 seaports had ceased because PNNL had not received its fiscal year 2005 funding allocation.
GAO, Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain,
GAO-06-389 (Washington, D.C.: Mar. 22, 2006).
Stability issues can also arise when expected funding is cut. For example, if budget pressures cause breaks in production, highly specialized vendors may no longer be available or may have to restructure their prices to cover their risks. When this happens, unexpected schedule delays and cost increases usually result. A quantity change, even if it does not result in a production break, is a stability issue that can increase costs by affecting workload. Case study 8, from a GAO report on Navy shipbuilding, illustrates this point.
Case Study 8: Program Stability Issues, from Defense Acquisitions,
GAO-05-183
Price increases contributed to growth in materials costs. For example, the price of array equipment on Virginia class submarines rose by $33 million above the original price estimate. In addition to inflation, a limited supplier base for highly specialized and unique materials made ship materials susceptible to price increases. According to the shipbuilders, the low rate of ship production affected the stability of the supplier base. Some businesses closed or merged, leading to reduced competition for their services and higher prices. In some cases, the Navy lost its position as a preferred customer and the shipbuilder had to wait longer to receive materials. With a declining number of suppliers, more ship materials contracts went to single and sole source vendors. Over 75 percent of the materials for Virginia class submarines—reduced from 14 ships to 9 over a 10-year period—were produced by single source vendors.
GAO, Defense Acquisitions: Improved Management Practices Could Help Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183
(Washington, D.C.: Feb. 28, 2005).
Significantly accelerating (sometimes called crashing) development schedules also present risks. In such cases, technology tends to be incorporated before it is ready, tests are reduced or eliminated, or logistics support is not in place. As case study 9 shows, the result can be a reduction in costs in the short term but significantly increased long-term costs as problems are discovered, technology is back-fit, or logistics support is developed after the system is in the field.
Case Study 9: Development Schedules, from Defense Acquisitions,
GAO-06-327
Time pressures caused the Missile Defense Agency (MDA) to stray from a knowledge-based acquisition strategy. Key aspects of product knowledge, such as technology maturity, are proven in a knowledge-based strategy before committing to more development. MDA followed a knowledge-based strategy without fielding elements such as the Airborne Laser and Kinetic Energy Interceptor. But it allowed the Ground-Based Midcourse Defense program to concurrently become mature in its technology, complete design activities, and produce and field assets before end-to-end system testing—all at the expense of cost, quantity, and performance goals. For example, the performance of some program interceptors was questionable because the program was inattentive to quality assurance. If the block approach continued to feature concurrent activity as a means of acceleration, MDA’s approach might not be affordable for the considerable amount of capability that was yet to be developed and fielded.
GAO, Defense Acquisitions: Missile Defense Agency Fields Initial Capability but Falls Short of Original Goals, GAO-06-327 (Washington, D.C.: Mar. 15, 2006).
In developing cost estimates, analysts often fail to adequately address risk, especially risks that are outside the estimator’s control or that were never conceived to be possible. This can result in point estimates that give decision makers no information about their likelihood of success or give them meaningless confidence