Allocate, Phase, and Convert a Risk-Adjusted Cost Estimate to Then-Year Dollars and Identify High-Risk Elements

Uncertainty is calculated on the total cost estimate results, not year by year. Therefore, since a budget is requested in then-year dollars, it is necessary to convert the cost estimate into then-year dollars by phasing the WBS element costs over time. Because WBS element results at a specific confidence level will not sum to the parent levels, it will be necessary to pick the level in the WBS from which risk dollars are to be managed. The difference between the point estimate and the risk result at the selected confidence level is the amount of contingency reserve to be set aside for mitigating risks in lower WBS level elements.

Once the amount of contingency reserve has been identified, reserves need to be identified and set aside for the WBS elements that harbor the most risks so that funding will be available to mitigate risks quickly. To identify which WBS elements may need contingency reserve, results from the uncertainty analysis are used to prioritize risks, based on probability and impact as they affected the cost estimate during the simulation. Knowing which risks are important will guide the allocation of contingency reserve.

r

m

Risk and uncertainty analysis is just the beginning of the overall risk management process. Risk management is a structured and efficient process for identifying risks, assessing their effect, and

developing ways to reduce or eliminate risk. It is a continuous process that constantly monitors a program’s health. In this process, program management develops risk handling plans and continually tracks them to assess the status of program risk mitigation activities and abatement plans. In addition, risk management anticipates what can go wrong before it becomes necessary to react to a problem that has already occurred. Identifying and measuring risk by evaluating the likelihood and consequences of an undesirable event are key steps in risk management. The risk management process should address five steps:

identify risks, 1.

analyze risks (that is, assess their severity and prioritize them), 2.

plan for risk mitigation, 3.

implement a risk mitigation plan, and 4.

track risks. 5.

Steps 1 and 2 should have already been taken during the risk and uncertainty analysis. Steps 3–5 should begin before the actual work starts and continue throughout the life of the program. Over time, some risks will be realized, others will be retired, and some will be discovered: Risk management never ends. Establishing a baseline of risk expectations early provides a reference from which actual cost risk can be measured. The baseline helps program managers track and defend the need to apply risk reserves to resolve problems.

Integrating risk management with a program’s systems engineering and program management process permits enhanced root cause analysis and consequence management, and it ensures that risks are handled at the appropriate management level. Furthermore, successful risk mitigation requires communication and coordination between government and the contractor to identify and address risks. A common database of risks available to both is a valuable tool for mitigating risk so that performance and cost are monitored continually.

Regular event-driven reviews are also helpful in defining a program that meets users’ needs while minimizing risk. Similarly, relying on technology demonstrations, modeling and simulation, and

prototyping can be effective in containing risk. When risks materialize, risk management should provide a structure for identifying and analyzing root causes.

Effective risk management depends on identifying and analyzing risk early, while there is still time to make corrections. By developing a watch list of risk issues that may cause future problems, management can monitor and detect potential risks once the program is under contract. Programs that have an EVM system can provide early warning of emerging risk items and worsening performance trends, allowing for implementing corrections quickly.

EVM systems also require the contractor to provide an estimate at completion and written corrective action plans for any variances that can be assessed for realism, using risk management data and techniques. Moreover, during IBR, the joint government and contractor team evaluates program risks associated with work definition, schedule, and the adequacy of budgets. This review enhances mutual understanding of risks facing the program and lays the foundation for tracking them in the EVM system. It also establishes a realistic baseline from which to measure performance and identify risk early.

Risk management is continual because risks change significantly during a program’s life. A risk event’s likelihood and consequences may change as the program matures and more information becomes known. Program management needs always to reevaluate the risk watch list to keep it current and examine new root causes. Successful risk management requires timely reporting to alert management to risks that are surfacing, so that mitigation action can be approved quickly. Having an active risk management process in place is a best practice: When it is implemented correctly, it minimizes risks and maximizes a program’s chances of being delivered on time, within budget, and with the promised functionality.

Best Practices Checklist: Cost Risk and Uncertainty 11.

A risk and uncertainty analysis quantified the imperfectly understood risks that are …

in the program and identified the effects of changing key cost driver assumptions and factors.

Management was given a range of possible costs and the level of certainty in 3

achieving the point estimate.

A risk adjusted estimate that reflects the program’s risks was determined. 3

A cumulative probability density function, an S curve, mapped various cost 3

estimates to a certain probability level and defensible contingency reserves were developed.

Periodic risk and uncertainty analysis was conducted to improve estimate 3

uncertainty.

The following steps were taken in performing an uncertainty analysis: …

Program cost drivers and associated risks were determined, including those 3

related to changing requirements, cost estimating errors, business or economic uncertainty, and technology, schedule, program, and software uncertainty.

All risks were documented for source, data quality and availability, and ù

probability and consequence.

Risks were collected from staff within and outside the program to counter ù

optimism.

Uncertainty was determined by cost growth factor, expert opinion (adjusted ù

to consider a wider range of risks), statistics and Monte Carlo simulation, technology readiness levels, software engineering maturity models and risk evaluation methods, schedule risk analysis, risk cube (P-I matrix) method, or risk scoring.

A probability distribution modeled each cost element’s uncertainty based on 3

data availability, reliability, and variability.

A range of values and their respective probabilities were determined either ù

based on statistics or expressed as 3-point estimates (best case, most likely, and worst case), and rationale for choosing which method was discussed. Documentation of the rationale for choosing the probability distributions ù

Probability distribution reflects the risk shape and the tails of the ù

distribution reflect the best and worst case spread as well as any skewness. Distribution bounds were adjusted to account for stakeholder bias using organization default values when data specific to the program are not available.

If the risk driver approach is used, the data collected, including probability ù

of occurrence and impact, were applied to the risks themselves.

Prediction interval statistical analysis was used for CER distribution bounds. ù

The correlation between cost elements was accounted for to capture risk. 3

The correlation ensures that related cost elements move together during ù

the simulation, resulting in reinforcement of the risks.

Cost estimators examined the amount of correlation already existing in ù

the model. If no correlation is present, an insertion of 0.25 correlation was added.

A Monte Carlo simulation model was used to develop a distribution of total 3

possible costs and an S curve showing alternative cost estimate probabilities. High-priority risks were examined and identified for risk mitigation. ù

Strength of correlated cost elements were examined and additional ù

correlation added if necessary to account for risk.

The probability associated with the point estimate was identified. 3

Contingency reserves were recommended for achieving the desired confidence 3

level.

The mean of the distribution tends to fall around the 55%–65% confidence ù

level because the total cost distribution follows a lognormal trend (i.e., tendency to overrun rather than underrun costs).

Budgeting to at least the mean of the distribution or higher is necessary to ù

guard against potential risk.

The cost risk and uncertainty results were vetted through a core group of ù

experts to ensure that the proper steps were followed.

The estimate is continually updated with actual costs and any variances ù

recorded to identify areas where estimating was difficult or sources of risks were not considered.

The risk-adjusted cost estimate was allocated, phased, and converted to then- 3

year dollars for budgeting, and high-risk elements were identified to mitigate risks.

Results from the uncertainty analysis were used to prioritize risks based on ù

probability and impacts as they affected the cost estimate.

A risk management plan was implemented jointly with the contractor to identify …

and analyze risk, plan for risk mitigation, and continually track risk.

A risk database watch list was developed, and a contractor’s EVM system 3

was used for root cause analysis of cost and schedule variances, monitoring worsening trends, and providing early risk warning.

Event-driven reviews, technology demonstrations, modeling and simulation, 3