Using Maturity Models to Evaluate Offeror Capability

The Software Engineering Institute‘s (SEI) Capability Maturity Model Integration (CMMI) is a widely adopted approach for process improvement, employed by numerous companies in countries around the world. It was developed using the earlier Capability Maturity Model (CMM) for software as a foundation. CMMI includes a model, which defines the goals, objectives, and expectations for processes in a number of process areas. It also includes an assessment method which can be used to rate how well the model is being applied in practice. Using the popular staged representation, process maturity is rated at one of five levels, where level one is basic, ad hoc management (see FIGURE 7). Level three is typically viewed as a minimum level of

acceptable performance, where processes are defined and applied across a development organization. Levels four and five build on this foundation to include statistical process control and a continuous focus on improvement.

FIGURE 7. CMMI Levels.

There is sometimes a temptation for acquisition organizations to rely on CMMI ratings as a predictor of performance on actual development programs, and to simplify the evaluation of process capability during source selection by simply asking for and perhaps verifying CMMI results, or levels, during source selection. This is not good practice. The SEI provides the following cautions:

a. Levels are good indicators of potential organizational performance.

b. They describe how the next project could perform based on a sampling of existing projects.

c. Capability levels and maturity levels reside at the organizational level (corporation, major division) and are not an indication of how any individual project is performing.

There is no OSD or Air Force policy requiring that offerors/developers/contractors have a minimum maturity level (such as Level 3) to do business with the Government. Mark D.

Schaeffer, OSD (AT&L), Systems and Software Engineering, stated the following in the July-August 2007 issue of Defense AT&L Magazine: ―DoD does not place significant emphasis on

capability level or maturity level ratings, but rather promotes CMMI as a tool for internal process improvement. This lack of emphasis on ratings is prudent in the light of findings that not all suppliers are exhibiting behavior consistent with their attained CMMI maturity level rating.‖

Additionally, according to data collected and analyzed by the DCMA, there does not appear to be a link between maturity levels and program performance. Furthermore, there does not appear to be cost and schedule improvement by moving from Level 3 to Level 5 process maturity.

While CMMI has been shown to be a valuable tool for internal process improvement, it is clear that great care must be taken when trying to interpret and apply CMMI-related information in source selection. Acquisition organizations that have the resources, knowledge, and experience available to take a CMMI-based approach to evaluating offeror capability in the source selection timeframe should consider the following:

a. Do not assume that because the contractor asserts a specific maturity or capability level rating that they will execute that level of process proficiency on a new program. These ratings are at the business unit (or organizational unit) level and are not necessarily applicable to a specific program individually.

b. Unless the specific solicitation involves a program that is already in progress (e.g., a follow-on to a previous phase or a modification to an existing contract), using CMMI appraisals (Standard CMMI Appraisal Method for Process Improvement – SCAMPI) as part of the source selection process will provide little to no insight into how the contractor will perform on the next program. Typically, the contractor will not select, tailor, and execute process for the specific program until after it has been awarded.

c. Do not include requirements for any specific maturity or capability level ratings. Rather, include (in the SOW/CSOW) that the contractor is required to execute a minimum set of processes or process areas (PAs) that are important to the program. These PAs may include, for example, Requirements Development (RD), Requirements Management (REQM), Risk Management (RSKM), and Configuration Management (CM). Although it is recommended that simple statements of required maturity or capability levels be avoided, developers can be required to execute at least level 3 equivalent processes on the program. Note that process areas defined by CMMI as level 3 process areas are generally considered to encompass the key processes necessary for program execution.

Disciplined process execution can be tied to award fee. If complete avoidance of any CMMI terminology is desirable, the minimum set of PAs to be implemented can be addressed in the SOW/CSOW.

d. Maturity or capability level performance information relative to the contractor business unit (or organizational unit) within which a specific program will reside may offer some insight into the source of the standard processes that may be implemented on that program. If this information is of interest, it can be requested in RFP Section L by requiring a full Appraisal Disclosure Statement (ADS) and Findings Briefing for any SCAMPIs that have been conducted on that business unit in the last three years. The ADS can identify and clarify relevant appraisal conditions and factors, and can provide some understanding that the process artifacts and tailoring procedures are available from which to draw for individual projects. Also, note that in general, valid, independent appraisal results will undergo quality checks and will be accepted by the SEI. Further, while provisions are made for ―sensitive‖ data, if the results of the appraisal are to be announced publicly or used in a formal proposal, then the appraisal results must be placed on the SEI's Published Appraisal Results Site (PARS).

e. Past performance information can be useful in determining how well a contractor executed their standard processes and how well those processes benefited programs

previously. However, the past performance questionnaire must be specifically tailored to expose this information in a useful manner.

f. Language can be included in the contract (e.g., Section H clause) that allows the Government to conduct process reviews or appraisals at periodic points in the program at the Government‘s discretion. One or two of these reviews at key points in the program (e.g., post-IBR or pre-Critical Design Review (CDR)) will provide insight into how the contractor has tailored and implemented processes on the program. The notion that the Government will check on how processes are tailored and implemented is a strong motivator for the contractor to follow their processes after contract award.

3.4.4 Sources of Additional Guidance

a. ―Choosing a Supplier: Due Diligence and CMMI,‖ News@SEI 2004 Number 3, CMMI in Focus, David M. Phillips.

b. ―Understanding and Leveraging a Supplier’s CMMI Efforts: A Guidebook for Acquirers.‖

c. Air Force Federal Acquisition Regulation Supplement (AFFARS) Mandatory Procedure 5315.3 – Source Selection