This report shows a clear relationship between the deployment of SE practices and improved Pro-ject Performance. While the output of this survey activity is complete in its current form, and needs no further work to find application in the defense industry today, it also suggests some di-rections for future initiatives.
8.1 36BCORRELATE REPORT FINDINGS WITH OTHER SOURCES
While numerous other initiatives have been undertaken within government, industry, and acade-mia to characterize the best SE practices (or lack thereof) applied on defense programs, the benefit of this report is to support informed decision-making based on both quantitative and qualitative measures of effectiveness. This data can be used to complement findings derived from other sources, such as studies, reports, or root cause analyses of program performance issues, to develop a well-rounded picture of the state-of-the-practice for SE within the defense industry and to priori-tize improvement actions in areas that are likely to have the greatest benefit in improved program performance.
8.2 37BDEVELOP IMPROVEMENT RECOMMENDATIONS
Based on the findings of this report, the NDIA Systems Engineering Effectiveness Committee (SEEC) will develop recommendations for government and industry actions needed to improve the practice of systems engineering on DoD programs.
Candidate areas for these recommendations may include, but are not limited to
• updates to OSD policy and guidance to reinforce the application and support of sound sys-tems engineering practices on programs
• improved training in targeted SE capability areas (significant strengths or weaknesses)
• recommendations on standard measures to be collected and analyzed
• suggested improvement to evaluation criteria for program plans, reviews, or risk analyses
• greater communication of proven SE best practices (e.g., publications, conferences, work-shops)
Note that numerous other efforts are already underway to improve systems engineering capabili-ties across the defense industrial base. For example, in the Office of the Deputy Under Secretary of Defense (A&T), the Director, Systems and Software Engineering has established a number of initiatives focusing on SE [Schaeffer 2007]. These include
• issuing a Department-wide Systems Engineering (SE) policy
• issuing guidance on SE, T&E, and SE Plans (SEPs)
• integrating DT&E with SE policy and assessment functions—focusing on effective, early
• working with Defense Acquisition University to revise curricula (SPRDE, T&E, PQM, LOG, PM, ACQ, FM, CONT)
• establishing the SE Forum to ensure senior-level focus within DoD
• leveraging close working relationships with industry and academia
• instituting system-level assessments in support of DAB, OIPT, DAES, and in support of pro-grams
• instituting a renewed emphasis on modeling and simulation in acquisition
To maximize the likelihood of positive action, recommendations developed by the SEEC will give utmost consideration to leveraging existing initiatives such as these, where there is already considerable inertia and government/industry support for improvement activities, before propos-ing new initiatives that would otherwise be competpropos-ing for attention and resources.
8.3 38BADDITIONAL ANALYSIS OF COLLECTED DATA
The analysis discussed in this report does not extract all of the knowledge available from the col-lected data set; additional analysis is possible. Many areas of study are possible; two examples are presented below.
For example, responding projects were executing in various phases across the life cycle. While data on Project Performance was collected, it was not compared with position in the life cycle.
Early in a project, estimates of cost-at-completion and project completion dates seldom vary from original estimates. Only as progress (or lack of progress) occurs are deviations from these original estimates recognized. Thus, on-budget and on-schedule claims later in a project are more credible than the same claims early in the project. This factor could be included in a more sophisticated analysis of Project Performance.
As another example, the survey collects data on organizational CMMI maturity levels. Achieve-ment of these levels requires the achieveAchieve-ment of specified CMMI goals, and includes the expecta-tion of performance of various CMMI practices. Many of the survey quesexpecta-tions assessing SE Ca-pabilities are related to these same practices. An analysis of the consistency between the claimed maturity levels and the performance of practices could reveal the degree of deployment of CMMI from the organizational level to the project level.
8.4 39BPERIODIC REPEAT OF THE SURVEY
Broader representation of programs and companies across the defense industrial base could help provide additional insight beyond this initial survey analysis. As government- and industry-based initiatives prevail, one could also expect to see improvements in SE Capabilities applied to pro-jects.
Meanwhile, defense systems continue to reach unprecedented levels of complexity in a dynamic environment that is continually evolving in areas such as program mission, emerging technolo-gies, development approaches, tools, teaming relationships, and acquisition strategies.
A periodic re-execution of this survey and similar subsequent efforts could quantify the improve-ments in SE Capability, and could also ascertain the impact of these changes on Project Perform-ance.
8.5 40BSURVEY OF ACQUIRERS
Everything in this survey is presented from the perspective of the supplier. Additional knowledge could be gained by examining projects from the perspective of the acquirer. This could be accom-plished through the development of a similar, but not identical, questionnaire. While it would be valuable to be able to join the results of the current survey with such an acquirer survey, this most probably will not be feasible due to the anonymous nature of the current survey data. Addressing both perspectives together while maintaining confidence in nondisclosure is an important chal-lenge for future work in this area..