Software Engineering –
Framing DoD’s Issues
Ms. Kristen Baldwin
Director, Systems Analysis
15 September 2009
DDR&E Organization
Director, Defense Research
and Engineering (DDR&E)
Honorable Zachary J. Lemnios
Director,
Research
Dr. David Honey
Principal Deputy
Dr. André van Tilborg
Director,
Systems Engineering
Mr. Stephen Welby
Principal Deputy
Mr. Terry Jaggers
Director,
Rapid Fielding
Mr. Earl Wyatt
Principal Deputy
Mr. Ben Riley
Director, Developmental
Test & Evaluation (DT&E)
Mr. Chris DiPetto
(Acting)
Principal Deputy
Mr. Chris DiPetto
Defense Advanced
Research Projects
Agency (DARPA)
Dr. Regina Dugan
Defense Technical
Information Center
(DTIC)
Mr. Paul Ryan
Principal Deputy
Mr. Alan R. Shaffer
WSARA 2009
- D,DT&E
- D,SE
- Dev Planning
NDIA Top Software Issues
September 2006
1. The impact of requirements upon software is not consistently quantified
and managed in development or sustainment.
“Requirements”
2. Fundamental system engineering decisions are made without full
participation of software engineering.
“SE/SW Integration”
3. Software life-cycle planning and management by acquirers and
suppliers is ineffective.
“Sustainment”
4. The quantity and quality of software engineering expertise is
insufficient to meet the demands of government and defense industry.
“Human Capital”
5. Traditional software verification techniques are costly and ineffective
for dealing with the scale and complexity of modern systems.
“
SW Testing”
6. There is a failure to assure correct, predictable, safe, secure execution
of complex software in distributed environments.
“Assurance”
7. Inadequate attention is given to total lifecycle issues for COTS/NDI
impacts on lifecycle cost and risk.
“SW COTS / NDI / Reuse”
Issue 6: Software Assurance
•
Better evidence needed to make software
assurance claims
•
Testing is necessary but not sufficient
– Evidence is available from many sources
•
Software engineering field suffers from
pervasive lack of evidence
– Incidence and severity of software failures
– Assurance of existing software systems
NDIA Top SW Issues - 2006 vs.
SW Systemic Analysis - 2008
N a tio n a l D e fe n se In d u stria l A sso cia tio n (N D IA )
T o p 7 S o ftw a re Issu e s
A u g u st 2 0 0 6
D U S D (A & T ) S S E D ire cto ra te
P ro g ra m R e vie w S o ftw a re S yste m ic A n a lysis F in d in g s
S o ftw a re R e q u ire m e n ts
S y s te m s /S o ftw a re
In te g ra tio n
S o ftw a re H u m a n C a p ita l
S o ftw a re A s s u ra n c e
S o ftw a re A s s u ra n c e
S o ftw a re E n g in e e rin g
M a n a g e m e n t
�
P ro je ct P la n n in g
�
M a n a g e m e n t O ve rsig h t
�
S o ftw a re C o n fig u ra tio n M a n a g e m e n t
S o ftw a re M e tric s
�
S o ftw a re M e trics
�
E V M
K n o w le d g e S h a rin g
�
P ro ce ss
�
R e p o rtin g
S o ftw a re R e q u ire m e n ts
�
E n g in e e rin g
�
M a n a g e m e n t
�
A cq u isitio n S tra te g y
S o ftw a re D e v e lo p m e n t
�
S o ftw a re T e stin g *
�
S o ftw a re S u sta in m e n t/M a in te n a n ce*
�
S o ftw a re C O T S /N D I*
�
T e ch n o lo g y R e a d in e ss
�
S o ftw a re A rch ite ctu re
S o ftw a re S u s ta in m e n t
S o ftw a re C O T S /N D I
S o ftw a re T e s tin g
S o ftw a re H u m a n C a p ita l
�
R e so u rce s
�
Q u a lity L e ve l
S y s te m s /S o ftw a re
In te g ra tio n
�
S yste m s o f S yste m s
�
In te ro p e ra b ility
�
T e ch R e fre sh
Top SW Issues – Oct 2008
NDIA SE Conference
•
Workshop ranked 18 Software / System Acquisition &
Process Initiatives on Payoff v. Ease of Implementation
•
Top Four Issues:
– Start-up teams and infrastructure
– Software T&E approaches
– Incorporate “core SW metrics” in Tech Reviews, Milestones, processes, EV
– SE/SW architecture review boards – early engagement
•
Changes since Oct 2008
– WSARA 2009
– Competitive Prototyping
– Starting Acquisition Pre-MS’B’
Software/Systems Acquisition & Process Initiatives
0% 33% 67% 100% 0% 33% 67% 100% Ease
P
ayo
ff
Incorporate "core SW metrics" into milestones/reviews/processes/EV SW test and evaluation approaches
Start-up teams and infrastructure
SE/SW Architecture review board with early engagement Lifecycle cost estimation for reused SW assets
Synergize the "core SW metrics" broadly and effectively use information
Program Management interfaces across PM/SE/SW and PM’s funding/decision sources SW aspects/issues in systems-of-systems
Collaborative approach for SW products that cross org/PM/SE/SW/HW boundaries Educate key leadership using existing SW guidebooks
Contracting approach for Incremental Commitment Model for evolutionary acquisition Centers-of-excellence for contracts, EV, sim
Harmonize the OSD/AirForce/Army/Navy guidebooks
Adoption of new commercial best practices
Roles for individual SW engineers to prioritize their workloads TRL concepts/measurement for SW
SW programs’ differences/commonality Capture best-of-class SW programs
Software/Systems Acquisition & Process Initiatives
0% 33% 67% 100% 0% 33% 67% 100% Ease
P
ayo
ff
Incorporate "core SW metrics" into milestones/reviews/processes/EV SW test and evaluation approaches
Start-up teams and infrastructure
SE/SW Architecture review board with early engagement Lifecycle cost estimation for reused SW assets
Synergize the "core SW metrics" broadly and effectively use information
Program Management interfaces across PM/SE/SW and PM’s funding/decision sources SW aspects/issues in systems-of-systems
Collaborative approach for SW products that cross org/PM/SE/SW/HW boundaries Educate key leadership using existing SW guidebooks
Contracting approach for Incremental Commitment Model for evolutionary acquisition Centers-of-excellence for contracts, EV, sim
Harmonize the OSD/AirForce/Army/Navy guidebooks
Adoption of new commercial best practices
Roles for individual SW engineers to prioritize their workloads TRL concepts/measurement for SW
SW programs’ differences/commonality Capture best-of-class SW programs
Recent DDR&E / D,SE Software
Maturity Efforts
•
Renewed emphasis on systems reliability and
lifecycle costs of shortfalls
– Consolidating SW reliability guidance
•
Use of parametric models to project numbers of
latent SW defects and discovery rates
– Support planning for satellite
launches
– Aircraft production decisions
– Operational Test Readiness
System Testing - DT events
Challenges
•
Advance and Employ Systems Engineering
– Manage complexity – the role of software engineering
– Reduce risk, measure Assurance – the role of evaluation and
testing
– Earlier engagement - Development Planning
−
Pre-Milestone ‘A’, Pre-Acquisition
−
Foundation for software engineering, test and evaluation
DDR&E Imperative 3
Reduce the Cost, Acquisition Time, and Risk
of our Major Defense Systems
Development
Development Planning
DP
(Early SE)
SE
DT&E
LCS
DP: Development Planning
DT&E: Developmental Test and Evaluation
LCS: Life Cycle Sustainment
SE: System Engineering
CBA: Capabilities Based Assessment
CDD: Capability Development Document
CPD: Capability Production Document
ICD: Initial Capabilities Document
MDD: Materiel Development Decision
A
C
CBA
B
ICD
Development
Technology
Engineering and
Manufacturing
Development
Production and
Deployment
Materiel
Solution
Analysis
MDD
CPD
CDD
O&S
FRP
DR
PDR
CDR
New Emphasis on Development
Planning and Early SE
New Challenges for Programs
•
Need for Program Office formation and PM skill-sets
after MDD and prior to MS A
•
Increased importance of the Technology Development
Strategy (TDS) (as a surrogate Acquisition Strategy) at
MS A
•
Schedule and funding shifts – EMD into TD
•
Earlier engagement with industry and different
contracting strategies for technology maturation,
competitive prototyping, data rights, PDR before MS B,
etc.
•
Explicit need for earlier, formal SE process application
