Systems Engineering Sizing in the Age of Acquisition Reform

(1)

Systems Engineering Sizing

in the Age of Acquisition Reform

Ricardo Valerdi Michael Ernstoff Paul H. Mohlman

Center for Software Engineering Consultant The Aerospace Corporation University of Southern California 1 1 940 Victoria Avenue 2350 E. El Segundo Blvd. Los Angeles, CA 90089 Los Angeles, CA 90066 El Segundo, CA 90245 rvalerdi 8 sunset.usc.edu m.ernstoff 8 attbi.com [email protected]

Don Reifer Evin Stump

Reifer Consultants, Inc Galorath, Inc

P.O. Box 4046 100 N. Sepulveda #I801 Torrance, CA 90501 El Segundo, CA 90245 dreifer8earthlink.net estump8 aaIorath.com

Abstract

As organizations develop more complex systems, increased emphasis is being placed

I

on Systems Engineering (SE) to ensure that cost and schedule are within budget. Correspondingly, the failure to adequately plan and fund the systems engineering effort appears to have contributed to a number of cost overruns and schedule slips, especially in the development of complex ground and space systems. Government and commercial organizations have recently placed increased emphasis on accurately planning the SE function and on understanding the factors that influence the resources needed to implement and perform SE.

In an attempt to better quantify the SE activity, the DoD acquisition community has been exploring Systems Engineering Revitalization and migrating back to a reinstatement of prescribed military standards. Several models and tools have become available to aid in forecasting systems engineering resource needs, but few guidelines exist to help engineers and program managers determine which approach is best suited for estimating any particular effort.

To provide such guidelines, this paper provides an overview of eight existing cost models

-

three of which focus on software and five on hardware. These models include SE components and employ unique approaches to sizing the SE effort. An overview of the genesis and assumptions of each model shed some light on their individual applicability. To complete the paper, future research topics are discussed along with recommendations on tasks to enhance the accuracy of SE sizing.

l€fh Annual Forum on COCOMO and Software Cost Modeling

(2)

I. Introduction

This paper represents the beginning of collaborative efforts between industry and

I

academia to refine the process of estimating SE size and costs. The authors hope that this paper establishes the initial step in that direction and will continue to work towards a relevant model that can be used to better quantify SE in different application domains.

The U.S. Air Force recently initiated the Systems Engineering Revitalization program which highlights the importance of Systems Engineering in military ground and space systems (Air Force 2003). While it has been shown that the appropriate level of SE effort leads to better control of project costs (Honour 2002), identifying the necessary level of SE effort is not yet a mature process. Some projects use the traditional 15% of the prime mission product or prime mission equipment to estimate SE, while other projects tend to use informal rules of thumb. These simplified and inaccurate methods can lead to excessively high bids by allocating too many hours on SE or even worse, by underestimating the amount of SE needed.

SE is a sine qua non' of the acquisition and implementation process. This paper focuses on providing a starting point to enhance the estimation of SE resources. The objectives of this paper are to (1) describe current estimating models that address SE components, (2) compare the genesis, assumptions, and applicability of these models, and (3) suggest future research in this area that can help mature the process of estimating SE size, resources and cost.

II. Methodology

The authors reviewed the available cost estimation models with respect to SE and

I

selected what was believed to be the most influential models currently in use by the acquisition community. One finding during the review was that SE costs were extremely sensitive to the "sizing" rules that formed the basis of these models. These rules help estimators answer questions like "how big will this system be?" and "what is the size of the job?" The emphasis on sizing also allowed the authors to compare common aspects of the different cost models. A similar comparative analysis of cost models was previously completed (Kemerer 1987), which focused exclusively on models for software development. In contrast, the following considers both software and hardware cost models as they both involve SE.

Ill. Overview of Cost Models

Cost models have been an essential part of DoD acquisition since the 1970s. Hardware

I

models were the first to be developed and were followed by software models in the

1980s. To put the analysis in context, we provide a timeline (Figure 1) (Ferens 1999) and background on several models that shows the progression of parametric cost estimation.

1

essential element

ldh Annual Forum on COCOMO and Software Cost Modeling

(3)

Hard-

ware

Soft- ware

Other

Figure 1. Parametric Model Timeline Source: Ferens 1999. Used with permission.

The eight cost models studied are shown in Table 1. The authors believe that these models are representative of the cost estimation field and are the most commonly used in the acquisition process. The scope of this paper considers models for which data were available.

Table 1. Cost Models With Systems Engineering Components

Model Name OwneriDeveloper

I

Domain

I

COCOMO 11: The Constructive Cost Model (COCOMO) was developed by Dr.

Barry Boehm when he was at TRW and made popular by his classic textbook, Software Enqineerinq Economics, Prentice-Hall, 1981, which was updated in 2000 and made more relevant to current methods used by software engineers in building their products. A new calibration for the model will be available in 2004, which will contain data from approximately 200 software projects.

ldt' Annual Forum on COCOMO and Software Cost Modeling

Page 3 of 3 COCOMO ll

PRICE-H PRICE-S

Raytheon SE Resource Forecasting Tool SEER-H SEER-SEM SSCM USCM8 USC PRICE Systems, LLC PRICE Systems, LLC Raytheon Galorath, Inc. Galorath, Inc.

The Aerospace Corporation Los Angeles Air Force Base

Software Hardware Software Hardware Hardware Software Hardware Hardware

(4)

PRICE-H: The Parametric Review of Information for Costing and Evaluation (PRICE) Hardware model was the first widely available parametric cost estimation model, developed in 1973 by Martin Marietta Price Systems (initially RCA Price, then GE Price) to estimate development and production costs of hardware, based on parameters such as weight and manufacturing complexity. It has been continuously updated since its creation to make it more relevant to current best practices.

PRICE-S: The Parametric Review of lnformation for Costing and Evaluation (PRICE) Software model was originally developed in 1977 by Martin Marietta Price Systems to estimate software development cost and schedule, using design parameters such as software size, application, and complexity. It has also been updated to reflect the current state-of-the-practice in software engineering.

Raytheon's System Engineering Resource Forecasting Tool: Raytheon's System Engineering Resource Forecasting tool is the result of work begun by Michael Ernstoff at Hughes Aircraft Company's Electro-Optical and Data Systems Group, prior to its acquisition by Raytheon in December 1997. Management's objective was to obtain a tool that could consistently set quantitative boundaries on the scope of SE efforts for space-based sensors that were constrained by budgetary limitations. Subsequently, utilization of the tool has served as a sanity check on the magnitude of bottom-up estimates. Ivan Vincenzini at Raytheon El Segundo has pioneered efforts to apply the tool to space-based radar sensors in addition to the originally envisioned space-based infrared sensors.

SEER-H: The System Evaluation and Estimation of Resources (SEER) Hardware model was developed by Galorath Incorporated and initially released in 1991. Development has continually progressed since then. SEER-H version 5.1 reflects what the hardware world is currently shipping.

SEER-SEM: The SEER Software Engineering Model (SEER-SEM) was also developed by Galorath Incorporated. Originally based on the Jensen model of software effort and schedule estimation, it has evolved over the years to address modern practices for software engineering. SEER-SEM was initially released in 1988. Development has been ongoing since then. SEER-SEM version 7.0 is expected for release in the fall of 2003.

SSCM: The Small Satellite Cost Model, developed by The Aerospace Corporation, is a parametric cost model consisting of a series of mathematical cost estimating relationships that relate spacecraft bus cost to physical, technical, and performance parameters that are known or believed to strongly influence spacecraft costs. The basis for these relationships is a database of actual costs and technical parameters for 35 modern small satellites.

1 dh Annual Forum on COCOMO and Software Cosf Modeling

(5)

USCM8: The Unmanned Space Vehicle Cost Model version 8 (USCM8) was developed by Tecolote Research, Inc. under direction from the Space and Missile Systems Center (SMC) at the Los Angeles Air Force Base as a parametric cost estimating tool based on cost estimating relationships (CERs) derived from a historical database. USCM has formed the basis of numerous Independent Cost Estimates (ICE) for long-range planning and tradeoff studies, as well as detailed Component Cost Analyses (CCA) in support of the Defense Acquisition Board (DAB) process. USCM1, the first edition, was published in November 1969 by the Cost Analysis Division of the Space and Missile Systems Organization and was based on a historical database of 5 military, 5 NASA and 1 commercial space vehicle programs. USCM8 was published in October 2001 and is based on 23 military, 11 NASA and 9 commercial programs.

The next section compares the eight aforementioned models in five key areas relevant to systems engineering:

1. Model inputs for software or hardware size 2. Definition of Systems Engineering

3. Model inputs for Systems Engineering

4. Life Cycle stages used in the model 5. Domain of applicability

These areas provide information on the applicability of each model to Systems Engineering sizing.

1. Model inputs for software or hardware size

COCOMO 11: Software size can be estimated using SLOC (Source Lines of

Code), FPs (Function Points), or APs (Application Points). Modified and reuse software size is calculated as equivalent SLOC of new code based on a non- linear reuse model. A separate non-linear reuse model is used to size software designed for reuse.

PRICE-H: Hardware size can be estimated by using weight, manufacturing

complexity, platform, quantities, schedule, engineering difficulty, the skill level of those that will perform the work, and the amount of new work generated.

PRICE-S: Software size can be estimated using either SLOC FPs, or POPS

(Predictive Object Points). Additional effort multipliers include the programming languages employed, the function performed by the software, the amount of new work generated, the skill level of those that will perform the work, the development process used, the constraints imposed by hardware, and how the customer plans to use the software.

Raytheon's System Engineering Resource Forecasting Tool: The key-driving

factor in this model is system complexity. Complexity may correlate with

ldh Annual Forum on COCOMO and Software Cost Modeling

(6)

software lines of code and hardware weight or volume, but it is complexity, rather than the physical parameters, that drives systems engineering resource needs. The Raytheon System Engineering Resource Forecasting tool estimates system complexity by assigning a score to each subsystem interface and compiling a weighted sum of the scores. Learning curves are used to scale down the contribution from repeating interfaces and to account for design re-use. In many ways, the procedure is analogous to that of counting function points.

SEER-H: The key sizing inputs used by SEER-H depend on the type of hardware,

specifically electronics versus mechanical. Weight is the main driving parameter for mechanical hardware, printed circuit board count and size for electronics. Other influential parameters are materials used, certain design and manufacturing options, team experience, and team tools.

SEER-SEM: The software effort can be sized using SEER-SEM based on

function points, source lines of code, or user-defined metrics. Users can combine or select a single metric for any project element or for the entire project. COTS WBS elements also have specific size inputs defined either by Features, Object Sizing, or Quick Size, which describe the functionality being integrated.

SSCM: SSCM 2002 calculates the size for each hardware subsystem using

several attributes, including weight and other primary parameter(s) that are particular to a system, such as structural material or pointing knowledge. The cost size can be determined for either government or commercial missions.

USCM8: Hardware size is calculated for each USCM8 Work Breakdown

Structure item using weight in conjunction with primary parameter(s) that are particular to a system, such as number of communication channels. The size is segregated into nonrecurring costs and recurring costs for the first production article and for the production build quantity. The end of the nonrecurring costs phase is denoted by the completion of prototype qualification. The initialization of the recurring costs phase is signaled by the release of design drawings to flight hardware manufacturing.

2. Definition of SE

COCOMO 11: Support to SE is defined in terms of the additional effort that

software organizations expend in supporting SE activities which include, but are not limited to, the following: Integrated Product Teams developing system specifications and operational concept documents, Interface Control Working Groups defining hardware/software interfaces, equation definition teams developing validated algorithms, implementation teams developing systems/subsystems, and test and evaluation teams verifying and validating that systems requirements have been satisfied.

ldh Annual Forum on COCOMO and Software Cost Modeling

(7)

PRICE-H: SE is bundled into the Systems lntegration and Project Management components of the model. The Systems lntegration component includes the merging of hardware products into a single unified system, integrating hardware and software components into one system, and assembly integration and test. The Project Management component includes the efforts to assure the proper and most cost-effective performance of the contract between the customer and the company. Included in this is also technical management which involves: design review activities, reliability and maintainability studies, product assurance efforts, configuration management duties, and production engineering tasks such as assembly and test methods.

PRICE-S: The SEIProject Management element includes the Systems Engineering effort to define the software system, and the Project Management Effort to manage the software development project. The Systems Engineering activity encompasses the effort to define system level requirements, the integrated planning and control of the technical program efforts of design engineering, specialty engineering, development of test procedures, and system oriented testing and problem resolution.

Raytheon's Systems Engineering Resource Forecasting Tool: The scope of SE is defined in terms of a set of SE products that were compiled from the Systems Engineering product lists in EIA632, IEEE1220, the NASA Systems Engineering handbook and other sources. What results is an indentured list of products and a dictionary defining each product. The scope of each proposed SE effort is custom-defined by restricting the list of products to only include those that must be delivered as part of a proposed program. All resource projections are associated with required physical deliverables; there are no "level-of-effort" tasks.

SEER-H: The functions included in the SE and lntegration (SE&I) element encompass: ( I ) the SE effort to transform an operational need into a description of system requirements and/or a preferred system configuration; (2) the logistics engineering effort to define, optimize, and integrate logistics support considerations to ensure the development and production of a supportable and cost effective system; and (3) the planning, monitoring, measuring, evaluation, and directing of the overall technical program. Specific functions include those for control and direction of engineering activities, costlperformance tradeoffs, engineering change support and planning studies, technology utilization, and the engineering required for safety, reliability, and quality control and assurance. Also included is the effort for system optimization, configuration requirements analysis, and the submittal and maintenance of Interface Control Documents (ICDs).

SEER-SEM: The System Requirements Design activity includes the creation of initial system requirements and related tasks. If the system has software and hardware components, specific functions are typically allocated to software.

(8)

SSCM: SSCM combines systems engineering, program management and lntegration Assembly & Test as a single cost. SE and Program Management include quality assurance, reliability, requirements activities, program management, dataheport generation, and special studies not covered by or associated with specific satellite subsystems. lntegration Assembly & Test (IA&T) includes researchlrequirements specification, design and scheduling of IA&T procedures, ground support equipment, systems test and evaluation, and test data analyses.

USCM8: USCM8 combines SE, Program Management and Data as a single cost, which is given the acronym SEPMD. SEPMD, also called Program level costs, include those accounts for program management, reliability, planning, quality assurance, systems analyses, project control, and other costs that cannot be related to any other specific area of activity. SE includes all effort associated with the engineering organization, which allocates and controls the distribution of system-level requirements and specifications to lower level subsystems and equipment items. Also included are costs associated with controlling system- level documents such as specifications, weight management, system reliability, and overall quality assurance. Program management includes all effort associated with defining, planning, directing, and controlling company functions, subcontractors, and suppiiers in order to accomplish program objectives. Data includes costs for program-related graphic and written information, whether technical or non-technical. Most data requirement costs that fall into this category are controlled by a contract data requirements list (CDRL) attached to the system's contract.

Model inputs for SE

COCOMO I/: Software support to SE is estimated as an additional cost using heuristics or rules of thumb developed by polling experts. COCOMO is essentially a software-only estimation model. This support effort is then allocated to life cycle phases using additional rules of thumb developed for that purpose. PRICE-H: SE is estimated as part of Systems lntegration and Project Management.

PRICE-S: SE is included in the Systems EngineeringIProject Management cost element. This cost element is estimated for each phase of the project life cycle. The core equation relates labor effort to software size (or volume) and productivity. These two factors are combined to calculate the labor hours for the SE effort during each phase of the project.

Raytheon's System Engineering Resource Forecasting Tool: The Raytheon System Engineering Resource Forecasting tool requires numerous input parameters, so that any moderate error does not excessively skew the overall

lgh Annual Forum on COCOMO and Software Cost Modeling

(9)

result. Furthermore, the input parameters have been carefully chosen so as to solicit consistent replies. For example, estimators are not asked to estimate the percent reuse; they are asked to provide the number of times the design team has previously used each specific subsystem. The parameters include: system complexity, requirements volatility, schedule flexibility, schedule aggressiveness, security classification, platform, and staffing.

SEER-H: The system level cost capability (currently in development, not in the

released product) includes parameters specific to systems engineering cost estimation. They include development complexity, development experience, production complexity, and production experience.

SEER-SEM: SEER-SEM focuses on software projects; therefore its inputs are

geared toward software project developments. All inputs would influence SE, and affect other outputs as well. However, there are no inputs that are exclusively dedicated to the estimation of the SE effort.

SSCM: SSCM uses several drivers including design life, development time, type

of orbit (planetary or earth orbit), and type of customer (government or commercial) as inputs to size the cost of SE, program management and Integration Assembly & Test (IA&T). SSCM2002 does not treat SE as a separate cost.

USCM8: USCM8 does not break out systems engineering as a separate cost.

USCM8 combines systems engineering, program management and data as a single cost, represented by the acronym SEPMD. The input to calculate SEPMD is a factor multiplied by the sum of the space vehicle cost plus the cost of the integration, assembly and system test. The SEPMD is segregated into nonrecurring costs and recurring costs for the first article and for the production build quantity.

4. Life cycle staqes

COCOMO 11: COCOMO allows its users to allocate effort and duration estimates

to either a waterfall or MBASE life cycle. MBASE is a life cycle generator that is fully compatible with modern incremental and spiral life cycle models like the Rational Unified Process (RUP). These phases include: (1) Inception, (2) Elaboration, (3) Construction, and (4) Transition.

PRICE-H: The model identifies five stages of the hardware life cycle: (1) initial

concept, (2) design, (3) production, (4) operation, and (5) disposal.

PRICE-S: PRICE uses the nine DoD-STD-2167A development phases: (1)

Concept, (2) System Requirements, (3) Software Requirements, (4) Preliminary

ldh Annual Forum on COCOMO and Software Cost Modeling

(10)

Design, (5) Detailed Design, ( 6 ) Codelunit test, (7) lntegration & Test, (8) Hardwarelsoftware Integration, and (9) Field Test.

Raytheon's System Engineering Resource Forecasting Tool: The products of SE are usually specific to a life cycle phase. By limiting the scope of the products included in deliverables for the proposed program, one ensures that the resource projection reflects the appropriate portion of the life cycle, if not using a cradle-to- grave resource projection. These products are organized into an indentured list of product categories whose main headings are: (1) Management Plan, (2) System Design, (3) System Analysis, (4) Specifications & Interfaces, (5) Status Reports and Reviews, (6) Assembly, lntegration & Test Requirement Plans & Procedures, (7) Test Equipment, Facilities & Delivery, (8) Assembly, lntegration

& Test, (9) Validation, and (1 0) Post-Delivery Support.

SEER-H: The stages inc!uded in SEER-H are: (1) development, (2) production, (3) operations and (4) support.

SEER-SEM: The following list encompasses various activities of development covered by SEER-SEM, and definitions of each. Activities may be defined differently across development organizations, although they usually can be mapped to SEER-SEM's designations. The effort associated with each activity would still occur in some form or another. These activities include: (1) System Concept, (2) System Requirements Design, (3) Software Requirements Analysis,

(4) Preliminary Design, (5) Detailed Design, (6) Code and Unit Test, (7) Component lntegration and Testing, (8) Program Test, (9) Systems lntegration through OT&E & installation, and (1 0) Operation Support.

SSCM: The SSCM2002 life cycle covers the inception and development of the spacecraft bus, from Authority To Proceed through to launch and initial on-orbit checkout. SSCM2002 also includes lntegration Assembly & Test, Program Management (PM), Systems Engineering (SE) and Launch and Orbital Operations Support (LOOS). LOOS consists of pre-launch planning, trajectory analysis, launch site support, launch-vehicle integration (spacecraft portion), and initial on-orbit operations before ownership is turned over to the operational user (typically 30 days).

USCM8: The USCM8 life cycle covers the inception and development of the space vehicle, through launch and initial on-orbit operations. USCM8 includes hardware costs as well as the costs for analysis, design, production, integration, and testing efforts directly associated with each component~subsystem of the space vehicle. USCM8 also includes aerospace ground equipment (AGE) and launch and orbital operations support (LOOS). AGE consists of the equipment required to support the space vehicle during ground test and preparation for flight operations. LOOS includes any effort associated with pre-launch planning, launch and ascent, and initial on-orbit operations. This period ends when the

(11)

newly deployed satellite is turned over to the operational user, typically after a period of two to three weeks.

The life cycle stages for the three software and five hardware models are shown in Figure 2. Separate mapping has been done between the software models and hardware models because of their fundamental difference in scope.

Model Life Cvcle Staqes

COCOMO 11

/

Inception

I

Elaboration Iconstrustion

I

Transition

I

,

PRICE-S

PRICE-H

1

ixr:;af concept

I

Desiqr;

I

Praductzun

I

operatici?

I

UBposal

I

RSERFT SEER-H \ SSCM I Orbrinl Ops suppon 5. Domains of applicabilitv I I I

COCOMO 11: Commercial & government software development. USCM8

PRICE-H: Electronic and mechanical hardware assemblies and systems for avionics and space systems.

PRICE-S: Commercial & government software development.

Figure 2. Life Cycle Stages Compared

Inception

Raytheon's System Engineering Resource Forecasting Tool: Infrared and radar sensor development programs.

SEER-H: Electronic and mechanical hardware development and production, and operations and support.

Developmen Launch

SEER-SEM: Software intensive projects, including development and

maintenance.

Orhnal ops

s~~~~~

SSCM: Government and commercial space vehicle buses for earth orbit or planetary missions. Payloads are not included. Focus tends to be on Class C

and D vehicles (DoD HDBK 343).

ldh Annual Forum on COCOMO and Software Cost Modeling Page 11 of 11

(12)

USCM8: Military, NASA and commercial space vehicle bus and communications payloads development. Focus tends to be on Class A and B vehicles (DoD HDBK 343).

IV. Discussion

&

Recommendations

The increasing frequency and number of programs that have run significantly over-

I

budget and behind schedule (GAO 2003) because SE problems were not adequately understood should, by itself, be reason enough for the acquisition community to press for improvement in forecasting SE resource needs. However, even if the history of SE problems is ignored, the future paints an even more demanding picture. The undeniable trend is toward increasingly complex systems dependent on the coordination of interdisciplinary developments where effective system engineering is no longer just another technology, but the key to getting the pieces to fit together. It is known that increasing front-end analysis reduces the probability of problems later on, but excessive front-end analysis may not pay the anticipated dividends. The key is to accurately estimate early in a program the appropriate level of SE to ensure mission success within cost and schedule budgets. Our findings have shown that most current estimation tools treat SE as a subset of a software or a hardware effort. In that complex systems are not dominated by either hardware or software, SE ought not to be viewed as a subset of hardware or software. Rather, because many functions can be implemented using either hardware or software, SE is becoming the discipline for selecting, specifying and coordinating the various hardware and software designs. Given that role, the right path is to forecast SE resource needs based on the tasks that systems engineering must perform and not as an arbitrary percentage of another effort. Hence, SE estimation tools must provide for aligning the definition of tasks that SE is expected to do on a given project with the program management's vision of economic and schedule cost, performance and risk.

Tools that forecast SE resources largely ignore factors that reflect the scope of the SE effort, as insufficient historical data exists from which statistically significant algorithms can be derived. To derive cost-estimating relationships from historical data using regression analysis, one must have more data points than variables. It is difficult to obtain actual data on systems engineering costs and on factors that impact those costs. For example, a typical factor may be an aggressive schedule, which will increase the demand for SE resources. The result is a tool set that inadequately characterizes the proposed program and therefore inaccurately forecasts SE resource needs.

V. Conclusion

&

Directions for Future Research

The estimation of SE size is a time-consuming and complex process. The

I

acquisition community is pushing for more accurate quantification of the SE work that is performed on hardware and software systems. In order to realize more

lgh Annual Forum on COCOMO and Software Cost Modeling

(13)

sophisticated methods for estimating SE effort, the models should be based on empirical data. Rather than using rules of thumb, models should be data-driven. Instead of having limited relevance to only standard life cycles, they should be more flexible. Further work is required to make the task of managing SE more quantitative. One of the first steps would be to more fully understand the factors that drive the cost and why there exists so much variation.

The development of a tool comparison matrix would be of value to enable the selection of the most appropriate tool for a particular project. Such a comparison would establish uniform definitions and factors that would encompass the breadth of how the different tools assess SE. Such a comparison would facilitate identification of areas lacking sufficient detail and establish understanding to appropriately quantify SE. These findings would suggest common areas of enhancement and development for addition research.

If industry and academia are to develop and validate a tool to more accurately forecast SE resources, they must first have good information on numerous projects involving SE. Getting this data is nearly impossible in the present environment where there is little standardization on the scope of the various Systems Engineering

activities. For example, what is systems integration? Does SE include those redesign activities found necessary to make the components work together as intended in a system? If SE includes component redesign, might one project discipline that is running out of resources merely throw their responsibility over the fence into the SE bullpen by merely ignoring certain issues? The one discipline claims they completed their work within budget, but SE overruns its budget because it was stuck with resolving design issues that should have been completed previously. The first step, then, for developing a better tool for forecasting SE resource needs is to standardization on a Work Breakdown Structure and the definition of SE, so that one engineer's integration includes the same items as another's.

Knowing what the cost is of a program is not the only data point needed. Additional information is also needed, such as identifying which factors have driven the size and effort elements of the program cost. Experts in the field can identify with reasonable certainty the parameters that drive SE costs, such as complexity, requirements volatility, and schedule aggressiveness. Standardized methods need to be established to identify these parameters and other variables based on what the Systems Engineer knows at the outset of a program. These methods must be designed so that same results can be generated consistently; without consistency any observed correlation would be meaningless. The cost drivers for the proposed program, solicited from the program manager, systems engineer, and other knowledgeable individuals, must be in agreement. The second step to developing a better SE forecasting tool is the development of a methodology to consistently and accurately characterize a program would enable the synthesis of a more accurate SE forecasting tool.

(14)

Once one has established a solid description of what is to be estimated (i.e., the scope of the effort) and a set of methods to quantitatively characterize the parameters impacting systems engineering resource needs, a tool can be developed using cost estimating relationships derived from expert opinion. The output of this tool can then be tested against actual history to determine accuracy and precision. Over time, as the tool is exercised, data should be collected that would gradually validate more and more of the cost estimating relationships. The process takes time; thus, the third step is to adopt a suitable model for actual use. It is only by using the model that its weaknesses and strengths can be discovered.

Glossary

AGE AP CCA CDRL COCOMO CER COTS DAB DoD EIA FP GAO IA&T ICD IEEE ICE LOOS MBASE PM POP PRICE RUP SE SEER SE&I SEPM D SLOC SMC SSCM OT&E USCM8 WBS

Aerospace Ground Equipment Application Points

Component Cost Analysis

Contract Data Requirements List Constructive Cost Model

Cost Estimating Relationship Commercial off the shelf Defense Acquisition Board Department of Defense Electronic Industries Alliance Function Points

General Accounting Office lntegration Assembly & Test Interface Control Document

Institute of Electrical & Electronics Engineers Independe~t Cost Estimate

Launch & Orbital Operations Support

Model Based Architecting & Systems Engineering Program Management

Predictive Object Points

Parametric Review of Information for Costing and Evaluation Rational Unified Process

Systems Engineering

System Evaluation and Estimation of Resources Systems Engineering & lntegration

Systems Engineering Program Management and Data Source Lines of Code

Space and Missile Systems Center Small Satellite Cost Model

Operational Test & Evaluation

Unmanned Satellite Cost Model version 8

Work Breakdown Structure

1 8Ih Annual Forum on COCOMO and Software Cost Modeling

(15)

References

Air Force Systems Engineering Revitalization website:

http://ax.losangeles.af.mil/se~revitalization/main.htm, accessed on 0911 5/03. Boehm, B. W., Abts, C., Brown, A. W., Chulani, S., Clark, B. K., Horowitz, E., Madachy,

R., Reifer, D., Steece, B., "Software Cost Estimation with COCOMO II", Prentice Hall, 2000.

Ernstoff, M., "Estimation of System Complexity", International Society of Parametric Analysts - Los Angeles Chapter, April 2001.

Ernstoff, M., Vincenzini, I., "Guide to Products of System Engineering", International Council on Systems Engineering, Las Vegas, NV, August 1999.

Ferens, D., "Parametric Estimating - Past, Present, and Future", 1 gth PRICE European Symposium, October 1999.

GAO-03-1073, Defense Acquisitions Improvements Needed in Space Systems Acquisition Management Policy, www.gao.gov/cgi-bin/getrpt?GAO-03-1073,

September 2003.

Honour, E. C., "Toward an Understanding of the Value of Systems Engineering", First Annual Conference on Systems Integration, Hoboken, NJ, 2002.

Kemerer, C., "An Empirical Validation of Software Cost Estimation Models", Communications of the ACM, Vol. 30, No. 5, pp. 416-429, 1987.

Military Handbook for Design, Construction, and Testing Requirements for one of a kind Space Equipment, DoD HDBK 343,01, February, 1986.

Military Standard for Defense System Software Development, DoD-STD-2167, 4 June 1985, and DoD-STD-21 67A, 27 October 1987.

PRICE-H, "Your Guide to PRICE-H: Estimating Cost and Schedule of Hardware Development and Production", PRICE Systems, LLC, Mt. Laurel, NJ, 2002. PRICE-S, "Your Guide to PRICE-S: Estimating Cost and Schedule of Software

Development and Support", PRICE Systems, LLC, Mt. Laurel, NJ, 1998. SSCM Pro edition, The Aerospace Corporation, 2002.

USCM8 Knowledge Management System, Tecolote Research, Inc., 2002.

I$" Annual Forum on COCOMO and Software Cost Modeling

(16)

Acknowledgements

The authors would like to thank those who contributed to the manuscript as references or reviewers specifically: Daniel Ferens (Air Force Research Labs), Yue Chen (USC- CSE), Marilee Wheaton (The Aerospace Corporation), Dave Hickman (The Aerospace Corporation), Carl Billingsley (The Aerospace Corporation), Karen McRitchie (Galorath), Elaine Lim (The Aerospace Corporation), Larry Sidor (The Aerospace Corporation), Daniel Nigg (The Aerospace Corporation) and Arlene Minkiewicz (PRICE Systems).

Biographies

Ricardo Valerdi

Ricardo is a Research Assistant at the Center for Software Engineering and a PhD

I

student at the University of Southern California in the Industrial and Systems Engineering department. His research is focused on the cost modeling of systems engineering work. While completing a Masters degree in Systems Architecting & Engineering at USC he collaborated in the creation of COSYSMO (Constructive %terns Engineering Model). He earned his bachelor's degree in Electrical Engineering from the University of San Diego. Ricardo is currently a Member of the Technical Staff at the Aerospace Corporation in the Economic & Market Analysis Center. Previously, Ricardo worked as a Systems Engineer at Motorola and at General Instrument Corporation.

Michael Ernstoff

Michael Ernstoff, formerly with Raytheon Systems and Hughes Aircraft Company. Mr.

I

Ernstoff retired from Hughes Aircraft Company' Electro-Optical and Data Systems Group in 1996 after spending over 30 years engaged in the development of traveling wave tubes, active-matrix liquid-crystal displays, and tactical and strategic thermal imaging systems. Shortly after retiring, he returned to the same but renamed organization, Raytheon Systems, to complete the formulation of an Excel based parametric estimation tool for predicting system engineering resource needs on complex sensor technology development programs. Since those tool development efforts were substantially completed two years ago, the tool has reportedly been used with surprisingly outstanding success as a sanity check on several major bottom's up estimates. Mr. Ernstoff earned his Bachelor's Degree in Electrical Engineering and his Master of Science degrees from Cornell University. He is a Senior Member of the IEEE, principle inventor on over a dozen patents, and has presented papers at national conferences hosted by the IEEE (Institute of Electrical & Electronic Engineers), SID (Society of Information Display), AlAA (American Institute of Aeronautics and Astronautics) and INCOSE (International Council cf System Engineers).

ldh Annual Forum on COCOMO and Software Cost Modeling Page 16 of 16

(17)

Paul Mohlman

Mr. Mohlman is in the Cost and Requirements Department at The Aerospace

I

Corporation, which is the corporate focal point for space systems cost analysis and estimation. His experience spans over 30 years in the aerospace and petroleum industries and has previously worked at TRW, Rockwell, Texaco, Getty Oil, Mobil and Halliburton. Mr. Mohlman received his Bachelor of Science in Mechanical Engineering from the California State University, San Luis Obispo, California. He is a life member of the United States Air Force Association, since 1991. Mr. Mohlman has issued and presented reports on cost, schedule, systems engineering, launch vehicle integration, strategic planning, launch vehicle and satellite systems readiness, component failure assessment, space systems database, satellite orbital experience, refinery capital project costs, oil-shale synthetic fuel facility costs, and management of oil company refining and production subsidiaries.

Don Reifer

Donald Reifer is one of the leading figures in the field of software engineering and

I

management with over 30 years of experience in academia, industry and government. He is currently President of RCI, a firm specializing in software security and anti- tampering research.

Evin Stump

Mr. Evin Stump is a Senior Consultant for Galorath Incorporated. His experience as an

I

engineer spans 50 years in the aerospace industry, the last 25 of which have been primarily in engineering cost analysis and modeling. His undergraduate engineering studies were at Loyola University of Los Angeles. He earned the MS in Operations Research from the University of Texas at Austin, and the Professional Designation in Government Contract Management from UCLAINCMA. He is a former president of the Southern California chapter of the International Society of Parametric Analysts and has written several award winning papers for that and other professional organizations.