System of Systems Lead
System Integrators: Where
Do They Spend Their Time
and What Makes Them
More or Less Efficient?
Jo Ann Lane1, * and Barry Boehm21Center Systems and Software Engineering, University of Southern California, 941 West 37th Place, SAL Room 328, Los Angeles, CA 90089-0781
2Center Systems and Software Engineering, University of Southern California, 941 West 37th Place, SAL Room 326, Los Angeles, CA 90089-0781
SoS LSI: WHERE DO THEY SPEND THEIR TIME AND THEIR EFFICIENCY
Received 13 June 2007; Accepted 7 September 2007after one or more revisions Published online in Wiley InterScience (www.interscience.wiley.com).
DOI 10.1002/sys.20085
ABSTRACT
As organizations strive to expand system capabilities through the development of system-of-systems (SoS) architectures, they want to know “how much effort” and “how long.” In order to answer these questions, it is important to first understand the types of activities performed in SoS architecture development and integration and how these vary across different SoS implementations. This paper provides preliminary results of research conducted to deter-mine types of SoS Lead System Integrator (LSI) activities and how these differ from the more traditional system engineering activities described in EIA 632 (Processes for Engineering a System). It also looks at concepts in organizational theory, complex adaptive systems, and chaos theory and how these might be applied to SoS LSI activities to improve success rates and efficiency in the development of these “very large” complex systems. © 2007 Wiley Periodicals, Inc. Syst Eng
Key words: Lead System Integrator; System of Systems
*Author to whom all correspondence should be addressed (e-mail: [email protected]; [email protected]).
Systems Engineering © 2007 Wiley Periodicals, Inc.
1. INTRODUCTION
As organizations strive to expand system capabilities through the development of system-of-systems (SoS) architectures, they want to know “how much effort” and “how long.” Efforts are currently underway at the Uni-versity of Southern California (USC) Center for Sys-tems and Software Engineering (CSSE) to develop a cost model to estimate the effort associated with system of systems engineering (SoSE) activities. The research described in this paper is in support of the development of this cost model, the Constructive SOS Integration cost MOdel (COSOSIMO). Research conducted to date in this area has focused more on technical charac-teristics of the SoS. Feedback from USC CSSE industry affiliates indicates that the extreme complexity typi-cally associated with SoS architectures and political issues between participating organizations have a major impact on the SoSE effort. This is also supported by surveys of system acquisition managers [Blanchette, 2005] and studies of failed programs [Pressman and Wildavsky, 1973].
Once a collection of integrated systems is designated as an SoS, an organization is established to oversee, coordinate, and guide the evolution of the SoS using SoSE processes. This organization may be an SoS Lead System Integrator (LSI), a prime contractor, or a gov-ernment team working in conjunction with support contractors. This paper further investigates effort and schedule issues on “very large” SoS programs managed by LSIs and strives to determine organizational charac-teristics that significantly impact overall success and productivity of the program. The goals of this investi-gation are to:
1. More clearly define the LSI activities for which the COSOSIMO model will estimate effort and show how they differ from the more traditional system engineering activities described in EIA 632 [EIA, 1999].
2. Identify factors from organizational, complexity, and chaos theory that may apply to and affect the efficiency of LSI activities.
2. BACKGROUND
The first step in developing a cost model is to describe the effort you are trying to estimate as well as the characteristics of the activities that you are trying to estimate. Key to the COSOSIMO cost estimation model is understanding what an SoS is, how it is different from the more traditional system, and what are the major sources of effort.
2.1. What Is an SoS
An SoS is generally described as the integration of a set of new and existing systems that each have their own purpose. An initial survey of the literature showed that there are a considerable number of definitions of SoSs, with the earliest definitions appearing in Berry [1964] and Ackoff [1971]. Many today seem to be converging on the definition in [Maier, 1998, page 271]: “A system-of-systems is an assemblage of components which indi-vidually may be regarded as systems and which possesses two additional properties: operational pendence of the components and managerial inde-pendence of the components.” However, one still finds that large, complex systems are sometimes referred to as SoSs, even though they do not possess the two additional properties specified in the Maier definition. An example of this is a military aircraft that contains a flight control system, an avionics system, and weapons systems. While these aircraft components may be de-veloped by vendors other than the aircraft manufacturer (managerial independence), they typically cannot oper-ate or have their own purpose outside of the aircraft (operational independence). In addition, the Depart-ment of Defense (DoD) SoS SE Guide [DoD, 2006] describes five types of SoSs within the DoD: new development, development of new capability by inte-grating current systems, mixed system maturity levels, sustaining capabilities/engineering, and business sys-tems. It became clear during this SoS definition survey that it would be extremely difficult, if not impossible, to develop a single cost model that could handle all of the various types of SoSs. Therefore, the SoS cost modeling team attempted to define a set of SoSs that would benefit from a cost model [Lane and Valerdi, 2007] and that could be used to guide the development of an initial COSOSIMO. After reviewing the various SoS concepts, Lane and Valerdi [2007] concluded that the SoSs of most interest to the cost model development effort had the following characteristics:
• Economically-Oriented SoS Stakeholders:For an SoS cost estimation model to be of value, it is important that there exist strategically oriented organizations that have a need for this type of information. These organizations include clients that will pay for the system and signoff on major milestones as well as user communities that will be responsible for SoS operation and sustain-ment. This characteristic is important for the de-velopment of the SoS cost model since the success of COSOSIMO depends upon historical data from motivated stakeholders to support model calibration and validation.
• SoSE Responsibilities and Processes:The pro-gram must identify a single LSI team, prime contractor, or government team that is responsi-ble for the definition of the SoS architecture and the total component system integration and test activities at the SoS level. It is important that this organization have complete technical oversight
over the entire SoS and SoS component suppliers, as well as the engineering processes at the SoS level. This ensures that there is an organization responsible for defining an overall architectural vision for the SoS and maintaining the architec-tural concepts throughout the development, inte-gration, and test phases. This is in contrast to SoSs that are more ad hoc in nature and based on much shorter-term strategies that can change frequently due to business needs and performance.
• Component-Based SoS Architecture: The SoS is primarily the integration of a set of component systems that work together to provide capabilities not provided by any single component system within the architecture. The key architecture fea-ture in an SoS is the framework that supports the integration of existing as well as new component systems. The characteristics of this architecture determine the ease/difficulty in integrating SoS component systems in both initial development and dynamic operational scenarios.
• Component System Independence: Each of the component systems envisioned within the SoS must be independent with respect to develop-ment, maintenance, and operation. This provides for clear boundaries for the various cost models that can be used to estimate total SoS develop-ment costs. This view, focusing on the two key aspects identified in the Maier SoS definition, minimizes the potential overlap between the SoS cost estimation model for SoSE activities and the system engineering and software cost estimation models used to estimate the implementation of SoS-enabling functionality in the component sys-tems.
Many SoS architectures are described as net-centric architectures that allow the component systems to ex-change information and perform tasks within the frame-work that they are not capable of performing on their own outside of the framework. This is often referred to as an “emergent behavior.” Key issues in developing an SoS are the security of information shared between the various component systems, how to get the right infor-mation to the right destinations efficiently without over-whelming users with unnecessary or obsolete
information, and how to maintain dynamic networks so that “nodes” can come and go.
The example in Figure 1 illustrates the types of enterprise capabilities that an organization might at-tempt to integrate through an SoS. This SoS, a regional area crisis management system, is designed to inform and support coordination of activities for crisis respond-ers in the field. For example, the sheriff’s deputies might connect laptop computers in their cars to the SoS via a mobile ad hoc network. Through this connection, they can receive detailed information from various county agencies. This might include county maps showing property boundaries, utility easements and locations, floor plans for buildings of interest, wants and warrants information on suspected criminals that include photographs of the suspect, and live feeds from helicopter cameras in an area of interest.
2.2. SoS LSI Activities
Based on information gleaned from LSI statements of work, most of the LSI effort for new developments focuses on defining the SoS concepts and overall archi-tecture; conducting source-selection activities to ac-quire component systems to be integrated into the SoS framework; managing changes to the SoS-level archi-tecture, communications requirements, and design; tracking changes to the component systems that are
Figure 1. Sample SoS. [Color figure can be viewed in the online issue, which is available at www.interscience. wiley.com.]
happening in parallel with the SoS development activi-ties; and integrating and testing the SoS component systems at the SoS level. Because of the complexity and changing user needs associated with these SoSs, they are often developed using an incremental or spiral de-velopment process that spans many years. Activities similar to those for new developments are performed to evolve the SoS over time to accommodate new user needs and changing technologies. However, in the on-going evolution of the SoS, there is less emphasis on source selection and more emphasis on negotiation with and coordination of the component system suppliers.
3. HOW DO LSI ACTIVITIES DIFFER FROM TRADITIONAL SYSTEM ENGINEERING ACTIVITIES
The question often asked is, “how do the COSOSIMO LSI activities differ from the more traditional system engineering activities?” To better explain these differ-ences, SoS LSI activities have been analyzed [Lane, 2005] with respect to the types of issues LSIs typically face and to the applicable process areas in the Software Engineering Institute (SEI) Capability Maturity Model Integration (CMMISM)1 project management and engi-neering process categories described in SEI [2001]. The results of this analysis are described in the following paragraphs.
3.1. LSI Activity Overview
Relatively recent observations of very large LSI pro-grams [Boehm, 2005; Lane, 2005b] have attempted to identify LSI key activities. During the bid and proposal activities for an LSI SoS contract or soon after an LSI SoS contract has been awarded, the LSI team quickly begins to concurrently define the scope of the SoS, plan the activities to be performed, analyze the requirements, and start developing the SoS architecture or framework. As the scope, requirements, and architecture start to firm up, the LSI begins source selection activities to identify the desired component system suppliers. As the suppliers start coming on board, the LSI organization must focus on teambuilding, re-architecting, and feasi-bility assurance with the selected suppliers. Teambuild-ing is critical since the LSI organization(s) and the selected suppliers have often been competitors in the past and now must work together as an efficient, inte-grated team. Re-architecting is necessary to make ad-justments for the selected component systems that may not be compatible with the initial SoS architecture or other selected components, and feasibility assurance is
conducted to better evaluate technical options and their associated risks. Many of the technical risks in an SoS framework are due to incompatibilities between differ-ent compondiffer-ent systems or limitations of older compo-nent systems with today’s technology.
As the SoS development teams begin to coalesce, the LSI focuses on incremental acquisition activities for the development and integration/test of the required com-ponent systems for the SoS. During this process, there are often continuous changes and new risks that must be managed. In addition, the LSI is continuously look-ing for opportunities to simplify the SoS architecture and reduce effort, schedule, and risks. Key management issues for the LSI include:
• Number of stakeholders—The stakeholders in an SoS development effort are numerous. They come from sponsoring and funding organizations as well as the various user communities that have high expectations for the planned SoS.
• Number of development organizations—Be-cause the component systems are often “owned” by an organization other than the sponsoring or LSI organizations, there is often a separate devel-opment organization associated with each com-ponent system in the SoS. In addition, some of the component systems can be systems of sys-tems in their own right. This means that there may be lower level suppliers associated with each component system, adding to the number of de-velopment organizations.
• Number of parallel, independent (or not so inde-pendent) developments—The hope in an SoS development effort is that the overall SoS archi-tecture can be defined well enough up front to allow for many concurrent development efforts in parallel. However, there can be compatibility and commonality issues between component systems and what were initially thought to be independent developments become dependent developments, often affecting integration and test activities. As an example, a slip in schedule for one component system may mean that several other component systems cannot perform their planned integration and test activities for a given increment/build. Annette Krygiel describes this issue [Krygiel, 1999, page 47] and provides case studies that illustrate it well: “[W]hile integrating a single large scale system is hard, the integration of a set of systems independently developed and oper-ated for distinct missions is harder… As such, these differences have resource, schedule, per-formance, as well as management implications.”
• Number of decision “approvers”—Studies [Blanchette, 2005; Pressman and Wildavsky, 1973] have shown that as the number of people involved in the decision-making process in-creases, the probability of getting a timely (or even any decision) often decreases. In the SoS development arena, the SoS stakeholders, the component system “owners” as well as the LSI organizations are often all involved in making key decisions. Also, because the component sys-tems are independently managed and can operate independent of the SoS, there can be conflicts between SoS and component system needs, lead-ing to prolonged negotiations and non-optimal compromises.
• Cross-cutting risks—These are risks that cut across organizational boundaries and/or compo-nent systems (as opposed to compocompo-nent system risks that can be managed by the component system supplier).
A key characteristic of the SoS that can impact its success is the fact that the component systems within an SoS are often “owned” by another organization. This means that SoS timelines are often controlled by other “outside” goals and timelines. SoS-enabling features are often incorporated into SoS components along with the other enhancements and features planned by the component “owner.” There may be long-lead enhance-ments that are not required by the SoS architecture/sys-tem, but are more important to the component owner or user organization and will delay implementation of the SoS features. Also, these other on-going changes (not required for the SoS) may impact the stability of the component (including its architecture). A current exam-ple of this is with the Future Combat System (FCS) program: the limited resources and specialists avail-able to develop new features needed to support to-day’s Iraq operations (FCS spin outs) vs. features needed to support the FCS SoS of the future. While this may be perceived as more of a schedule issue, it can also impact effort since component delivery de-lays can result in inefficient integration activities and significant rework.
Key LSI activities have been the focus of several USC CSSE workshops conducted with industry affili-ates. The key activities identified in these workshops were compared to the CMMISM process areas. The following sections describe the results of the compari-sons in both the management and technical process areas.
3.2. Key LSI Activities in the CMMISM Project Management Process Category The CMMISM Project Management process areas key to LSI efforts are project planning, project monitoring and control, supplier agreement management, inte-grated project management, risk management, inteinte-grated teaming, and quantitative project management. As LSI organizations try to scale up their traditional system engi-neering processes, they find that there are often new and unexpected issues. Typical issues include:
• Traditional planning and scheduling may lead to unacceptably long schedules, requiring the LSI to be more creative in both their technical and im-plementation approaches.
• Planning and tracking activities must integrate inputs from a variety of different organizations, each with its own (and probably different) proc-esses.
• Traditional oversight and coordination can spread key LSI personnel too thin.
• More emphasis is required for contracting and supplier management and oversight. Incentives are often needed to better align priorities and focus of the component system supplier organi-zations. In addition, contracts must provide mechanisms to allow suppliers to participate more in the change management process to help assess impacts and to develop efficient ap-proaches to proposed changes.
• Standardization of all processes may be over-whelming. The LSI needs to decide what to stand-ardize and what to let the suppliers control.
• The decision-making process involves consider-ably more organizations. As mentioned above, this can make the decision making process much more complex and time-consuming and it may have significant impacts on the overall schedule and effort.
• Risk management for cross-cutting risks needs to cross organizational boundaries. It is important that risk management activities for cross-cutting risks don’t select strategies that are optimal for one area of the SoS, but are to the detriment of other areas.
3.3. Key LSI Activities in the CMMISM Engineering Process Category
The CMMISM Engineering process areas key to LSI efforts are requirements development, requirements management, technical solution, product integration, verification, and validation. In the SoS LSI environ-ment, there is a definite change in the focus of these engineering activities. For example:
• Requirements managed or controlled by the LSI are primarily at the SoS level and only address the component systems with respect to their integra-tion into the SoS framework/architecture.
• The LSI needs to know when to system engineer and when not to system engineer. When SoS architecture conflicts or incompatibilities are identified, tradeoffs must be conducted to deter-mine if it is better to accommodate the conflict or incompatibility at the SoS framework level or to require the supplier(s) to make component sys-tem changes.
• SoS technical solution, product integration, veri-fication, and validation focuses primarily on the communications between the component sys-tems.
• Other component system technical solutions, in-tegration, verification, and validation activities are the responsibility of the component system “owners.”
• LSI may or may not be responsible for actual development of component systems for the SoS. If the LSI is responsible for the development of one or more components, it may lead to perceived conflicts of interest and must be carefully man-aged. The goals of each component development must be carefully balanced to ensure the overall SoS success.
3.4. How Traditional Processes Are Adapting to the SoS Environment
Since SoS development efforts usually span many years and include many incremental or evolutionary itera-tions, there are opportunities for the LSI organizations to adapt and mature their processes to the LSI environ-ment. The following are some observations on how these organizations are evolving their processes:
• Traditional planning and scheduling: LSIs try to plan activities as independent projects that can be performed concurrently. This requires that up-front SoS architecting be performed in sufficient detail to allow subprojects to be somewhat inde-pendent of each other. It also requires that risk-driven processes be used to identify and manage risks early at SoS and sub-project levels. These risk-driven processes often include Life Cycle Objectives (LCO) reviews, Life Cycle Architec-ture (LCA) reviews, and Feasibility Rationale (FR) studies [Boehm and Lane, 2006].
• More agile processes [Lane and Boehm, 2007]: LSIs are attempting to blend traditional processes
with more agile processes. They are more agile when dealing with risk, change, and opportunity management for future increments, but plan for stabilized evolutionary increments in the near term. Key to this approach is knowing when to plan, control, and stabilize and when to be more flexible, agile, and streamlined. The agile teams are responsible for performing acquisition intel-ligence, surveillance, and reconnaissance func-tions, and then rebaselining future increment solutions as necessary.
• Competing priorities: LSIs use stakeholders to negotiate priorities with other on-going compo-nent system enhancements and maintenance.
• Project monitoring and control: LSIs attempt to manage key personnel resources in order to avoid burnout. One of the ways this is accomplished is to prioritize oversight areas.
• Integrated project management: LSIs must iden-tify key cross-cutting processes for stand-ardization and allow for flexibility in other areas. Component system and supplier organizations are allowed to use their own proven processes— they have been selected for their technical exper-tise and ability to produce.
• Decision-making process: LSIs need to reduce the number of required SoS-level decisions to the extent possible. Many organizations are involved at this level and decisions can be difficult to make because of competing priorities and the availabil-ity of the required decision makers. Decisions that are not timely can delay needed work or cause unnecessary rework. As mentioned earlier, studies [Blanchette, 2005; Pressman and Wildavsky, 1973] have also shown that the prob-ability of success of making any decision de-creases as the number of required decision approvers increases.
• Risk management: Cross-cutting risks need to be managed and balanced across system and organ-izational boundaries. Each risk needs a responsi-ble “owner.” Without “owners,” critical risks can move to the backburner and forgotten until it is almost too late. One approach is to create risk portfolios and “owners” for each portfolio to manage cross-cutting risks.
• Integrated product teams, often with repre-sentatives from the component system suppliers, typically play a much larger role and have more responsibilities.
• The people processes are at least as important as the technical processes. Personal, organizational, and political motivations and priorities can im-pact the success of the project.
3.5. Comparison to System Engineering Activities in EIA 632
To better understand “how are LSI activities different that the traditional system engineering activities,” workshops with USC CSSE affiliates have compared the list of LSI activities to those identified in EIA 632. Workshop attendees who had experience with both traditional SE and SoSE were asked to participate in a survey to better understand the relevance of the EIA 632 activities to typical SoSE activities. Table I shows the results of this analysis. As one can see, the system
engineering activities identified in EIA 632 are gener-ally applicable to SoS LSI efforts. What is different is that these activities are primarily focused at the SoS architecture level and the corresponding activities at the component system level are seldom managed, moni-tored, or performed by the LSI. Also note that none of the EIA 632 activities was identified as “not applicable” for the LSI. However, one activity, Effectiveness Analy-sis (focuses on cost effectiveness, total ownership cost, environmental impacts, etc.), was identified as only a possible LSI responsibility.
In a second workshop, USC CSSE affiliates familiar with SoSE programs were asked to review a candidate list of SoSE activities and add any that were missing or delete any that were not applicable. The responses were then consolidated and compared to the EIA 632 activi-ties. The results indicated that all of the identified SoSE activities mapped to an EIA 632 activity and no EIA 632 activity was missing from the consolidated list.
The EIA 632 activities provide good coverage of the technical activities performed by the SoS LSI, but cur-rent observations indicate that they do not adequately reflect (a) the organizational management and political negotiation activities that are a major source of effort for LSI organizations or (b) that many activities need to be tailored to focus on only the information of interest to the LSI (as identified in the discussions on CMMISM process areas). Also, current observations indicate that the types of issues that can impact the overall effort are related more to the greater SoS complexity and the number of organizations, often with competing priori-ties, that are involved in the SoS development process. These issues are often not a significant factor in smaller system engineering projects.
Further research is required in this area to better quantify the major sources of SoS LSI effort over the SoS life cycle as well as the range of impacts of the more significant effort scale factors.
4. WHAT CAN BE LEARNED FROM
ORGANIZATIONAL, COMPLEX ADAPTIVE SYSTEMS, AND CHAOS THEORIES
One of the goals in the development of the COSOSIMO model is to allow users to look at the impact of various architecture decisions and development approaches on the overall LSI effort. As we have observed LSI organi-zations mature, we have seen how their management approach and focus seems to change. And as we note these changes, we are intrigued by the idea that there are many similarities with concepts from organiza-tional, complexity, and chaos theories. Kreitman [2007] presents examples from complex systems development, of which SoS development is considered a subset [Sheard, 2006]. These theories help explain why LSIs are evolving their processes in the ways that we are observing.
By further investigating these theories, we hope to discern organizational characteristics that can help im-prove the chance of success of complex SoS develop-ment efforts and even improve the efficiency of the development effort. In this vein, research was con-ducted, and the following concepts and related organ-izational characteristics were analyzed as candidate factors for the COSOSIMO model.
Chaos theory explains and helps identify underlying order in apparently random data or events. Behaviors thought to be random may have natural boundaries or patterns. Once these boundaries or patterns are deter-mined, the order becomes apparent. This order is some-times referred to as “emergent order” [Highsmith, 2000; Kauffman, 1995, Wheatley, 1992].
So how does this apply to SoS development and LSI activities? According to Highsmith [2000], complex systems are often nonlinear, leading us to look at chaos theory. These new, never been done before, complex systems require innovation and creativity, often leading to new technologies. Because there are so many un-knowns, these tasks are difficult to predict and difficult to control.
One of many observations of interest is that we are now recognizing organizations as systems [Wheatley, 1992]. Highsmith [2000] further states that it takes a complex adaptive system (organization) to develop a complex adaptive system (or SoS). He defines a com-plex adaptive system (CAS) as an ensemble of inde-pendent agents:
• Who interact to create an ecosystem
• Whose interaction is defined by the exchange of information
• Whose individual actions are based on some sys-tem of internal rules
• Who self-organize in nonlinear ways to produce emergent results
• Who exhibit characteristics of both order and chaos (the edge of chaos)
• Who evolve over time.
To encourage innovation and creativity requires the team to be mission- or vision-driven; have both techni-cal and organizational flexibility, along with the ability to re-organize (self-organization) as new information becomes available; and work in a learning environment with a collaborative management style [Highsmith, 2000; Wheatley, 1992; Kreitman, 2007]. Sheard [2005, page 4] further states that complex systems with learn-ing capabilities can adapt until “they are poised on the edge of chaos.” It is at this point, the sweet spot between
too much control and too little control, that they have “maximum fitness relative to their environment,” al-lowing them to take in information from the outside world and quickly change themselves to react to it, and in the process, “increasing their own fitness faster than competing systems.”
By creating a learning environment for the develop-ment team, teams have the flexibility to explore critical options, resulting in innovative “emergent” outcomes. Innovative learning focuses on “what we know,” “what do we need to know,” and “what do we not understand.”
Speed and focus are important when pursuing innova-tive ideas, but it is also important to pause occasionally to allow the team to learn from increment to increment. Highsmith [2000, page 156] recommends a “go fast, pause to reflect, then go fast again, while minimizing the risk of a major wreck” way. A spiral development process would also add a step to adjust plans after the reflection in order to better refocus efforts. This all ties in with Wheatley’s recommendation to “think globally, act locally” [Wheatley, 1992, page 44]. By this she means that teams should focus more on the mission, be flexible in their approaches, and not to spend time on detailed, long-term planning—detailed, long term planning is often a waste of time in this type of environ-ment since plans often change significantly as a result of on-going learning. This leads to Highsmith’s charac-teristics of adaptive development cycles [Highsmith, 2000]: mission-driven, component-based, iterative, time-boxed, risk-driven, and change-tolerant.
5. SoS LSI SUCCESS CHALLENGES
Many of the challenges to the successful development of an SoS are not always technical in nature—they are more related to political stakeholder issues and manage-ment decisions, some of which eventually impact the technical solution. This section describes some of the key challenges and ways that are being used to mitigate them.
5.1. Organizational and Management Impacts to Success
As described in Pressman and Wildavsky [1973], there can be many issues and obstacles that can significantly affect the success of major government projects or programs. Reasons for lack of success on very large programs can include changing leaders, new (unfore-seen) organizations or players, inflation, different goals/focus between organizations/players, changing priorities, the ability or inability to shortcut (resulting in problems that can be associated with unwise short-cuts or the problems associated with not being able to cut through bureaucratic mazes in an efficient way), and incompatible goals and desires. There are not always solutions to some of these, but anticipating and manag-ing them can decrease the impact to the system or program of interest.
5.2. Candidate Ways To Improve Success Rates and Efficiency
By viewing SoS LSI teams as complex adaptive sys-tems/organizations and focusing on development ap-proaches that support innovation and learning, one can
see a match between CAS organizations and the proc-esses that are evolving in current LSI organizations:
1. Plan for risk-driven spiral processes and organi-zations along with stabilized evolutionary builds (mission-driven, iterative, risk-driven, and change-tolerant).
2. Streamline SoS-level processes to take advantage of suppliers’ own processes—this will lead to fewer steps and fewer decisions (allow suppliers to self-organize).
3. Not too fast (beware of speed problems—The LSI needs to do up-front architecting and engi-neering to get everyone moving in the right di-rection. There is also a need to develop prototypes and conduct analyses up-front to minimize technical risks and potential perform-ance problems (flexibility to explore critical op-tions). If time is not taken to perform these activities adequately, it may cause excessive re-work down the road.
4. Base program on performance, not promises. Tie key decisions to LCA simulations/models to re-duce risk (flexibility to explore critical options). 5. Have the appropriate infrastructure in place ( sup-port innovation and learning). “Infrastructure” is the services and capabilities required to support development. This includes labs, development processes, standards, and the right technical tal-ents/staff. The right resources and the right peo-ple at the right time can have a big positive impact on the program.
6. The new program must fit into arrangements that have been made with other purposes in mind, although this will increase the number of re-quired coordination points (mission driven, change tolerant). Also, one needs to be aware that component system “owners” may be evolving their systems in directions that are incompatible with SoS requirements.
Consistent with these trends, Sheard [2005] presents recommendations for moving system engineers to the edge of chaos in the areas of system architecting, de-sign, and impact assessments to support the changing environment and requirements. These include:
• Moving from the “mechanics of parts” to the “dynamics of the whole”
• Viewing elements as related instead of separate
• Interpreting organizational charts as a guide to centers of core activity and innovation instead of as the “truth” and indicators of entitlements and bureaucracy
• Responding to and influencing change instead of trying to control or minimize change.
6. CONCLUSIONS
Back to the SoSE cost model, COSOSIMO: What does this mean with respect to trying to estimate the LSI effort to architect, manage the development, and field a functional SoS? LSI organizations are realizing that if more traditional processes are used to architect and integrate SoSs, it will take too long and too much effort to find optimal solutions and build them. Complexity theory and its implications for complex adaptive sys-tems are presenting ways to potentially shorten sched-ule and effort and end up with a higher quality results. If these results are as significant as the literature indi-cates they might be, they should be reflected in the COSOSIMO estimation model. By capturing the ef-fects of these differences in organizational structure and system engineering processes in the COSOSIMO esti-mation model, management will have a tool that will better predict LSI effort and to conduct “what if” com-parisons of different development strategies.
7. FUTURE WORK
The conclusions indicate that there are some strong relationships between SoS LSI effort, organizational management, political relationships, and complex adaptive system theories. This information has been captured and documented in the COSOSIMO parame-ter set [Lane and Boehm, 2007]. However, additional data is required to determine the actual weights of the cost drivers. Additional research is planned to collect these data from SoS LSI organizations and developers and analyze it with respect management approaches, team organization and flexibility, the overall decision-making processes, and the evolution of processes and technologies from increment to increment. The results of these investigations will be used to refine the pro-posed COSOSIMO size drivers and develop an appro-priate range of values for the cost driver ratings.
REFERENCES
R. Ackoff, Towards a system of systems concepts, Manage-ment Sci Theory Series 17(11) (July 1971), 661–671. B. Berry, Cities as systems within systems of cities, The
Regional Science Association Papers, 1964, Vol. 13. S. Blanchette, U.S. Army Acquisition—the program
execu-tive officer perspecexecu-tive, Special Report CMU/SEI-2005-SR-002, March 2005.
B. Boehm, Teaching the elephant to dance: Agility meets systems of systems engineering and acquisition, Ground System Architectures Workshop (GSAW) Keynote
Ad-dress, March 2005, http://sunset.usc.edu/GSAW/ gsaw2005/s10/boehm.pdf
B. Boehm and J. Lane, 21st century processes for acquiring 21st century software-intensive systems of systems, CrossTalk 19(5) (2006), 4–9.
Department of Defense (DoD), System of systems engineer-ing guide: Considerations for systems engineerengineer-ing in a system of systems environment, draft version 0.9, Wash-ington, DC, 2006.
Electronic Industries Alliance, EIA Standard 632: Processes for engineering a system, Arlington, VA, January 1999. J. Highsmith, Adaptive software development: A
collabora-tive approach to managing complex systems, Dorset House, New York, 2000.
S. Kauffman, At home in the universe: The search for the laws of self-organization and complexity, Oxford University Press, Oxford, 1995.
K. Kreitman, From “the magic gig” to reliable organizations: A new paradigm for the control of complex systems, Symp Complex Syst Eng, http://cs.calstatela.edu/wiki/in-dex.php/Symposium_on_Complex_Systems_Engineering, accessed January 1, 2007.
A. Krygiel, Behind the wizard’s curtain, CCRP Publication Series, July 1999, http://www.dodccrp.org/files/ Krygiel_Wizards.pdf
J. Lane, System-of-systems processes, Proc CSE Annual Research Review, 2005, http://csse.usc.edu/events/2005/ arr/proceedings/presentations/Review/Session4/Lane.pdf J. Lane and B. Boehm, Modern tools to support DoD software intensive system of systems cost estimation, data and analysis center for software, (August 2007).
J. Lane and R. Valerdi, Synthesizing system-of-systems con-cepts for use in cost estimation, Syst Eng 10 (2007), 297–308.
M. Maier, Architecting principles for systems-of-systems, Syst Eng 1(4) (1998), 267–284.
J. Pressman and A. Wildavsky, Implementation: How great expectations in Washington are dashed in Oakland; or, why it’s amazing that federal programs work at all, this being a saga of the Economic Development Administra-tion as told by two sympathetic observers who seek to build morals on a foundation of ruined hopes, University of California Press, Berkeley, 1973.
SEI, Capability Maturity Model Integration (CMMI), CMU/SEI-2002-TR-001, Pittsburgh, PA, December 2001.
S. Sheard, Practical applications of complexity theory for systems engineers, Systems and Software Consortium, Herndon, VA, 2005.
S. Sheard, Systems of systems necessitates bridging systems engineering and complex systems sciences, presented at Second Annu SoS Eng Conf, Defense Acquisition Univer-sity, Ft. Belvoir, VA, July 25–26, 2006, http://sosece.org M. Wheatley, Leadership and the new science: Learning about
organization from an orderly universe, Berrett-Koehler-San Francisco, CA, 1992.
Jo Ann Lane is currently a Principal at the University of Southern California Center for Systems and Software Engineering conducting research in the area of system of systems engineering. In this capacity, she is currently working on a cost model to estimate the effort associated with system-of-system architecture definition and integration. She is also a part time instructor teaching software engineering courses at San Diego State University. Prior to this, she was a key technical member of Science Applications International Corporation’s Software and Systems Integration Group responsible for the development and integration of software-intensive systems and systems of systems. Ms. Lane received her B.A. in Mathematics and her M.S. in Computer Science from San Diego State University and is currently a Ph.D. candidate in Systems Architecting and Engineering at the University of Southern California. Ms. Lane is currently a member of INCOSE and IEEE.
Barry Boehm is currently the TRW Professor of Software Engineering, Computer Science Department, University of Southern California (USC), and Director for the USC Center for Systems and Software Engineering. Prior to this, Dr. Boehm served within the U.S. Department of Defense (DoD) from 1989 to 1992 as director of the DARPA Information Science and Technology Office and as director of the DDR&E Software and Computer Technology Office. He worked at TRW from 1973 to 1989, culminating as chief scientist of the Defense Systems Group, and at the Rand Corporation from 1959 to 1973, culminating as head of the Information Sciences Department. He entered the software field at General Dynamics in 1955. His current research interests involve recasting systems and software engineering into a value-based framework, including processes, methods, tools, and an underlying theory and process for value-based systems and software definition, architecting, development, validation, and evolution. His contributions to the field include the Constructive Cost Model (COCOMO) family of systems and software engineering estimation models, the Spiral Model and Incremental Commitment Model of the systems and software engineering process, and the Theory W (win–win) approach to systems and software management and requirements determination. He has received the ACM Distinguished Research Award in Software Engineering and the IEEE Harlan Mills Award, and an honorary ScD in Computer Science from the University of Massachusetts. He is a Fellow of the primary professional societies in computing (ACM), aerospace (AIAA), electronics (IEEE), and systems engineering (INCOSE), and a member of the U.S. National Academy of Engineering.
