Data Collection

Three sources of data were used to support this research. The data collection sources included error reports from US Army SWI events, SWI challenges resulting from a survey of literature to develop features based on interdependencies, and an external source that is related to the project management or programmatic leadership oversight for the system responsible for correcting the error. The features are the independent

variables that potentially impact the schedule prediction. Each data collection effort is discussed further to show how it can represent potential schedule impacts.

45

Figure 6 - Framework for NBM Development Process

3.1.1 Error Reports

The error reports generated from 2010 to 2015 during the U.S. Army SW Integration events provided the raw data that was mined for the development of the independent variables for the NBM. The U.S. Army SWI is a SoS event that results in a unified capability. This SoS event integrates and tests Command, Control, Computers, Communications, and Information (C4I) and Intelligence, Surveillance, and

Reconnaissance (ISR) Systems. The systems share and exchange information to build an integrated capability that can be reconfigured at the system and SoS level to provide a different capability based on the specific user’s intent. The SWI events are required to

Literature Survey Results

IDI (Days) CEC SEVERITY ACAT1 SysDepend OrgDepend priorEvent sameEvent SysType SysGroup Core

1 1 2 2 2 2 1 1 1 1 1

46

certify a set of SW for interoperability prior to fielding. A random selection of error reports from three events was provided to support this research.

Each error report is the result of an incident that created an unexpected and undesired outcome in SWI testing. The tester recorded factual information that was stored in a local database and tracked until the error is resolved. The primary purpose of the record was to document the incident with basic information to include the relevant dates and content of its resolution, and the system(s) along with the system(s) owners that are deemed responsible for the error. The resolution required consensus of stakeholders including the Project Manager, Tester, and the User. These error reports were critical to this research because they provide the basis for the knowledge integration phase that data mines each report to determine the SWI challenges and the external data so that the resulting Learning Database more accurately portrays the interdependencies within the SWI environment.

3.1.2 Literature Survey

Each of the challenges presented in section 2.5.1 through 2.5.4 provide an assessment of the interdependencies expected in the SoS created during SWI. The literature survey was used to identify factors that create schedule delay. The factors were further defined to include system interdependencies, independent management, risk, and complexity. By identifying these potential variables, their impact can be quantified through data mining the Army error reports. The full list of bibliographic information as provided in the list of references indicates those used for the literature survey of SWI Challenges. These SWI challenges when mapped to the Army data provide the features as independent variables that are used in the NBM for prediction. A matrix indicating the

47

SWI challenges documented in each of the journal articles used in the Literature Survey is provided as Appendix A.

The goal of the focused literature survey was to determine common impacts defined in journal articles and conference papers that potentially create schedule

volatility. Journal articles were surveyed for commentary regarding descriptions of SoS or System integration risks, issues or challenges that were used to develop a singular list that served as the basis for feature selection. The author, publication and list of challenges or risks the article identified were recorded. The relevant literature was limited to peer reviewed journal articles or conference papers based on the result of online journal searches. Each search focused on the period from 2006 to 2016. The original searches for SWI challenges resulted in few relevant articles that focused on SWI in a SoS

environment. It was found that the original search results provided information regarding individual system development, but did not provide details that explain the SoS focus areas that are important to the research outlined in this dissertation. Instead, the causal factors found in the complex SoS environment were the intended focus of this research and an essential component to defining the features that will determine the predicted outcome in the NBM. Therefore, subsequent searches were expanded to include the following:

• SWI challenges in a SoS environment

• SWI testing challenges in a SoS environment

• SW Resource Estimation (Focused on the SWI phase)

• SW Schedule Estimation (Focused on the SWI phase)

48

The causal factors developed from these searches were matched with the error reports to represent interdependencies that are inherent in SoS Environment. The resulting SWI challenges (Section 2.7) show the factors that create uncertainty and complexity during the integration phase of SW development that lead to SWI errors.

3.1.3 External Data Source

An external data source was used to align the system programmatic assignment to the SWI challenges resulting from the literature survey. This data point is the system acquisition category (ACAT) that is assigned to every US Department of Defense

program of record. The ACAT is assigned by the acquisition executive as defined in U.S.

DoD (DoD 5000, 2015) and provides the level of oversight for the program based on budgetary thresholds. The ACAT for each system was retrieved from the Weapon Systems Handbook (Department of Defense, 2016) and aligned with each error report.