Gaps in Schedule Prediction Models - Predicting Schedule Delay Caused By Errors During Software

Chapter 1: Introduction

1.5 Gaps in Schedule Prediction Models

Historically, SW Development resource estimation models are often inaccurate and require multiple updates that are made without a reliable predictive tool. Accurate schedule prediction is limited and (Boehm & Valerdi, 2011) estimate that the accuracy

of the initial resource estimates to be as low as 39% when compared to the actual schedule performance.

Most existing SW prediction models are used to estimate errors for the individual system, which is important to managing the development of that system. One researcher (Mende, Koschke, & Peleska, 2011) used case study analysis to assess the usefulness of SW defect prediction models that focus on identifying errors in avionics component and concluded that 50% of the component/system errors were not identified by the model.

While system level models have a role in schedule estimation, they lack the ability to fully address the integration of several systems and the impact the resulting integration errors have on schedule delay.

Few schedule estimation models target the SWI phase (Madni & Sievers, 2014).

This may be due to the limited amount of data available to assess SWI since the data regarding errors is often proprietary. While it is promising that new approaches to models for managing complexity are emerging as discussed further in section 2.2, most of the current research does not provide data or enough use cases to assess the accuracy or validity of the prediction model. Additionally, these models require significant amounts of data, and do not offer a simplistic approach that users understand and are more apt to use. A 2013 survey of 50 countries concluded that predicting schedule is the main

concern of managers (Lopez-Martin, 2015). As discussed in Section 2.2, there is a gap in existing models that provide a predictive tool to adjust schedule during SWI or other phases of SW Development based on relevant factors. Further discussion in Section 2.11 will show that the NBM is a viable option to fill the gap as an accurate schedule

estimation tools for the SWI phase.

13 1.6 Research Scope and Limitations

This study focused on a novel approach to develop features for a NBM that accurately predicts schedule delay. The research targeted three key objectives.

1. Develop and validate the accuracy of the NBM to predict the schedule integration delay created by SWI errors.

2. Develop the NBM features through a Literature survey of recent SWI Challenges.

3. Determine the contribution of each feature to the predicted delay based on the most accurate model.

The assumptions and limitations of this dissertation research are provided as:

1. Each error report is considered an independent use case.

2. The Army SWI approach that produced the error reports is “big bang” that is executed at the end of the SW development cycle and does not include phased integration approaches.

3. The focus of this study is not the decision process to include whether the predicted delay is acceptable to the manager.

4. The data used to support this study has been sanitized so that systems are anonymous however; the facts for the actual data elements were not modified.

1.7 Summary of Dissertation Organization

The remainder of this research is presented with the fllowing organizational structure:

Chapter 2 provides the Literature Review that includes a review of 30 sources of SWI Challenges that provide insights in to the uncertainty found during SI and SOSI.

Additionally, a review of literature regarding current models related to SW development with a focus on schedule is included. Other topics included in the review are a

discussion on Systems Thinking and Knowledge Integration as fundamental concepts for defining SWI Challenges that serve as features for the NBM. The remainder of the review focused on defining and assessing the NBM theoretical framework and the use of the Cross-Industry Standard Process for Data Mining (CRISP-DM) process as the basis for model development.

Chapter 3 is the Research Method that included data collection, development of a Learning database and the NBM development. Key to the NBM development is the use of feature engineering to select the NMB features that result in the most accurate model.

A discussion of the different measures of accuracy includes the prediction set and global accuracy. The confusion matrix and other measures that are unique to classification models are used to assess the accuracy on the most relevant integration delay intervals (IDI).

Chapter 4 includes the results of NBM variations based on the feature selection process. This chapter also provides the accuracy assessment of each NBM and the contribution of each feature in the most accurate model. Finally, Chapter 5 includes the Conclusions of the research and recommended Future Work.

Chapter 2 – Literature Review

This dissertation research focused on Software Integration (SWI) testing activities and the impact errors that are routinely found during this phase of system development have on schedule. The literature review presents a body of relevant work from the past ten years that focuses on three important areas: SWI environment, SWI challenges, and NBM Development with discussion on factors important to a NBM. These focus areas define the difficulties with schedule prediction in the SWI environment that includes a discussion on challenges inherent in a SoS environment.

The literature review begins with a discussion of the SWI environment (Section 2.1) and the difficulties it presents to managers. Next, a review of current SW

development estimation models (Section 2.2) is provided with a focus on efforts that recognize uncertainty with the differing approaches for increasing accuracy in predicting the schedule during the SW Development. The different models reviewed include Fuzzy logic models (FLM), parametric models (PM), simulation and probabilistic models as representative of the uncertainty that is prevalent in SW development estimation. The role of Systems Thinking (Section 2.3) and Knowledge Integration (Section 2.4) are reviewed as concepts that support understanding prediction during SWI. A survey of journal articles is used to determine the SWI challenges (Section 2.5) routinely

experienced during SWI with the intent of developing NBM features. Finally, a review of literature that defines the important aspects of a Bayesian Probabilities (Section 2.8) and NBM Development (Section 2.9) process are discussed.

16 2.1 SWI Environment

The SWI environment is the culmination of the SW Development phase where multiple systems that are required to operate as a unified capability are integrated and tested. The SWI environment includes the software systems and the associated hardware designated for integration, the project managers and system engineers, the testers with their associated test equipment, and the user. Generally, legacy and new systems participate in the integration event. According to VanMoll (2008) a survey of over 60 complex projects showed that 50% of development project leaders see the system

integration as “problematic or an extremely difficult phase.” SWI typically occurs at the end of the SW Development phase and some estimate it utilizes 40% of the resources (Steindl & Mottok, 2012b) required for SW Development. Many of these resources are utilized to support root cause analysis and problem resolution when an error occurs.

Often the systems engineers and project managers or system owners are collaborating for the first time, resulting in a PRC that is frequently underestimated or approached by managers that are not well equipped to predict the extent and impact the errors will have on the schedule. This dissertation researches how managers can represent the SWI

environment challenges as NBM features to predict schedule impact after an error occurs.

The general proposition is that SWI challenges: (1) create errors that cause schedule delays that can be predicted by a NBM and (2) enable feature selection through

knowledge integration. The relationships between SWI challenges, SWI Errors and the development of a NBM are depicted in Figure 2.

Figure 2 - SWI Environment Relationships to NBM

2.2 Existing SW Development Estimation and Prediction Models

Researchers that focused on the SW development resource estimation models including schedule, generally focused on mitigating and understanding the role of uncertainty. These studies use different approaches to define the causes of uncertainty and propose different prediction techniques to improve traditional estimation models.

The goal of most of these models is focused on individual software development where most of the expertise and system understanding is found (Madni & Sievers, 2014).

However, these SW development prediction tools do not recognize the unique challenges that occur during the integration phase. Many linear models are limited in their accuracy because of the nonlinear relationship as well as the multiple dynamic and interrelated dependencies within a SOS. However, there are models that are intended to provide better accuracy in complex environments such as SW integration. A review of

representative models is reviewed below to show current approaches to prediction in the SW development environment.

2.2.1 Fuzzy Logic Model Using Function Points

One category of prediction models uses function points (FP) to improve schedule prediction in complex environments. López-Martín, Chavoya and Meda-Campaña (2015) proposed a FLM using adjusted function point (AFP) as an independent variable to estimate new project size with the intent to improve the development schedule prediction. The AFP is a composite variable that includes nineteen independent variables: internal logical files, external inquires, and fourteen characteristics (López-Martín et al., 2015) that are specific characteristics of the project. The resulting mathematical model was validated through 20 SW projects as use cases based on their mean absolute residuals (MAR = actual schedule minus predicted schedule). The research concluded that the MAR for the FLM is more accurate at 33% when compared to the 29% accuracy of simple linear regression (SLR). Similar assessment of FPs calibrated by FLM (Kaushik, 2013) also reported small but promising improvements over traditional SW development estimation. Kaushik (2013) used artificial data to reach these conclusions.

2.2.2 Probabilistic Models

Bayesian Belief Networks (BBN) are also used for schedule prediction to quantify schedule risk. BBN is a directed acyclic graphical (DAG) method for assessing the relationships between elements in a probabilistic manner. Each BBN nodes can have multiple parents and multiple child nodes resulting in a complex set of relationships. For this reason, BBNs heavily rely on expert judgment, surveys, and interviews to establish the variables and their relationships that are essential to the accuracy of the BBN (Luu, Kim, Tuan & Ogunlana, 2009). BBNs also require a significant amount of data domain

expertise regarding multiple relationships. A 2014 study (Misirli & Bener, 2014) shows that most BBNs lack data and the majority depend solely on expert knowledge that described determination of the conditional probabilities as time consuming and difficult to determine. The same survey shows BBNs used for SW engineering primarily focused on system level fault prediction (Misirli 2014). There was no percentage provided for schedule prediction explicitly, however, 27% of survey respondents reported SW Engineering Management as the prediction subject that includes schedule and other resource estimations factors.

NBM offers a simpler approach as a unique BBN that has one parent (target) and multiple child nodes (features or predictors) while retaining the accuracy of the BBN.

The NBM looks only at the dependencies between the target and predictors, which simplifies the model and the understanding required to analyze the results. Also, there is evidence that NBMs can be successfully constructed from data and not rely on expert judgment that is typically used to create decision tools (Loutchkina et al., 2014). Because of its simplicity, NBM is widely used in different environments. Defect prediction

models provide an example of NBM that are aimed at software component reliability (Steindl & Mottok, 2012a) during the SW development cycle and do not specifically focus on errors created during the SW integration and test phase (Mende et al., 2011).

Further discussion on NBM and the value of Bayesian probabilities in predicting in complex environments is provided in Sections 2.6 through 2.9.

20 2.2.3 Parametric Models

Parametric models with modifications to accommodate the complexity most system integration venues now experience have shown promise. Loutchkin (2014) provides a parametric model (PM) that estimates the technical risk that occurs during integration. This model combines a parametric model with Bayesian Belief Network (BBN) to understand and predict the potential risks a system will face for a given

integration environment. To provide the conditional probabilities required for the BBN, the researchers (Loutchkina et al., 2014) chose to use PMs for this purpose due to the unavailability of historical data to provide the node probabilities. This approach (Loutchkina et al., 2014) focuses solely on the technical risk and does not discuss the potential schedule impact that is related to the interdependencies that occur during SWI, nor are the project and organizational related factors considered. The BBN represents the probabilistic relationships between 69 nodes that connect to while the PMs define risks specific to a project. One project use case was used to evaluate the process for the

modeling concept with no accuracy measure provided. Additional discussion on BBNs is discussed in Section 2.2.4 below.

2.2.4 Simulation Models

A review of research on simulation models to support schedule estimation shows a focus on process. (Mizell & Malone, 2007) provides a method to assess the

uncertainty throughout the lifecycle of the project by focusing on the three parameters that are key to the initial SW cost estimation (that includes the schedule). The outcome presented by (Mizell & Malone, 2007) will provide the manager with an understanding of the magnitude of the uncertainty at each phase of the development to include

integration. This information will help the manager make better estimates as (Mizell &

Malone, 2007) showed through the assessment of the model on two projects. However, there is no evidence that this initial estimate is sufficient to support schedule adjustment during SW integration.

In other research, (Houston, 2014) simulation was used to propose a method using discrete event simulation to forecast the duration of test-fix-test (TFT) cycles during SW development (Houston, 2014). The model required inputs regarding the constraints of the test environment such as number of test beds, number of errors in the queue and modeled the rework cycle given these constraints. The TFT work is focused on process and

technical parameters, while the proposed NBM in this dissertation approaches schedule prediction through machine learning of historical data and does not explicitly consider the process. The TFT discrete simulation can also be considered complementary to the

proposed NBM since both are attempting to bring attention and analysis to this important phase of SW development. The accuracy of the TFT model ranged from 49-79%;

however, the independent variables used for the TFT model is not applicable to the SWI environment.

2.3 Systems Thinking

Systems thinking provides the mental model and insights into the holistic

environment (Lamb & Rhodes, 2009) that supports understanding the SWI environment.

The SWI environment is important to support development of variables that contribute to the delay created by the iterative nature of the PRC. Systems Thinking contributes to the recognition that solutions in complex environments are influenced by factors that include technical (HW/SW System) and organizational factors (Madni & Sievers, 2014).

As shown in Figure 3, these relationships and interdependencies are exhibited as the SWI Challenges that are discussed in Section 2.5.

Figure 3 - SWI Environment in Systems Thinking

Felder (2012) attributes the increased system complexity experienced in the aerospace industry to integrated systems capabilities that necessitates collaboration as key to systems thinking. Lamb and Rhodes (2009) also discuss collaborative system’s thinking as an essential component of resolving and managing SWI due to multiple dynamics that are at work in complex environments where interdependent systems are required to meet the user’s integrated capabilities requirements (Lamb, 2010). Lamb (2010) concludes that complexity has both organizational and technical implications that must be considered to solve critical issues and errors. During the Literature Survey, as the data was extrapolated from existing problem reports, attention was given to both types of variables as potential causal factors that contribute to the integration delay.

System’s Thinking SWI Environment

HW/SW Systems

Project Management

Influences Subject Matter Expert

Testers

System Integration

Factors

Langford (2012) advocates for considering systems thinking during system integration as the way to bring context to SI data. By viewing SWI through the context of systems thinking, the feedback loop between the integration event and the problem resolution cycle provides insights into the factors that contribute to the schedule delay. Balancing this “paradoxical nature” of the SoS is rooted in SoS thinking (B. J. Sauser, Reilly, &

Shenhar, 2009). SoS Thinking is defined as derivation of systems thinking which offers a methodology that considers the holistic rather than the individual (B. Sauser &

Boardman, 2008).

According to researchers (Felder, 2012; Lamb & Rhodes, 2009; Langford, 2012) tystems thinking provides the rationale defining the boundaries of system integration that include the organizational influences beyond the physical integration event. By looking at the expanded environment, the PRC must include the technical product and relevant organizational and project management factors. Allowing for both acknowledges the complexity reflected in the combination of factors that create the delay.

2.4 Knowledge Integration

For this dissertation, knowledge integration is used to demonstrate the combined impact of sources that are used to define the features for the NBM. Knowledge

integration is the “combination of existing knowledge to create new knowledge” (Sargis-Roussel & Deltour, 2012). For system integration, knowledge discovery is a key

component of data preprocessing to leverage various sources that support understanding influences in the SoS that impact schedule. The accuracy of NBM, similar to other models, is heavily dependent on these features (Strumbelj & Kononenko, 2014) that represent the independent variables in the NBM.

24 2.4.1 Knowledge In A SoS Environment

Knowledge is often dispersed in a SoS environment (Patnayakuni, Arun Rai, &

Tiwana, 2007) and thus requires merging disparate knowledge sources to accumulate actionable information. In order to transform the raw data from the Army SW Integration event into features that can be used for prediction, the SWI challenges presented in a focused literature survey were used to extract information that maps to one or more SWI challenges. Information sources external to the event were also included as features to further enable the model, which aligns with the systems thinking that recommends expanding the system boundaries to more accurately reflect (Rebovich, 2006) those influences on the prediction. Knowledge integration for this research relied on literature survey to establish SW integration challenges that are then used to elicit knowledge from historical data and external sources to create the features for the NBM.

Knowledge integration also includes the intent to institutionalize the newly created knowledge, which for this research resulted in a learning database for the model.

The combined knowledge from the literature survey, raw data preprocessing, and external information sources were used to create the training database. The external data consists of acquisition category for the system with the error that is an indicator of the program’s programmatic influences on the delay based on the acquisition category. The acquisition category is assigned by the acquisition executive and provides the level of oversight for the program (DOD directive 5000.1, 2015).

2.4.2 Feature Engineering and Feature Selection

Feature Engineering (FE) is a critical preprocessing step in Knowledge Discovery and Data Mining (KDDM). The feature engineering for this research is a cycle that

results in feature selection that improves the accuracy of the model based on analysis thresholds set for features as shown in Figure 4. Research has shown that additional nodes increase complexity of the NBM but do not necessarily increase accuracy (Holte, 1993).

Figure 4 - Feature Engineering Cycle

While there is emerging work that investigates automating this process, FE is still primarily manual. The intent of FE is to transform raw data into derived that are

meaningful to the domain being modeled features (Davis & Foo, 2016). FE typically requires domain knowledge to initiate the features followed by a process to trim features based on a constraint regarding strength of the contribution each feature has toward the

In document Predicting Schedule Delay Caused By Errors During Software Integration. by Kelly Dula Alexander (Page 26-0)