Chapter 5 – Conclusions
5.3 Future Work
As discussed in the Limitations, in order to implement further refinement of the model, additional analysis on outliers in the data that create extreme delay is
recommended for future work. The uncertainty represented by these error reports requires that an additional set of constraints be assessed by the manager to supplement the model with additional nodes or additional states to one or more of the nodes. These incidents are primarily System of System (System Type 1) and Error Category 3 and likely introduce an additional set of integration challenges that are not easily apparent in the available variables that represent the data (evidence nodes). Further analysis is recommended to determine the best way to adjust the model to include the potential for additional nodes or additional states to select nodes. The model can also be expanded to (1) Consider the impact that unexpected architecture changes resulting from the SW update or modification have on the integration delays. (2) Assess the impact of additional discretization techniques on accuracy. (3) Enhance training for the model by increasing the number of samples.
104 References
Baldwin, W. C., & Sauser, B. (2009). Modeling the characteristics of system of systems. Paper presented at the System of Systems Engineering, 2009. SoSE 2009. IEEE International Conference On, 1-6.
Basili, V. R., Briand, L. C., & Melo, W. L. (1996). A validation of object-oriented design metrics as quality indicators. IEEE Transactions on Software Engineering, 22(10), 751-761.
Bielza, C. (2014). Discrete bayesian network classifiers: A survey. ACM Computing Surveys, 47(1), 1.
Boehm, B., & Valerdi, R. (2011). Impact of software resource estimation research on practice: A preliminary report on achievements, synergies, and challenges. 33rd International
Conference on Software Engineering (ICSE) , Waikiki, Honolulu, HI USA. 1057-1065.
Cantot, P., & Luzeaux, D. (2011). Simulation and modeling of systems of systems [electronic resource]. London : Hoboken, N.J.: ISTE ; Wiley.
Chittister, C. G., & Haimes, Y. Y. (1996). Systems integration via software risk management.
IEEE Transactions on Systems, Man & Cybernetics, Part A (Systems & Humans), 26(5), 521-32. doi:10.1109/3468.531900
Dahmann, J. (2015). The state of systems of systems engineering knowledge sources. Paper presented at the System of Systems Engineering Conference (SoSE), 2015 10th, 475-479.
doi:10.1109/SYSOSE.2015.7151979
105
Dahmann, J., Lane, J., Rebovich, G., & Lowry, R. (2010). Systems of systems test and evaluation challenges. Paper presented at the System of Systems Engineering (SoSE), 2010 5th
International Conference On, 1-6. doi:10.1109/SYSOSE.2010.5543979
Dahmann, J., Rebovich, G., Lane, J., Lowry, R., & Baldwin, K. (2012). An implemented view of systems engineering for systems of systems. Aerospace and Electronic Systems Magazine, IEEE, 27(5), 11-16. doi:10.1109/MAES.2012.6226689
Davendralingam, N., & Kenley, C. R. (2013). A mechanism design framework for the acquisition of independently managed systems of systems. Paper presented at the System of Systems Engineering (SoSE), 2013 8th International Conference On, 171-176.
doi:10.1109/SYSoSE.2013.6575262
Davis, J. J., & Foo, E. (2016). Automated feature engineering for HTTP tunnel detection.
Computers & Security, 59, 166-185.
Dillon, R. L., Paté-Cornell, M. E., & Guikema, S. D. (2005). Optimal use of budget reserves to minimize technical and management failure risks during complex project development.
Engineering Management, IEEE Transactions On, 52(3), 382-395.
doi:10.1109/TEM.2005.850733
DiMario, M., Cloutier, R., & Verma, D. (2008). Applying frameworks to manage SoS architecture.
. Engineering Management Journal, 12(4), 18-23.
DOD directive 5000.1 (2015). . Washington DC: Government Printing Office.
Domingos, P., & Pazzani, M. (1997). On the optimality of the simple bayesian classifier under zero-one loss. Machine Learning, 29(2-3), 103-130.
106
Felder, W. (2012). The elephant in the mist: What we don't know about the design, development, test and management of complex systems. Journal of Aerospace Operations, 1(4), 317-327.
Ferreira, S., Faezipour, M., & Corley, H. W. (2013). Defining and addressing the risk of
undesirable emergent properties. Paper presented at the Systems Conference (SysCon), 2013 IEEE International, 836-840. doi:10.1109/SysCon.2013.6549981
Flores, M. J., Gámez, J. A., Martínez, A. M., & Puerta, J. M. (2011). Handling numeric attributes when comparing bayesian network classifiers: Does the discretization method matter?
Applied Intelligence, 34(3), 372-385.
Frese, R., & Sauter, V. (2014). Improving your odds for software project success. Engineering Management Review, IEEE, 42(4), 125-131.
Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian network classifiers. Machine Learning, 29(2-3), 131-163.
Gandhi, S. J., Gorod, A., & Sauser, B. (2011). A systemic approach to managing risks of SoS.
Paper presented at the Systems Conference (SysCon), 2011 IEEE International, 412-416.
doi:10.1109/SYSCON.2011.5929045
Gupta, A., Mehrotra, K. G., & Mohan, C. (2010). A clustering-based discretization for supervised learning. Statistics & Probability Letters, 80(9–10), 816-824.
Holte, R. (1993). Very simple classification rules perform well on most commonly used datasets.
Machine Learning, 11(1), 63.
Houston, D. (2014). A generalized duration forecasting model of test-and-fix cycles. Journal of Software: Evolution and Process, 26(10), 877-889.
107
Jain, R. (2010). A framework for end-to-end approach to systems integration. International Journal of Industrial and Systems Engineering, 5(1), 79-109.
Jain, R., Chandrasekaran, A., Elias, G., & Cloutier, R. (2008). Exploring the impact of systems architecture and systems requirements on systems integration complexity. IEEE Systems Journal, 2(2), 209.
Kodama, M. (2011). Knowledge integration dynamics [electronic resource] : Developing strategic innovation capability. Singapore: World Scientific.
Kupervasser, O. (2014). The mysterious optimality of naive bayes: Estimation of the probability in the system of "classifiers". Pattern Recognition and Image Analysis, 24(1), 1-10.
Lamb, C. T., & Rhodes, D. H. (2009). Collaborative systems thinking: Uncovering the rules of team-level systems thinking. Paper presented at the Systems Conference, 2009 3rd Annual IEEE, 413-418. doi:10.1109/SYSTEMS.2009.4815837
Langford, G. O. (2012). Engineering systems integration : Theory, metrics, and methods (1st ed.).
Hoboken: Taylor and Francis.
Larose, D. T. (2014). Discovering knowledge in data : An introduction to data mining (2nd ed.).
Somerset: Wiley.
Little, T. (2006). Schedule estimation and uncertainty surrounding the cone of uncertainty. IEEE Software, 23(3), 48-54.
Liu, H., Hussain, F., Chew Lim Tan, & Dash, M. (2002). Discretization: An enabling technique.
Data Mining and Knowledge Discovery, 6(4), 393-423.
108
López-Martín, A., Chavoya, A., & Meda-Campaña, M. E. (2015). A fuzzy logic model for predicting the development schedule of software projects. Paper presented at the Information Technology - New Generations (ITNG), 2015 12th International Conference On, 415-420.
doi:10.1109/ITNG.2015.73
Loutchkina, I., Jain, L. C., Nguyen, T., & Nesterov, S. (2014). System's integration technical risk assessment model (SITRAM). IEEE Transactions on Systems, Man, and Cybernetics, 44(3), 342. doi:March 3, 2014
Lu, Y., Chang, L., Yang, K., Zhao, Q., & Chen, Y. (2010). Study on system of systems capability modeling framework based on complex relationship analyzing. Paper presented at the IEEE Systems Conference,
Luu, V. T., Kim, S., Tuan, N. V., & Ogunlana, S. O. (2009). Quantifying schedule risk in construction projects using bayesian belief networks. International Journal of Project Management, 27(1), 39-50.
Madni, A. M., & Sievers, M. (2014). Systems integration: Key perspectives, experiences, and challenges. Systems Engineering, 17(1), 37-51.
Mane, M., & DeLaurentis, D. A. (2009). Impact of programmatic system interdependencies on system-of-systems development. Paper presented at the System of Systems Engineering, 2009. SoSE 2009. IEEE International Conference On, 1-6.
Maurer, M., Schneller, R., & Omer, M. (2014). A survey on complexity management in systems engineering. Paper presented at the Systems Conference (SysCon), 2014 8th Annual IEEE, 180-187. doi:10.1109/SysCon.2014.6819255
109
Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270-280.
doi:10.1177/1356389012451663
Mende, T., Koschke, R., & Peleska, J. (2011). On the utility of a defect prediction model during HW/SW integration testing: A retrospective case study. Paper presented at the 2011 15th European Conference on Software Maintenance and Reengineering, CSMR 2011, March 1, 2011 - March 4, 259-268. doi:10.1109/CSMR.2011.32
Misirli, A. T., & Bener, A. B. (2014). Bayesian networks for evidence-based decision-making in software engineering. Software Engineering, IEEE Transactions On, 40(6), 533-554.
doi:10.1109/TSE.2014.2321179
Mizell, C., & Malone, L. (2007). A project management approach to using simulation for cost estimation on large, complex software development projects. Engineering Management Journal, 19(4) doi:December 2007
Mizianty, M., Kurgan, L., & Ogiela, M. (2008). Comparative analysis of the impact of
discretization on the classification with naïve bayes and semi-naïve bayes classifiers. Paper presented at the Machine Learning and Applications, 2008. ICMLA '08. Seventh
International Conference On, 823-828. doi:10.1109/ICMLA.2008.29
Mori, T., Tamura, S., & Kakui, S. (2013). Incremental estimation of project failure risk with naive bayes classifier. Paper presented at the Empirical Software Engineering and Measurement, 2013 ACM / IEEE International Symposium On, 283-286.
doi:10.1109/ESEM.2013.40
110
Patnayakuni, R., Arun Rai, & Tiwana, A. (2007). Systems development process improvement: A knowledge integration perspective. Engineering Management, IEEE Transactions On, 54(2), 286-300. doi:10.1109/TEM.2007.893997
Pearl, J. (1977). On summarizing data using probabilistic assertions. Information Theory, IEEE Transactions On, 23(4), 459-465. doi:10.1109/TIT.1977.1055756
Petersen, K., Roos, P., Nyström, S., & Runeson, P. (2014). Early identification of bottlenecks in very large scale system of systems software development. Journal of Software: Evolution and Process, 26(12), 1150-1171.
Rebovich, G. (2006). Systems thinking for the enterprise: New and emerging perspectives. Paper presented at the System of Systems Engineering, 2006 IEEE/SMC International Conference On, 6 pp. doi:10.1109/SYSOSE.2006.1652297
Sargis-Roussel, C., & Deltour, F. (2012). Beyond cross-functional teams: Knowledge integration during organizational projects and the role of social capital. Knowledge Management Research & Practice, 10(2), 128-140.
Sauser, B., Reilly, R. R., & Shenhar, A. J. (2015). Why projects fail? how contingency theory can provide new insights - comparative analysis of NASA's climate orbiter. IEEE Engineering Management Review, 43(4), 113-130.
Sauser, B. J., Reilly, R. R., & Shenhar, A. J. (2009). Why projects fail? how contingency theory can provide new insights – A comparative analysis of NASA’s mars climate orbiter loss.
International Journal of Project Management, 27(7), 665-679.
Sauser, B., & Boardman, J. (2008). Taking hold of system of systems management. Engineering Management Journal, 20(4), 3-8.
111
Senesi, C., Javernick, A., & Molenaar, K. R. (2015). Benefits and barriers to applying probabilistic risk analysis on engineering and construction projects. Engineering Management Journal, 27(2), 49.
Sousa-Poza, A., & Kovacic, S. (2008). Research agenda for complex situations. Engineering Management Journal, 20(4), 32-39.
Steindl, M., & Mottok, J. (2012a). Considering schedule requirements of software integration in component based embedded systems. Paper presented at the Applied Electronics (AE), 2012 International Conference On, 291-296.
Steindl, M., & Mottok, J. (2012b). Optimizing software integration by considering integration test complexity and test effort. Paper presented at the Intelligent Solutions in Embedded Systems (WISES), 2012 Proceedings of the Tenth Workshop On, 63-68.
Strumbelj, E., & Kononenko, I. (2014). Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems, 41(3), 647-665.
Tang, A., Nicholson, A., Jin, Y., & Han, J. (2007). Using bayesian belief networks for change impact analysis in architecture design. Journal of Systems and Software, 80(1), 127-148.
Thamhain, H. (2013a). Contemporary methods for evaluating complex project proposals. Journal of Industrial Engineering International, 9(1), 1-14.
Thamhain, H. (2013b). Managing risks in complex projects. Project Management Journal, 44(2), 20-35.
112
Trammell, T. I., Moulton, A., & Madnick, S. E. (2016). Effects of funding fluctuations on software development: A system dynamics analysis. 28(2), 71-85. . Engineering Management Journal, 28(2)
van Moll, J. H., & Ammerlaan, R. W. M. (2008). Identifying pitfalls of system integration -- an exploratory study. Paper presented at the Software Testing Verification and Validation Workshop, 2008. ICSTW '08. IEEE International Conference On, 331-338.
doi:10.1109/ICSTW.2008.22
Vaneman, W. K. (2016). The system of systems engineering and integration "vee" model. Paper presented at the 2016 Annual IEEE Systems Conference (SysCon), 1-7.
doi:10.1109/SYSCON.2016.7490599
Vasantrao, K. V. (2012). Enhance accuracy in software cost and schedule estimation by using
"uncertainty analysis and assessment" in the system modeling process. International Journal of Research & Innovation in Computer Engineering, 1(1), 6-8.
Weickert, J., Hein, M., & Schiele, B. (2013). Pattern recognition: 35th german conference, GCPR 2013, saarbrücken, germany, september 3-6, 2013. proceedings. Heidelberg:
Springer.
Witten, I. H., Frank, E., & Hall, M. A. (2014). Data mining: Practical machine learning tools and techniques (3rd ed.). St. Louis: Elsevier Science.
Wu, Q. (2006). Improvement of decision accuracy using discretization of continuous attributes.
Fuzzy Systems and Knowledge Discovery : Third International Conference, FSKD 2006, Xi'an, China, September 24-28, 2006 : Proceedings, , 674; 674-683; 683.
113
Zhang, H. (2008). Naive bayes for optimal ranking. Journal of Experimental & Theoretical Artificial Intelligence, 20(2), 79; 79-93; 93.
114
Appendix A – Literature Survey Data
Interdependen
! 1 1 1 IEEE Conference Baldwin (2009)
1 1 1 IEEE Conference Dahmann (2011)
! 1 1 IEEE Aerospace and
Electronic Systems Dahmann (2015)
1 1 1 IEEE Conference Davendralingam
(2013)
1 1 1 IEEE Systems
Journal DiMario
1 1 1 Journal of Aerospace
Operations Felder (2012)
1 1 IEEE Conference Ferreira (2013)
1 1 1 1 IEEE Conference Gandhi (2011)
1 1 IEEE Systems
Journal Gorod (2008)
1 1 1 1 IEEE Conference Hessami (2014)
1 1 1 IEEE Aerospace and
Electronic Systems Jamshidi (2008)
1 1 1 1 IEEE Systems Jain (2008)
1 1 IEEE Conference Jain (2010)
1 1
Global Journal of Flexible systems
Management
Jovel (2009)
1 1 1 Cybernetics Louthikina (2014)
1 1 1 1 IEE SysCCon Lu (2010)
1 1 1 1 1 Systems Engineering Madni (2014)
1 1 IEEE Malone (2013)
1 1 1 1 IEEE Mane (2008)
1 1 1 IEEE SysCon Mauri (2014)
1 1 1 1 1 Project Management
Journal McLain (2009)
1 1 IEEE SysCon Maurer (2014)
1 1 1 IEEE Systems Moti (2014)
1 1 1 Journal of SW
Evolustion & Process Petersen (2014)
1 1 1 IEEE Systems
1 1 IEEE Conference Steindl (2015)
1 1 1 IEEE Conference VanMoll (2008)
1 1
115
Appendix B – Feature Contingency Tables (Entire Database)
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
TRUE 0.31 0.28 0.08 0 0 0.68
FALSE 0.22 0.05 0.01 0.01 0.03 0.32
Total 0.53 0.33 0.09 0.01 0.03 1
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
CEC-1 0.36 0.06 0 0 0 0.42
CEC-2 0.05 0.07 0.02 0.01 0.03 0.18
CEC-3 0.13 0.2 0.08 0 0 0.41
Total 0.53 0.33 0.09 0.01 0.03 1
Table B3 - ACAT Feature Contingency Table:)
Is the system an Acquisition Category (ACAT) 1? (Yes or No
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
TRUE 0.246 0.153 0.051 0.008 0.034 0.492
FALSE 0.288 0.178 0.042 0 0 0.508
Total 0.534 0.331 0.093 0.008 0.034 1
Table B4-SysDepend Feature Data Con tingency Table:
The error impacts more than 1 system? (True or False)
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
TRUE 0.254 0.127 0.068 0.008 0.034 0.492
FALSE 0.28 0.203 0.025 0 0 0.508
Total 0.534 0.331 0.093 0.008 0.034 1
Table B5 - OrgDepend Feature Contingency Table :
The error has a dependency on more than 1 organization? (True or False).
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
TRUE 0.169 0.076 0.008 0.008 0.034 0.297
FALSE 0.364 0.254 0.085 0 0 0.703
Total 0.534 0.331 0.093 0.008 0.034 1
Contingency Table
Contingency Table
Table B1 - Severity Feature Contingency Table: The error is Severity 1 or 2. (True or False)
Table B2 - ErrorCat Feature Contingency Table : What is the type of error? (1,2 or 3) Contingency Table
Contingency Table
Contingency Table
116
Appendix B – Feature Contingency Tables (Entire Database Continued)
Table B6 -PriorError Feature Contingency Table:
The system has errors in the prior events? (True or False)
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
TRUE 0.254 0.186 0.059 0.008 0.008 0.517
FALSE 0.28 0.144 0.034 0 0.025 0.483
Total 0.534 0.331 0.093 0.008 0.034 1
Table B7 - SameEvent Feature Contingency Table:
The system with the error has other errors in the current event? (True or False).
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
TRUE 0.407 0.322 0.085 0.008 0.034 0.856
FALSE 0.127 0.008 0.008 0 0 0.144
Total 0.534 0.331 0.093 0.008 0.034 1
Table B8 - SysGroup Feature Contingency Table: The system is a Group 1? (True or False)
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
TRUE 0.364 0.263 0.059 0 0.017 0.703
FALSE 0.169 0.068 0.034 0.008 0.017 0.297
Total 0.534 0.331 0.093 0.008 0.034 1
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
TRUE 0.246 0.186 0.059 0.008 0.034 0.534
FALSE 0.288 0.144 0.034 0 0 0.466
Total 0.534 0.331 0.093 0.008 0.034 1
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
TRUE 0.209 0.052 0.022 0.007 0.015 0.306
FALSE 0.343 0.261 0.06 0.007 0.022 0.694
Total 0.552 0.313 0.082 0.015 0.037 1
Table B9 - SysType Feature ContingencyTable: The error is charged to a SoS? (True or False)
Table B10 - Core Feature Contingency Table: Is the system a core system? (True of False).
Contingency Table
Contingency Table
Contingency Table Contingency Table
117
Appendix C – Example Prior and Posterior Probability Calculations
IDI-1 IDI-2 IDI-3 IDI-4 IDI-5 Total
TRUE 0.31 0.28 0.08 0.00 0.00 0.68
FALSE 0.22 0.05 0.01 0.01 0.03 0.32
Total 0.53 0.33 0.09 0.01 0.03 1.00
Contingency Table
Prior Probability of Each IDI Class
e.g. P(c)=P(IDI=1)=.53 Prior Probability of Evidence or Predictor e.g. P(Severity=False)=.32
Likelihood of CondiDonal Probability of Predictor given Class e.g. P(Severity=False|IDI=1)=.22
Posterior Probability :
P(IDI=1|Severity=False) = .22x.53/.32 = .36 P(IDI=2|Severity=False) = .05x.33/.32 = .05 P(IDI=3|Severity=False) = .01x.09/.32 = .003 P(IDI=4|Severity=False) = .01x.01/.32 = .0003 P(IDI=5|Severity=False) = .03x.03/.32 = .0028
PredicDon = Maximum Posterior Probability