Final Thoughts

This thesis explored the idea of automatically generating program invariants and applying them to research areas across the field of software engineering. PRECIS is an invariant generation technique that uses predicate data in combination with a linear regression combination to derive function-level invariants. The three-step process of data generation, predicate clustering, and invariant generation generates high-specificity, high-coverage, and high- quality invariants.

The invariants generated by PRECIS can be used to detect hardware faults. The invariants can be integrated into program text for software-level checking, or used on specialized hardware for low overhead. Fault injection experiments show that these invariants have high-coverage for a several types of manifested errors.

The same predicate information gathered by PRECIS can also be used for bug localization. PREAMBL localizes bugs to variable-length interpro- cedural paths through a statistical path-scoring algorithm. This process is optimized through the use of a path-tree data structure. These localized paths can then be associated with invariants to improve the software debug- ging process.

software development process. Past approaches to static invariant genera- tion have shortcomings due to the path-explosion problem, while dynamic approaches have remained too general and unspecific. This thesis explores an answer to these problems by using path information (in the form of pred- icates). Program path predicates provide a unique “fingerprint” for each ex- ecution, which “filters” the specific behavior for that particular path. Since each individual path tends to have relatively simple behavior, this allows for the piecemeal analysis of each path individually. These analyses can then be recombined to form a model for the entire program.

This thesis has shown the promise of predicate-based automatically gener- ated invariants for improvements in regression testing, bug localization, and fault detection. Continuing with a similar approach holds promise for even greater improvements in those areas, as well as new applications in other challenging areas of software development.

APPENDIX A

GETTING PRECIS AND PREAMBL

PRECIS and PREAMBL are available on an University of Illinois at Urbana- Champaign wiki page. Documentation on how to use the tools is included.

For more details, go to:

REFERENCES

[1] J. Misra, “Prospects and limitations of automatic assertion generation for loop programs,” SIAM Journal on Computing, vol. 6, no. 4, pp. 718–729, 1977.

[2] N. Bjorner, A. Browne, and Z. Manna, “Automatic generation of in- variants and intermediate assertions,” in Theoretical Computer Science. Springer-Verlag, 1997, pp. 589–623.

[3] S. Bensalem and H. Saidi, “Powerful techniques for the automatic gen- eration of invariants,” in CAV. Springer-Verlag, 1996, pp. 323–335. [4] A. Almossawi, K. Lim, and T. Sinha, “Analysis tool evaluation: Coverity

prevent,” Ph.D. dissertation, Carnegie Mellon University, 2006.

[5] Hewlett-Packard, “HP Fortify static code analyzer (SCA),” 2012. [Online]. Available: https://www.fortify.com/products/hpfssc/ source-code-analyzer.html

[6] Z. Li and Y. Zhou, “PR-Miner: Automatically extracting implicit pro- gramming rules and detecting violations in large software code,” in 13th ACM SIGSOFT Symposium on the Foundations of Software Engineer- ing (FSE’05), Sept 2005.

[7] Z. Li, S. Lu, S. Myagmar, and Y. Zhou, “Cp-miner: A tool for finding copy-paste and related bugs in operating system code,” in Proceedings of the Sixth Symposium on Operating System Design and Implementation (OSDI’04), 2004, pp. 289–302.

[8] G. Ammons, R. Bod´ık, and J. R. Larus, “Mining specifications,” in Proc. of POPL ’02, ser. POPL ’02. New York, NY, USA: ACM, 2002. [Online]. Available: http://doi.acm.org/10.1145/503272.503275 pp. 4–16.

[9] K. Jayaraman, D. Harvison, V. Ganesh, and A. Kiezun, “jFuzz: A con- colic whitebox fuzzer for java,” in Proceedings of the First NASA Formal Methods Symposium, 2009, pp. 121–125.

[10] P. Godefroid, N. Klarlund, and K. Sen, “Dart: directed automated ran- dom testing,” in ACM Sigplan Notices, vol. 40, no. 6. ACM, 2005, pp. 213–223.

[11] K. Sen, D. Marinov, and G. Agha, CUTE: A concolic unit testing engine for C. ACM, 2005, vol. 30, no. 5.

[12] C. Cadar, D. Dunbar, and D. Engler, “KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs,” in Proceedings of the 8th USENIX conference on operating systems design and implementation. USENIX Association, 2008, pp. 209–224.

[13] C. Cadar, V. Ganesh, P. Pawlowski, D. Dill, and D. Engler, “EXE: Automatically generating inputs of death,” ACM Transactions on In- formation and System Security (TISSEC), vol. 12, no. 2, p. 10, 2008. [14] M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin, “Dynamically

discovering likely program invariants to support program evolution,” in Proc. of ICSE ’99, ser. ICSE ’99. New York, NY, USA: ACM, 1999. [Online]. Available: http://doi.acm.org/10.1145/302405.302467 pp. 213–224.

[15] S. Hangal and M. S. Lam, “Tracking down software bugs using automatic anomaly detection,” in Proc. of ICSE ’02, ser. ICSE ’02. New York, NY, USA: ACM, 2002. [Online]. Available: http://doi.acm.org/10.1145/581339.581377 pp. 291–301.

[16] J. Henkel and A. Diwan, “A tool for writing and debugging algebraic specifications,” in Proc. of ICSE ’04, ser. ICSE ’04. Washington, DC, USA: IEEE Computer Society, 2004. [Online]. Available: http://portal.acm.org/citation.cfm?id=998675.999449 pp. 449–458. [17] C. Csallner, N. Tillmann, and Y. Smaragdakis, “DySy: Dynamic

symbolic execution for invariant inference,” in Proc. of ICSE ’08, ser. ICSE ’08. New York, NY, USA: ACM, 2008. [Online]. Available: http://doi.acm.org/10.1145/1368088.1368127 pp. 281–290.

[18] Y. Kannan and K. Sen, “Universal symbolic execution and its application to likely data structure invariant generation,” in Proc. of ISSTA ’08, ser. ISSTA ’08. New York, NY, USA: ACM, 2008. [Online]. Available: http://doi.acm.org/10.1145/1390630.1390665 pp. 283–294. [19] B. Liblit, M. Naik, A. Zheng, A. Aiken, and M. Jordan, “Scalable statis-

tical bug isolation,” ACM SIGPLAN Notices, vol. 40, no. 6, pp. 15–26, 2005.

[20] P. Arumuga Nainar, T. Chen, J. Rosin, and B. Liblit, “Statistical de- bugging using compound Boolean predicates,” in International Sympo- sium on Software Testing and Analysis, S. Elbaum, Ed., London, United Kingdom, July 9–12 2007.

[21] T. Chilimbi, B. Liblit, K. Mehra, A. Nori, and K. Vaswani, “Holmes: Effective statistical debugging via efficient path profiling,” in 31st In- ternational Conference on Software Engineering (ICSE 2009), J. Atlee and P. Inverardi, Eds. Vancouver, Canada: ACM SIGSOFT and IEEE, May 2009.

[22] Wikipedia, “Windows error reporting — wikipedia, the free encyclopedia,” 2012, [Online; accessed 12-April-2012]. [Online]. Available: http://en.wikipedia.org/w/index.php?title=Windows Error Reporting

[23] J. Han, M. Kamber, and J. Pei, Data Mining: Concepts and Techniques, 2nd ed. Morgan Kaufmann, Jan. 2006.

[24] C. Hoare, “An axiomatic basis for computer programming,” Communi- cations of the ACM, vol. 12, no. 10, pp. 576–580, 1969.

[25] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, “Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria,” in Proc. of ICSE ’94, ser. ICSE ’94. Los Alamitos, CA, USA: IEEE Computer Society Press, 1994. [Online]. Available: http://portal.acm.org/citation.cfm?id=257734.257766 pp. 191–200. [26] A. Avizienis, J. Laprie, B. Randell, and C. Landwehr, “Basic concepts

and taxonomy of dependable and secure computing,” Dependable and Secure Computing, IEEE Transactions on, vol. 1, no. 1, pp. 11–33, 2004. [27] N. Oh, P. Shirvani, and E. McCluskey, “Error detection by duplicated instructions in super-scalar processors,” Reliability, IEEE Transactions on, vol. 51, no. 1, pp. 63 –75, Mar. 2002.

[28] M. Rela, H. Madeira, and J. Silva, “Experimental evaluation of the fail- silent behaviour in programs with consistency checks,” in Proceedings of Annual Symposium on Fault Tolerant Computing, Jun. 1996, pp. 394– 403.

[29] K. Pattabiraman, G. Saggese, D. Chen, Z. Kalbarczyk, and R. Iyer, “Automated derivation of application-specific error detectors using dy- namic analysis,” Dependable and Secure Computing, IEEE Transactions on, vol. 8, no. 5, pp. 640–655, Sept.-Oct. 2011.

[30] P. Sagdeo, V. Athavale, S. Kowshik, and S. Vasudevan, “Precis: Infer- ring invariants using program path guided clustering,” Nov. 2011, pp. 532–535.

[31] N. Nakka, Z. Kalbarczyk, R. K. Iyer, and J. Xu, “An architectural framework for providing reliability and security support,” in Proceedings of the 2004 International Conference on Dependable Systems and Networks, ser. DSN ’04. Washington, DC, USA: IEEE Computer Society, 2004. [Online]. Available: http://dl.acm.org/citation.cfm?id= 1009382.1009774 pp. 585–594.

[32] P. J. Guo, “A scalable mixed-level approach to dynamic analysis of C and C++ programs,” M.S. thesis, MIT Department of Electrical Engi- neering and Computer Science, Cambridge, MA, May 5, 2006.

[33] N. Nethercote and J. Seward, “Valgrind: A framework for heavyweight dynamic binary instrumentation,” ACM Sigplan Notices, vol. 42, no. 6, pp. 89–100, 2007.

[34] G. Rothermel, R. Untch, C. Chu, and M. Harrold, “Prioritizing test cases for regression testing,” Software Engineering, IEEE Transactions on, vol. 27, no. 10, pp. 929–948, 2001.

[35] H. Leung and L. White, “Insights into regression testing [software test- ing],” in Proceedings of the Conference on Software Maintenance. IEEE, 1989, pp. 60–69.

[36] B. Beizer, Software Testing Techniques. Dreamtech Press, 2002. [37] H. Do, S. G. Elbaum, and G. Rothermel, “Supporting controlled ex-

perimentation with testing techniques: An infrastructure and its poten- tial impact.” Empirical Software Engineering: An International Journal, vol. 10, no. 4, pp. 405–435, 2005.

[38] F. Vokolos and P. Frankl, “Empirical evaluation of the textual differ- encing regression testing technique,” in Proceedings of the International Conference on Software Maintenance. IEEE, 1998, pp. 44–53.

[39] B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan, “Scalable statistical bug isolation,” in Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, ser. PLDI ’05. New York, NY, USA: ACM, 2005. [Online]. Available: http://doi.acm.org/10.1145/1065010.1065014 pp. 15–26.

[40] A. X. Zheng, M. I. Jordan, B. Liblit, M. Naik, and A. Aiken, “Statistical debugging: simultaneous identification of multiple bugs,” in Proceedings of the 23rd international conference on Machine learning, ser. ICML ’06. New York, NY, USA: ACM, 2006. [Online]. Available: http://doi.acm.org/10.1145/1143844.1143983 pp. 1105–1112.

[41] C. Liu, X. Yan, L. Fei, J. Han, and S. P. Midkiff, “Sober: Statistical model-based bug localization,” in Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, ser. ESEC/FSE-13. New York, NY, USA: ACM, 2005. [Online]. Available: http://doi.acm.org/10.1145/1081706.1081753 pp. 286–295.

[42] S. Yoo and M. Harman, “Tr-09-09: Regression testing minimisation, selection and prioritisation – A survey,” 2009.

[43] H. K. N. Leung and L. White, “A cost model to compare regression test strategies,” in Proceedings of the Conference on Software Maintenance, 1991, pp. 201–208.

[44] T. Ball, “On the limit of control flow analysis for regression test se- lection,” in Proceedings of the 1998 ACM SIGSOFT ISSTA, 1998, pp. 134–142.

[45] D. Binkley, “Semantics guided regression test cost reduction,” IEEE Transactions on Software Engineering, vol. 23, pp. 498–516, 1997. [46] A. Orso, N. Shi, and M. J. Harrold, “Scaling regression testing to large

software systems,” in Proceedings of the 12th ACM SIGSOFT Sympo- sium on the Foundations of Software Engineering (FSE 2004), 2004, pp. 241–252.

[47] T. Xie and D. Notkin, “Checking inside the black box: Regression test- ing based on value spectra differences,” in Proceedings of the 20th IEEE International Conference on Software Maintenance, 2004, pp. 28–37. [48] A. Mesbah, A. van Deursen, and D. Roest, “Invariant-based automatic

testing of modern web applications,” IEEE Transactions on Software Engineering, vol. 38, pp. 35–53, 2012.

[49] S. Mirshokraie and A. Mesbah, “JSART: JavaScript assertion-based re- gression testing,” in ICWE, vol. 7387, 2012, pp. 238–252.

[50] B. Morse, “A C/C++ front end for the daikon dynamic invariant detec- tion system,” Ph.D. dissertation, Massachusetts Institute of Technology, 2002.