With the advances in research and practices in software testing methodologies and tool supports, the tasks of executing test cases and collecting test results have achieved a
high degree of automation with the widespread use of automatic testing platforms. How- ever, despite the progresses already made in many aspects of test automation by the nu-merous efforts of researchers, the task of generating test cases remains mostly a manual process. In this paper, the Test-Duo framework has been proposed for further automating the task of generating functional test cases that are ready for execution on an automatic testing platform. With Test-Duo, the tester focuses on the tasks of refining and annotat-ing use cases and preparannotat-ing test data sets with tool support, leavannotat-ing the tasks of generat-ing the actual test cases to the Test Director component of Test-Duo. Since a large num-ber of test cases can be generated, the Test Driver component of Test-Duo works closely with the Test Director to marshal execution of the test cases on the underlying automatic testing platform. Furthermore, the tester can systematically improve the quality of use cases and the performances of annotation with test data sets in finding potential SUT bugs. The improvement is accomplished by iteratively subjecting the generated test cases to mutation testing. Both the mutants that are killed and those that remain alive are ana-lyzed, and corresponding strategies are applied to increase mutation score and to reduce the time required to kill the mutants. The effectiveness of this improvement process has been demonstrated in the case study. The use of mutation testing as a means for improv-ing the quality of generated test cases by Test-Duo is an answer to Yue and Harman’s call for more works to be done in the direction of generating test cases to kill mutants and improving the generated test cases based on the associated mutation analysis [8].
A number of directions for future research are possible. Here we mention three of them. First, in the Test-Duo framework, the Test Director randomly picks data items from the data sets for use as arguments in the test cases. As has been shown, the data sets should contain items that enable a data combination for exposing a fault in the SUT while running test cases. While we have shown that data sets can be improved by incre-mental refinement, strategies for constructing and selecting effective data sets can help as well. Secondly, while mutation testing has been used as the measurement and improve-ment framework, other measureimprove-ment mechanisms such as various coverage criteria should be explored as well. Lastly, mutation testing can involve a large amount of com-putation, especially when a large number of mutants are used. The architecture of Test-Duo can be extended to apply parallel and distributed computing techniques by de-ploying multiple instances of the Test Driver over clusters of computers to run the tests in parallel.
REFERENCES
1. K. Beck, Test-Driven Development: By Example, 2003, Addison-Wesley, MA.
2. T. D. Hellmann, A. Sharma, J. Ferreira, and F. Maurer, Agile Testing: Past, Present, and Future Charting a Systematic Map of Testing in Agile Software Development, in Agile Conference, 2012, pp. 55-63.
3. R. Mugridge and W. Cunningham, Fit for Developing Software: Framework for Integrated Tests, Prentice Hall, USA, 2005
4. “Robot framework official web site,” http://code.google.com/p/robotframework/.
5. I. Bratko, Prolog Programming for Artificial Intelligence, Addison Wesley, 2001, MA.
6. K. N. King and A. J. Offutt, “A fortran language system for mutation-based software testing,” Software: Practice and Experience, Vol. 21, 1991, pp. 685-718.
7. J. Offutt, “A mutation carol: Past, present and future,” Information and Software Technology, Vol. 53, 2011, pp. 1098-1107.
8. J. Yue and M. Harman, “An analysis and survey of the development of mutation testing,” IEEE Transactions on Software Engineering, Vol. 37, 2011, pp. 649-678.
9. M. Daran and P. Thevenod-Fosse, “Software error analysis: a real case study involv- ing real faults and mutations,” SIGSOFT Software Engineering Notes, Vol. 21, 1996, pp. 158-171.
10. J. H. Andrews, L. C. Briand, and Y. Labiche, “Is mutation an appropriate tool for testing experiments?” in Proceedings of the 27th International Conference on Soft- ware Engineering, 2005, pp. 402-411.
11. Y. C. Cheng, P. H. Ou, C.-T. Chen, and T. S. Hsu, “A distributed system for continuous integration with JINI,” in Proceedings of the 14th International Con- ference on Distributed Multimedia Systems, 2008, pp. 130-135.
12. M. A. Jackson, Principles of Program Design, Academic Press, Inc., Orlando, FL, 1975.
13. C. Larman, Applying UML and Patterns: An Introduction to Object-Oriented An- alysis and Design and Iterative Development, 3rd ed., Prentice Hall PTR, 2004.
14. M. Weiser, “Program slicing,” IEEE Transactions on Software Engineering, Vol.
SE-10, 1984, pp. 352-357.
15. R. J. Abbott, “Program design by informal English descriptions,” Communications of ACM, Vol. 26, 1983, pp. 882-894.
16. A. J. Offutt and R. H. Untch, “Mutation 2000: Uniting the orthogonal,” Mutation Testing for the New Century, W. E. Wong, ed., 2001, Kluwer Academic Publishers, MA, pp. 34-44.
17. R. A. DeMillo, Test adequacy and program mutation,” in Proceedings of the 11th International Conference on Software Engineering, 1989, pp. 355-356.
18. Y.-S. Ma, J. Offutt, and Y. R. Kwon, “MuJava: an automated class mutation system:
Research articles,” Journal of Software Testing, Verification and Reliability, Vol. 15, 2005, pp. 97-133.
19. E. H. Spafford, “Extending mutation testing to find environmental bugs,” Software:
Practice and Experience, Vol. 20, 1990, pp. 181-189.
20. L. Shufang, B. Xiaoying, and C. Yinong, “Automatic mutation testing and simula- tion on OWL-S specified web services,” in Proceedings of the 41st Annual Simula- tion Symposium, 2008, pp. 149-156.
21. R. A. DeMilli and A. J. Offutt, “Constraint-based automatic test data generation,”
IEEE Transactions on Software Engineering, Vol. 17, 1991, pp. 900-910.
22. M. Nica, S. Nica, and F. Wotawa, “On the use of mutations and testing for debug- ging,” Software: Practice and Experience, Vol. 43, 2013, pp. 1121-1142.
23. R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Hints on test data selection: Help for the practicing programmer,” Computer, Vol. 11, 1978, pp. 34-41.
24. H. Agrawal, R. A. DeMillo, et al., Design of Mutant Operators for the C Program- ming Language, 1989, Purdue University, pp. 82.
25. M. E. Delamaro, J. C. Maldonado, and A. P. Mathur, “Integration testing using interface mutation,” in Proceedings of the 7th International Symposium on Software Reliability Engineering, 1996, pp. 112-121.
26. M. E. Delamaro, J. C. Maidonado, and A. P. Mathur, “Interface mutation: an app- roach for integration testing,” IEEE Transactions on Software Engineering, Vol. 27,
2001, pp. 228-247.
27. S. Ghosh and A. P. Mathur, “Interface Mutation to assess the adequacy of tests for components and systems,” in Proceedings of the 34th International Conference on Technology of Object-Oriented Languages and Systems, 2000, pp. 37-46.
28. P. R. Mateo, M. P. Usaola, and J. Offutt, “Mutation at system and functional levels,”
in Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation Workshops, 2010, pp. 110-119.
29. T. Budd and D. Angluin, “Two notions of correctness and their relation to testing,”
Acta Informatica, Vol. 18, 1982, pp. 31-45.
30. A. J. Offutt and J. Pan, “Automatically detecting equivalent mutants and infeasible paths,” Software Testing, Verification and Reliability, Vol. 7, 1997, pp. 165-192.
31. C.-Y. Hsieh, C.-H. Tsai, and Y. C. Cheng, “Test-Duo: A framework for generating and executing automated acceptance tests from use cases,” in Proceedings of the 8th International Workshop on Automation of Software Test, 2013, pp. 89-92.
32. C. Denger and M. M. Mora, “Test case derived from requirement specifications,” in Fraunhofer IESE Report, 2003.
33. J. J. Gutierrez, M. J. Escalona, M. Mejias, and J. Torres, “Analysis of proposals to generation of system test cases from system requisites,” in CAiSE Short Paper Proceedings, O. Belo, et al., ed., 2005, CEUR-WS.org.
34. M. J. Escalona, J. J. Gutierrez, et al., “An overview on test generation from func- tional requirements,” Elsevier Science, Vol. 84. 2011, pp. 1379-1393.
35. P. N. Boghdady, N. L. Badr, M. Hashem, and M. F. Tolba, “Test case generation and test data extraction techniques,” International Journal of Electrical and Computer Sciences, Vol. 11, 2011, pp. 8.
36. A. Kaur and V. Vig, “Systematic review of automatic test case generation by UML diagrams,” International Journal of Engineering Research and Technology, Vol. 1, 2012, pp. 1-17.
37. J. Heumann, “Generating test cases from use cases,” http://www.ibm.com/developer- works/rational/library/content/RationalEdge/jun01/GeneratingTestCasesFromUseCa sesJune01.pdf, 2001.
38. P. Fröhlich and J. Link, “Automated test case generation from dynamic models,” in Proceedings of the 14th European Conference on Object-Oriented Programming, E.
Bertino, ed., 2000, pp. 472-491.
39. A. L. Blum and M. L. Furst, “Fast planning through planning graph analysis,” Ar- tificial Intelligence, Vol. 90, 1997, pp. 281-300.
40. M. El-Attar and J. Miller, “Developing comprehensive acceptance tests from use cases and robustness diagrams,” Requirements Engineering, Vol. 15, 2010, pp. 285-306.
41. P. Hsia, D. Kung, and C. Sell, “Software requirements and acceptance testing,” Annals of Software Engineering, Vol. 3, 1997, pp. 291-317.
42. G. Carvalho, D. Falcão, et al., “Test case generation from natural language require- ments based on SCR specifications,” in Proceedings of the 28th Annual ACM Symposium on Applied Computing, 2013, pp. 1217-1222.
43. J. Mingyue and D. Zuohua, “Automation of test case generation from textual use cases,” in Proceedings of the 4th International Conference on Interaction Sciences, 2011, pp. 102-107.
44. S. S. Somé and X. Cheng, “An approach for supporting system-level test scenarios
generation from textual use cases,” in Proceedings of ACM Symposium on Applied Computing, 2008, pp. 724-729.
45. J. J. Gutierrez, M. J. Escalona, et al., “A case study for generating test cases from use cases,” in Proceedings of the 2nd International Conference on Research Chal- lenges in Information Science, 2008, pp. 209-214.
46. J. Gutiérrez, G. Aragón, et al., “Automatic test case generation from functional requirements in NDT,” in Current Trends in Web Engineering, M. Grossniklaus and M. Wimmer, ed., 2012, pp. 176-185.
47. L. Briand and Y. Labiche, “A UML-based approach to system testing,” Software and Systems Modeling, Vol. 2, 2002, pp. 10-42.
48. C. Nebut, F. Fleurey, Y. L. Traon, and J. M. Jezequel, “Automatic test generation: a use case driven approach,” IEEE Transactions on Software Engineering, Vol. 32, 2006, pp. 140-155.
49. J. Warmer and A. Kleppe, The Object Constraint Language: Precise Modeling with UML, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999.
Chin-Yun Hsieh (謝金雲) is an Associate Professor at the Department of Computer Science and Information Engineering of the National Taipei University of Technology, Taiwan. He re-ceived his MS and Ph.D. degrees from the University of Missis-sippi and the University of Oklahoma, respectively, both in Com-puter Science. His research interests include object-oriented soft-ware engineering, pattern languages and softsoft-ware testing. He is a member of the Software Engineering Association of Taiwan (SEAT).
Chen-Shin Tsai (蔡振欣) is a Ph.D. candidate at the Depart- ment of Computer Science and Information Engineering, the Na-tional Taipei University of Technology, Taiwan. His research in-terests include software testing and object-oriented analysis and design.
Yu Chin Cheng (鄭有進) is a Professor at the Department of Computer Science and Information Engineering of the National Taipei University of Technology, Taiwan. Dr. Cheng teaches and conducts research in object-oriented systems, agile methods, soft-ware requirements and specifications, and softsoft-ware architecture.
He is a member of the IEEE Computer Society, the ACM, and the Software Engineering Association of Taiwan (SEAT).