In Chapter 5, we introduced and described a process for software reuse. This process outlines the major steps involved in software reuse in general. At this point, we want to review the steps from the decision to search and the description of the subject of the search up to the selection of a reusable asset in the context of the reuse of software tests. Therefore, the fundamental process of acquiring a test recommendation is outlined in Figure 7.6.
Basically, the process begins once the developer has written a class and starts to write tests (1) that, as well as implicitly describing how to test the class, also describe its intended behavior (i.e., the developer provides a syntactic and a semantic description of the system under development). Subsequently the de- veloper performs a reverse search using the test search engine, whereas “reverse” means that it does not look for components providing a specific interface, but returns those tests that require a similar interface to the one declared in the query, i.e., provided by the class under test (2).
The system judges their fitness for purpose (3) and those tests that pass this step represent possible test recommendations which are subsequently ranked (4) and delivered to the user. If necessary, the involved interfaces can be dy- namically adapted using the technology we introduced in earlier (cf. Section 4, pp. 55). Finally, the user inspects the recommended set of tests and decides which test(s) should be reused in the current project (5). By reusing a test and abandoning other recommendations, developers provide valuable feedback to the system, which can be automatically analyzed and used to improve the backend’s evaluation algorithms and influence future result ranking.
7.5. Implementation
Throughout this thesis we have referred to SENTRE which embodies the results of our theoretical considerations and practical experiences from research on the reuse of previously created software tests. Although we have given insights on its implementation at various points, it is necessary to briefly describe the architectural design of SENTRE and mention some facts about the current implementation. Naturally, the development of such a sophisticated tool in a completely new area of research is always a challenge and demands a lot of endurance and patience during its emergence. From the initial investigation of technologies and APIs, appropriate database implementations and finally tool- assisted test reuse in the IDE a lot of difficulties had to be overcome. Not only the information extraction from JUnit tests sometimes ended with disappointing results, but also the server-side implementation contained many pitfalls like the problems arising during the search for reusable assets within nested arrays in a MongoDB collection. Having said that, we do not want to go into all details and pitfalls of the implementation, but rather present “the big picture”.
Some technical information about SENTRE should make it easier for future researchers, who want to follow-up on our work, to build similar and enhanced systems. For the current parser, we have chosen to utilize the Java Compiler API4 which provides a set of tools to access the abstract syntax tree (AST) of Java files. With itsTreeVisitorit provides an easy way to inspect a given Java
file, based on the concept of the visitor pattern [Gam+94]. The information extracted by the test parser is stored in a MongoDB collection, which provides a convenient way to store documents with dynamic structures and offers fast index-based searches. Furthermore it supports searches with regular expressions, which made the translation of search queries much easier.
The SENTRE search engine is implemented using the Java Enterprise Edition and deployed on a JBoss 7.1 application server running on a Linux operating system. The underlying hardware (as of April 2014) comprises an Intel Core i7-2600 CPU
4http://docs.oracle.com/javase/7/docs/jdk/api/javac/tree/com/sun/source/util/package-
Database / MongoDB Application Server / JBoss AS
Web Server / Apache
Browser Eclipse Plug-In
Figure 7.7.: System Architecture Sketch of SENTRE.
with 3.40 GHz, 16GB of RAM available and approx. 20TB harddisk space for the system and the code repository. To communicate with external client software, we have implemented a web service using the capabilities of the JBoss AS. All access to the application server is performed via an Apache server using the AJP module. SENTRE is therefore based on a multi-tier architecture, where the services are separated and encapsulated.
The architecture of the reuse-assisted test recommendation system is visualized in Figure 8.3, which shows the different tiers of the system. The client tier, containing the Eclipse plug-in, will be the subject of the following chapter. Although SENTRE offers a browser UI, we have already discussed in Chapter 5 that an appropriate IDE integration is essential for effective utilization and adoption of software search engines by developers.
7.6. Summary
At the beginning of this section we discussed and identified typical scenarios for using dedicated test search engines.. We identified potential use cases for these systems and discussed them in the context of the traditional phases of the software development lifecycle. Subsequently, we introduced a set of complementary retrieval techniques that can be used to search for previously written test cases, based on the well-known idea of using interface descriptions of the class under test. Furthermore, we introduced the idea of value-based
searches that rely on a user’s specification that describes the mapping of test case values to their corresponding expected results. With the introduction of regular expressions as an extension to value-based searches, more general queries can be specified and the values can be described as patterns.
While the preceding retrieval techniques rely on static content analysis, the description of code-based searches introduced dynamic evaluation of reuse candidates to test search engines. Based on the application of potentially reusable test cases to the user’s class under test, the system can identify and rank valuable tests. After describing the retrieval of exception test, the chapter concluded with an overview of the architecture and current implementation of SENTRE, our search engine for reusable software tests.
In the following chapter we are going to describe the implementation of an Eclipse plug-in for test reuse that automates the search for reusable software tests, integrates it into the developer’s IDE and makes the whole process more user friendly.
Contribution of this chapter
• We have discussed and identified a set of archetypal usage scenarios for dedicated search engines for software tests.
• We have presented interface-based searches for test reuse and an approach to automatic query refinement and ranking.
• We have discussed searches for reusable software tests based on mappings of test case values to expected results.
• In addition to value-based searches, we have enhanced this retrieval tech- nique with a pattern-based value specification using regular expressions. This strategy potentially improves the recall compared to plain value-based searches, where the test case values and expected result must be a perfect match.
• We have introduced the idea of code-based searches for software tests that can be used for dynamic result evaluation of reusable software tests.
• Based on our considerations of a software reuse process, we have intro- duced a micro-process for the reuse of software tests.
• We have described the system architecture and implementation of SENTRE, our search engine for reusable test data.
“
I have them, you have them, we all have them – and the point is to do what we can to prevent them and to discover them as early as possible.”
Software Testing Techniques [Bei90] Boris Beizer, Software Engineer
8
Reuse-Assisted Test
Recommendation
Software search engines are only as good as the underlying technologies used to drive their repositories, the parsers used to build their indexes and the retrieval algorithms used to search for results. However, in Chapters 3 and 5 we learned that although good solutions to these challenges are necessary for a successful search engine, they are not sufficient. A successful search engine must also provide a user friendly interface and environment that automates the process of reuse to the greatest extent possible. Over the past decade a lot of research has been conducted in the area of software search, but only a minor proportion has focused on the requirements for reuse-oriented recommendation systems in software engineering.
Based on our work conducted on Code Conjurer [Jan07; HJA08] and on the characterization of similar tools [JHA14], in this chapter we introduce a reuse-
oriented test recommendation system for Eclipse. The plug-in acts as a client to SENTRE and utilizes its capabilities to provide on-demand test recommen- dations [JA13; Erl13]. First, we discuss the requirements for a reuse-oriented test recommendation system and outline the general process of tool-supported test reuse. Subsequently, we describe the design and implementation of the tool, before we present some usage examples of our prototype implementation accompanied with a couple of screenshots.
8.1. Characteristics
Earlier in this thesis we discussed the general characteristics of so-called reuse- oriented code recommendation systems and presented our findings also in a chapter of the book on Recommendation Systems in Software Engineering [JHA14]. This chapter will therefore focus on adapting such systems for test reuse. Our goal is to focus on their implementation and how the idea of a reuse-oriented test recommendation system can actually be realized.
As already mentioned, the implementation of a recommendation system for reusable software tests needs to be seamlessly integrated into the development environment of its users. In order to minimize the barriers to its use, the sys- tem needs to “feel” familiar to developers and needs to be non-intrusive. If a recommendation system for software tests disturbs the workflow of the user by continuously demanding attention users may quickly become annoyed and de- activate or remove the system. Hence, it is necessary that the recommendations offered by the system are well integrated into the IDE and are ready on demand whenever the user desires them.
Therefore, like the reuse-oriented code recommendation system Code Con- jurer [Jan07; HJA08], a recommendation system for software tests needs to implement an autonomous background agent which continuously monitors the developer’s actions. More specifically, it needs to inspect the test cases associated with the class under development and unobtrusively spring into action, when
the user edits them. Autonomous background agents are therefore a central part of our test-reuse environment that we will introduce below (cf. Section 8.3). Based on our findings in Chapter 5, we emphasize that a recommendation system for reusable software tests needs to fulfill the following requirements:
Proactive service The test-reuse environment needs to constantly monitors the users’ testing activities and becomes active when a test case is opened in the editor. The system autonomously decides when to trigger a search for potentially reusable assets, i.e., when to start the process depicted in Figure 8.1. The user should not be aware of the system’s activity and should not be diverted from their normal workflow in any way.
Context awareness In order to work appropriately, a reuse-oriented test rec- ommendation system needs to be aware of the developer’s context. For example, since the JUnit assertions in a developer’s tests usually contain just method invocations or literals as expected values, the type of the ex- pected result is not obvious. Nevertheless, this information can be obtained from the class under test via its interface declaration.
IDE integration The test-reuse environment should provide seamless IDE in- tegration. In other words, the users should not need to learn any new concepts to learn it and should become familiar with the system with as little effort as possible. For example, a single assert statement can be recommended using the auto-complete feature of the IDE.
Candidate evaluation Recommendation systems are only useful, when they provide valuable results to their users. This applies even more to reuse- oriented systems, since they need to tip the make or reuse dilemma in favor of reuse. Hence, the test-reuse environment needs to evaluate the potentially reusable tests and create a ranked list with the potentially most- useful recommendations on top. As a failing test potentially represents the discovery of a bug in the CUT, these tests should be ranked first, but nevertheless also inspected carefully.
Ready-on-demand The evaluation of tests can potentially be time consuming. In order to be useful for developers, a test-reuse environment must be re-
sponsive and provide results as quickly as possible. Therefore we utilize the idea of speculative analysis [Bru+10] and perform background evaluation of potential results even before the user requests them.
Feedback evaluation The system needs to examine whether the user was satisfied with its recommendations. Since recommendations are usually ranked using a weight function, the system can adjust the corresponding weights for future result evaluation and ranking.