Software testing is generally regarded as a labor intensive, challenging and ex- pensive task that is nevertheless essential to the development of quality software. A prominent example of the huge impact of a single line of faulty code is the so-called goto fail bug, where a single line of code (i.e., a simple ‘goto fail;’
statement) corrupted the SSL stack of Apple’s OS X and iOS3.
For the remainder of this thesis, we have chosen to focus on tests written using the JUnit testing framework. Since it is a widely used tool for software testing, there is a broad range of literature available [BG14] and most readers should be familiar with JUnit’s testing model. The methodology and ideas developed in this thesis should, however, be easily applicable to other testing frameworks and languages as well. Based on the previous definition of software testing terms we are going to develop a generic model of software tests in Section 6.2.1 of this thesis.
The Class Under Test
Unfortunately, there is still no unambiguous standard methodology for software testing, and therefore it is no surprise that testers may write totally different tests for testing the same conditions for a given piece of software. As we will see, it is possible to describe functionally equivalent tests in syntactically very different ways. This is not surprising, since it is well known that two implementations of the same program will rarely look exactly alike. In this section, we consider a very simple scenario, where a tester wants to investigate whether a distance calculator performs the computation of the distance between two given points correctly.
Therefore we start with the little program in Listing 2.1, which serves as the class under test for the subsequently presented test cases.
Listing 2.1: Distance Calculator – Class Under Test. 1 p u b l i c cla ss E u c l i d { 2 p u b l i c d o u b l e dist (d o u b l e x1 ,d o u b l e y1 , 3 d o u b l e x2 ,d o u b l e y2 ) { 4 r e t u r n Math . sqrt (( x2 - x1 ) *( x2 - x1 ) +( y2 - y1 ) *( y2 - y1 ) ) ; 5 } 6 }
For our purposes, we define the example class Euclid with the methoddist,
which calculates the distance of two pointsp(x1|y1)andq(x2|y2)using the well- known formula for the Euclidean distance:
d(p, q) =p(x2− x1)2+ (y2− y1)2 (2.2.1)
A tester can arbitrarily choose any points, such asp(4|2),q(8|5)with a distance
of d = 5, to test the program for bugs. According to our previous definitions, the
general description of such a test
dist: (4, 2, 8, 5) → 5
maps the given test case values (4, 2, 8, 5)to the corresponding expected
result of 5. It therefore states that the test is performed executing the dist
operation of the system under test.
JUnit TestCase Example
Even though tests written for the code example from Listing 2.1 are intended to examine bugs in only one single calculation – which is performed in the return statement in line 3 – there are virtually unlimited ways of writing a test case for this class. The typical approach would be to write a test case that defines the correct outcome of the invocation of the distance calculator’sdistmethod
depending on an ordered quadruple of input parameters, like the one presented in Listing 2.2.
Listing 2.2: Test for the Distance Calculator withassertEquals. 1 p u b l i c cla ss E u c l i d T e s t { 2 @T est 3 p u b l i c void t e s t D i s t a n c e C a l c u l a t i o n () { 4 E u c l i d calc = new E u c l i d () ; 5 a s s e r t E q u a l s (5 , calc . dist (4 , 2 , 8 , 5) ) ; // P , Q 6 a s s e r t E q u a l s (5 , calc . dist (0 , 0 , 3 , 4) ) ; // O r i g i n 7 a s s e r t E q u a l s (12.5 , calc . dist ( -3 , -4 , 4.5 , 6) ) ; // Fl 8 } 9 }
This test adheres to the conventions of the IPO model: it takes, for instance, the input(0, 0, 3, 4)as test case values for the software under test and evaluates
the test result against the expected result for equality. In this case, the developer also provides an expected value in the way defined in the JUnit documentation, i.e., as the first parameter of the assert statement. Hence, from this test case it is possible to extract the mapping of test case values to the corresponding expected results(0, 0, 3, 4)→5directly from the code.
Test Syntax vs. Test Semantics
Since JUnit classes are well-known POJOs4 there is, however, no automated check for the correct order of the parameters of the assertEquals statement. This fact imposes a first obstacle to the automated parsing of JUnit tests, since a tester may write line 5 in Listing 2.2 conversely as
assertEquals(calc.dist(4, 2, 8, 5), 5);
which would – strictly speaking – be represented by the following mapping of test case values to expected result: 5→(4, 2, 8, 5). This would not be correct
in the context of the class under test, although it represents a valid statement. But before addressing this issue, we want to investigate another way of writing a test for the same distance calculator class as presented in Listing 2.3.
Listing 2.3: Test for the Distance Calculator withassertTrue. 1 p u b l i c cla ss E u c l i d T e s t {
2 @T est
3 p u b l i c void t e s t D i s t a n c e C a l c u l a t i o n () { 4 E u c l i d calc = new E u c l i d () ;
5 int p q D i s t = calc . dist (4 , 2 , 8 , 5) ; 6 int o r i g i n D i s t = calc . dist (0 , 0 , 3 , 4) ;
7 d o u b l e f l o a t i n g D i s t = calc . dist ( -3 , -4 , 4.5 , 6) ; 8 a s s e r t T r u e ( p q D i s t == 5) ; 9 a s s e r t T r u e ( o r i g i n D i s t == 5) ; 10 a s s e r t T r u e ( f l o a t i n g D i s t == 12. 5) ; 11 } 12 }
This test case is more challenging to an automated knowledge extractor, since there are several issues that make it difficult to create the mappings of input val- ues to expected outputs. Although the use of assertTrueinstead of the equality
assertion can be resolved relatively easily, the definition of variables containing the test result and their usage in the assertion is somewhat challenging. Since the test is now split into two parts, where a) the CUT is invoked using test case values, b) the test result is assigned to a variable and c) this variable containing the test result is compared to the expected result the extraction of the tests from this test case demands more sophisticated algorithms.
Testing without using JUnit
For the sake of clarity this test case also makes it necessary to revisit the previous definition of the term “test”. Although this test case might appear to deviate from the definition in Section 2.4 this is, however, not the case. A test was defined as the execution of the software under test using test case values (e.g., line 5 in Listing 2.3) and comparing the test result to an expected result (e.g., line 8 of the same listing).
The last kind of test case we want to take a brief look at is not a JUnit test case in the narrow sense, but a test case expressed using plain Java code and a boolean value that indicates whether the returned result matches the expected result or
not. Nevertheless, the code presented in Listing 2.4 meets our definition of a test case and a test since it aims to reveal an unexpected result by executing the system under test using test case values.
Listing 2.4: Test for the Distance Calculator without JUnit. 1 p u b l i c cla ss D i s t C a l c V e r i f i e r {
2
3 // this is true if the test is p a s s e d
4 p u b l i c b o o l e a n v e r i f y D i s t a n c e C a l c u l a t i o n () { 5 E u c l i d calc = new E u c l i d () ; 6 if ( calc . dist ( -5 ,5) != 0) { 7 r e t u r n fal se; 8 } 9 r e t u r n true; 10 } 11 12 }
The code in Listing 2.4 provides evidence that even before the advent of JUnit or other similar testing frameworks it was already possible to test Java classes and look for defects using so-called plain old Java objects (POJO). A human can recognize the Calculator as the system under test, the tuple(−5, 5)as test case
values and0as the expected result.
However, for a parser such a piece of code does not contain enough structural information that would reveal it to be a test case containing a test for the above calculator. Hence, the problem of extracting the test case values and expected results out of such assets is out of scope of this thesis, since our approach envisages automatic knowledge extraction from test cases. In the following, we will focus on test cases which are automatically recognizable as such.
The following chapters will introduce software search engines and reuse-oriented recommendation systems along with the techniques applied in these field. Based on the foundations presented, we are going to create a search engine for reusable software tests and develop a reuse-oriented test-reuse environment, which en- ables users to reuse software tests directly from within their IDE.