Unit Testing with JUnit and CppUnit

Unit Testing with

JUnit and CppUnit

2

Software Testing

Fundamentals (1)

z

What is software testing?

z The process of operating a system or component under

specified conditions, observing or recording the results, and making an evaluation of some aspect of system or component. (IEEE Definition)

z

Why do we test?

z To ensure the quality and satisfactionof the product z To gain the confidencein the correctness of the product z

Testing vs. debugging

z Testingis to show that a program has bugs z Debuggingis to locate and correct the error or

3

Software Testing

Fundamentals (2)

z

Software Testing Techniques

z

White-box testing

z

The tester has access to the details of the

program under test and performs the testing

according to such details.

z

Examples: path testing, branch testing

z

Black-box testing

z

Black-box testing treats the program under test as

a “black box.” No knowledge about the

implementation is assumed.

z

Examples: equivalent partitioning, boundary value

analysis

4

Software Testing

Fundamentals (3)

z

Testing should begin “in the small” and progress

toward testing “in the large”

z Unit testing

z Concentrates on each unit (i.e., component) of the software

z Integrated testing

z Building a system from its components and testing the

resultant system for detecting component interaction problems

z System testing

z Concern with testing an increment to be delivered or entire

system

z Acceptance testing

z Performed by the clients/users to confirm that the product

5

Unit Testing

z

Unit Testing

z Is normally considered as an adjunct to the coding step z Focuses verification effort on the smallest unit of software

design – the software component or module

z Using the component-level design description as a guide z Provide a release criterion for a programming task z

Unit Testing in the OO Context

z Smallest testable unit is the encapsulated class z Conducting class testing (i.e., unit testing)

z Methods within the class are tested

z The state behavior of the class is examined

Unit Testing: Methodology

Unit Testing: Methodology

. . .

int Max(int x, int y) { . . . . . . } }; class XTest { . . . void testMax() { X x;

int max = x.max(10,5); CPPUNIT_ASSERT(max == 10); }

. . . };

Example #1

Unit Testing: Methodology

. . .

int Max(int a[], int size) { . . . . . . } }; class XTest { . . . void testMax() { X x; int a[5]={3,4,5,1,0}; int max = x.max(a,5); CPPUNIT_ASSERT(max == 5); }

. . . };

Unit Testing: Methodology

. . .

int Sum(int a[], int size) { . . . . . . } }; class XTest { . . . void testSum() { X x; int a[5]={3,4,5,1,0}; int sum = x.Sum(a,5); CPPUNIT_ASSERT(sum == 13); } . . . }; Example #3 10 class X { . . . void f() { . . . . . . } }; class XTest { . . . void testf() { X x; x.f(); CPPUNIT_ASSERT(…); } . . . }; Statement coverage: are all statements of

f() executed?

Unit Testing Example:

Statement coverage (1)

11 class X { . . . void f(bool x) { . . . if (x) { . . . } . . . } }; class XTest { . . . void testf() { X x; x.f(false); CPPUNIT_ASSERT(…); } . . . };

There are unexecuted statements

Unit Testing Example:

Statement coverage (2)

Unit Testing Example:

Statement coverage (3)

13 class X { . . . void f(bool x) { . . . if (x) { . . . } . . . } }; class XTest { . . . void testf() { X x; x.f(true); CPPUNIT_ASSERT(…); } . . . }; All statements are executed Brach coverage: are all control transfer exercised? Unexercised

Unit Testing Example: Branch

coverage (1)

Unit Testing Example: Branch

coverage (2)

15 class X { . . . void f(int x) { . . . do { x--; . . . } while (x>0); . . . } }; class XTest { . . . void testf() { X x; x.f(0); CPPUNIT_ASSERT(…); } . . . }; Statement coverage? Branch coverage? Yes No

Unit Testing Example: Branch

coverage (3)

Unit Testing: Difficult to Test?

// The file is not under // control.

// How to do assert // for the file? }

. . . };

FileName should be parameterized

Sometimes, you need to rewrite your code to make it easier to test

Unit Testing: Difficult to Test?

. . .

void LoadFromFile(string fileName) { ifstream fin(fileName.c_str()); . . . } }; class XTest { . . . void testLoadFromFile() { X x; x.LoadFromFile(”a_file”) ; // The file is under // control

} . . . };

Better

Unit Testing: Difficult to Test?

. . .

void LoadFromFile(ifstream &fin) { . . . } }; class XTest { . . . void testLoadFromFile() { X x; ifstream fin(”a_file”); x.LoadFromFile(fin) ; // The file is under // control } . . . }; Another possibility

Though, the file is now assigned by the tester. The assert may still be difficult to implement, because the LoadFromFile()

function is large. Sometimes, you need to break a large function into

Unit Testing: Difficult to Test?

. . .

void LoadFromFile(ifstream &fin) { . . . if (condition #1) LoadPart1(); . . . LoadPart2(); } }; class XTest { . . . void testLoadFromFile() { } void testLoadPart1() { } void testLoadPart2() { } . . . }; Breaking a large function into smaller ones.

Test each smaller function first.

Testing for LoadFromFile().

20

Advantages of Unit Testing

z

Write test = understand better what has to be done

Silly bugs appear immediately = less time spent on

debugging

z

Provide a working specification of your functional

code = tests are good documentation

z

Speed-up the development – write test once, use

test to find errors many times

z

Gain confidence in your code

Repeatable and deterministic

z

Regression tests – incorrect changes discovered

immediately; unautomated refactoring enabled

21

Old-fashioned Low-level

Testing

z

Stepping through a debugger

drawbacks:

1. not automatic

2. time-consuming

z

Littering code with stream output calls

drawbacks

:

1. makes code ugly

(ok, nowadays you could use aspects to avoid it;-))

2. generates too much information

22

What is xUnit?

z

An automated unit test framework

Provides the Driver for unit(s)

Provides automatic test runs

Provides automatic result checks

Available for multiple languages:

z JUnit(from Kent Beck (XP) and Erich Gamma(Gang of Four)) z CppUnit

z httpUnit z NUnit z …

23

The Goals of JUnit

z

To write a

framework

within which we have some

glimmer of hope that

developers will actually write

tests

.

z The framework has to use familiar tools, so there is little

new to learn.

z

The second goal of testing is

creating tests that

retain their value over time

.

z Someone other than the original author has to be able to

execute the tests and interpret the results.

z

Creating a setup or fixture is expensive

z A framework has to enable reusing fixturesto run different

tests.

24

The Design of JUnit

z

Patterns Generate Architectures

z Used to present the design of JUnit.

z The idea is to explain the design of a system by starting

with nothing and applying patterns, one after another, until you have the architecture of the system.

z Presentation flow

(1) Present the architectural problem to be solved (2) Summarize the pattern that solves it

(3) Show how the pattern was applied to JUnit.

z The details can refer to JUnit A Cook's Tour

25

The Patterns Used in JUnit

26

How to Use JUnit (1)

z Write a test case (Fixture)

z Create your own test case as a subclass of JUnitTestCase z Add an instance variable for each known object in the fixture z Override the setUp()method to initialize object(s) under test z Override the tearDown()method to release object(s) under test z Run the test

z Define a public test???()method for exercising the object(s) under test

z Verify the result

z Assert expected result of the test case using assertEquals(), assertTrue, …

z Clean up the fixture

z through the tearDown()method

public class BookTest2 extends TestCase { private Collection collection;

protected void setUp(){ collection = new ArrayList(); }

protected void tearDown(){ collection.clear(); }

public void testEmptyCollection(){ assertTrue(collection.isEmpty()); }

27

How to Use JUnit (2)

z Suite management

z A test suiteis a collection of test cases that are intended to be

used to show that a program under test has some specified set of behaviors

z Write a static suite()containing all the test???()in the fixture

class

z Optionally define a main()method that runs the TestCasein

batch mode z Error vs. Failures

z Error:unanticipatedproblem like an

ArrayIndexOutOfBoundsException

z Failure:is anticipatedand can be checked with assertions public static void main(String[] args) {

junit.textui.TestRunner.run(suite()); }

public static Test suite(){

return new TestSuite(BookTest.class); }

public static void suite(){

TestSuitesuite = new TestSuite();

suite.addTest(new BookTest("testEquals"));

suite.addTest(new BookTest("testBookAdd")); return suite;

}

28

JUnit FAQ: Best Practices

z

When should tests be written?

z Test-first programming is practiced by only writing new

code when an automated test is failing.

z When all the tests pass, you know you're done!

z When a bug is reported, first write unit test(s) to expose the

bug(s), then fix them.

z This makes it almost impossible for that particular bug to

resurface later.

z Good tests tell you how to best design the system for its

intended use.

z Test-driven developmentis a lot more fun than writing tests

29

JUnit FAQ: Best Practices

z

Do I have to write a test for everything?

z Investments in testing are equal investments in design.

z If defects aren't being reported, and your design responds

well to change, then you're probably testing enough.

z If you're spending a lot of time fixing defects and your

design is difficult to grow, you should write more tests.

z If something is difficult to test, it's usually an opportunity for

a design improvement.

30

JUnit FAQ: Best Practices

z

How simple is “too simple to break”?

z

If it can't break on its own, it's too simple to break.

Example

z

getX() method cannot break unless the compiler

is also broken. Therefore, don't test getX().

setX() method is also too simple to break.

However, if it does any parameter validation, you

likely need to test it.

31

JUnit FAQ: Best Practices

z

How often should I run my tests?

z Ideally every time the code is changed.

z Make sure all your unit tests always run at 100%.

z Frequent testing gives you confidencethat your changes

didn't break anything.

z For larger systems, you may just run specific test suites

that are relevant to the code you're working on.

z Run all the tests of the a system at least once per day (or

night).

32

JUnit FAQ: Best Practices

z What do I do when a defect is reported? z Write a failing testthat exposes the defect

z When the test passes, you know the defect is fixed!

z This is a learning opportunity

z Perhaps the defect could have been prevented by being more aggressive about testing everything that could reasonably break. z Why not just use print?

z It requires that output be scanned manually every time the

program is run to ensure that the code is doing what's expected.

z Tests should retain its value over time. z Why not just use a debugger?

33

JUnit Primer: Testing Idioms

z

Testing Idioms

z Code a little, test a little, code a little, test a little...

z Begin by writing tests for the areas of code that you're most

worried about breaking.

z Write tests that have the highest possible return on your

testing investment.

z When you need to add new functionality to the system,

write the tests first.

z If you find yourself debugging using System.out.println(),

write a test case instead.

z The next time someone asks you for help debugging, help

them write a test.

z Don't deliver software that doesn't pass all of its tests.

34

CppUnit Cookbook (1)

z Simple Test Case (Ordinarily, you will not use such simple test) z Subclass the TestCaseclass.

z Override the method runTest().

z When you want to check a value, call CPPUNIT_ASSERT(bool)

and pass in an expression that is true if the test succeeds

class ComplexNumberTest : public CPPUNIT_NS::TestCase { public:

ComplexNumberTest( std::string name ) : CppUnit::TestCase( name ) {}

void runTest() {

CPPUNIT_ASSERT( Complex (10, 1) == Complex (10, 1) ); CPPUNIT_ASSERT( !(Complex (1, 1) == Complex (2, 2)) ); }

};

bool operator ==( const Complex &a, const Complex &b ) { return a.real == b.real && a.imaginary == b.imaginary; }

35

CppUnit Cookbook (2)

z Fixture- a known set of objects served as a base for a set of test cases z To add new tests

z Add member variables for each part of the fixture z Override setUp()to initialize the variables

z Override tearDown()to release any resources allocated in setUp()

class ComplexNumberTest : public CPPUNIT_NS::TestFixture

{ private: Complex *m_10_1, *m_1_1, *m_11_2; protected: void setUp() { m_10_1 = new Complex( 10, 1 ); m_1_1 = new Complex( 1, 1 ); m_11_2 = new Complex( 11, 2 ); } void tearDown() { delete m_10_1; delete m_1_1; delete m_11_2; } }; 36

CppUnit Cookbook (3)

z How do you write and invoke individual tests using a fixture? z Write the test case as a method in the fixture class

z Create a TestCallerwhich runs that particular method

class ComplexNumberTest : public CPPUNTI_NS::TestFixture { private: Complex *m_10_1, *m_1_1, *m_11_2; protected: void setUp() {...} void tearDown() {...} void testEquality() { CPPUNIT_ASSERT( *m_10_1 == *m_10_1 ); CPPUNIT_ASSERT( !(*m_10_1 == *m_11_2) ); } void testAddition() { CPPUNIT_ASSERT( *m_10_1 + *m_1_1 == *m_11_2 ); } }; CPPUNIT_NS::TestCaller<ComplexNumberTest> test("testEquality", &ComplexNumberTest::testEquality ); CPPUNIT_NS::TestResult result; test.run( &result );

37

CppUnit Cookbook (4)

z Use TestSuiteclass that runs any number of TestCasestogether CPPUNIT_NS::TestSuite suite;

CPPUNIT_NS ::TestResult result;

suite.addTest( new CPPUNIT_NS::TestCaller<ComplexNumberTest>( "testEquality",

&ComplexNumberTest::testEquality ) ); suite.addTest( new CPPUNIT_NS::TestCaller<ComplexNumberTest>(

"testAddition",

&ComplexNumberTest::testAddition ) );

suite.run( &result );

z TestSuitecan contain any object implementing the Testinterface

CPPUNIT_NS::TestSuite suite; CPPUNIT_NS ::TestResult result;

suite.addTest( ComplexNumberTest::suite() ); suite.addTest( SurrealNumberTest::suite() ); suite.run( &result );

38

CppUnit Cookbook (5)

z Run TestSuiteusing TestRunner z Using Helper Macros

z Defined in cppunit/extensions/HelperMacros.hwhich must be integrated for initiating and finishing a test suite

(CPPUNIT_TEST_SUITE and CPPUNIT_TEST_SUITE_END) z Rewrite the fixture to include the HelperMacros.h

#include <cppunit/extensions/HelperMacros.h>

class ComplexNumberTest : public CppUnit::TestFixture {...}

CPPUNIT_TEST_SUITE( ComplexNumberTest ); //pass class name

CPPUNIT_TEST( testEquality ); // declare test case CPPUNIT_TEST( testAddition );

CPPUNIT_TEST_SUITE_END(); //end suite declartion

39

CppUnit Cookbook (6)

z If all the tests pass, you'll get an informative message. z If any fail, you'll get the following information:

z The name of the test case that failed

z The name of the source file that contains the test z The line number where the failure occurred

z All of the text inside the call to CPPUNIT_ASSERT() which detected the

failure

#include <cppunit/ui/text/TestRunner.h> #include "ComplexNumberTest.h"

int main( int argc, char **argv) {

CPPUNIT_NS::TextUi::TestRunner runner;

runner.addTest(ComplexNumberTest::suite());

bool wasSuccessful = runner.run(); return wasSuccessful ? 0 : 1; }

z Run TestSuiteusing TestRunner

//ComplexNumberTest.h CPPUNIT_TEST_SUITE( ComplexNumberTest ); CPPUNIT_TEST( testEquality ); ... CPPUNIT_TEST_SUITE_END(); 40

CppUnit Cookbook (7)

z Check whether the passed expression returns the value True

z CPPUNIT_ASSERT_EQUAL

z Check whether the first parameter is like the second one

z CPPUNIT_ASSERT_THROW

z Check whether the passed expression throws an exception of the passed type

z CppUnit Examples z See LinkedListTest

41

Test-driven Development (TDD)

z

Test-driven development (TDD) is an evolutionary

approach to development which combines

z Test-first development

z Write a test before you write just enough production code to fulfill that test

z Refactoring

z

The goal of TDD

z Think through your design before your write your functional

code

z To write clean code that works

z

TDD is primarily a design technique with a side effect of

ensuring that your source code is thoroughly unit tested

42

Unit testing and

Test-driven Development (TDD)

z

How to write JUnit/CPPUnit tests? Do it TDD!

z

Workflow:

1.

Think about functionality to be implemented &

scenarios in which the unit to be tested will play a role.

2.

Create a stub of the unit you want to implement.

3.

Write a test for each scenario and/or use of the unit.

4.

Make the unit fail the tests (nothing is implemented

yet!).

5.

Develop the unit until it passes every test.

z

Encountered new scenario/use?

43

References

z

W.K Chen, “ObjectOriented Programming

-Software testing”

z

JUnit A Cook's Tour.

http://junit.sourceforge.net/doc/cookstour/cookstour.

htm

z

CppUnit Cookbook.

http://cppunit.sourceforge.net/doc/1.8.0/cppunit_coo

kbook.html

z

Krzysztof Pietroszek, “Unit testing with JUnit and

CPPUnit”.

http://swen.uwaterloo.ca/~se2/project/project-junit.pdf