Software Testing
Testing can only reveal the presence of errors and not the absence of errors.
A successful test is a one which discovers more, yet-undiscovered, errors
An ideal testing process never ends.
Definition: Testing is a process of executing a program with data, with the sole intention of finding errors in the program.
Definition: “Quality assurance is a planned and systematic pattern of all actions necessary to provide adequate confidence that the item conforms to established requirements.”
Objectives of software testing
To find more errors in the software design and implementation
To confirm that the product meets (satisfies) the requirements, including non-functional
requirements
Testing is one type of validation
— Validation is a process that answers to the question,“Are we developing the right product?”
Testing and Debugging
Debugging requires testing efforts to find the errors first, and then retest the program after correcting the errors. (Thus, testing occurs before and after debugging.)
Testing is usually performed manually .
Debugging is often performed with the help of tool support.
Definitions
Testing is a process of finding the presence of errors.
Debugging is a process of finding and correcting errors.
Test Case
Example:
Let y = f(x) be the function to be tested; a test case will indicate for what range of values of (input) x the function will return (output) and for what input values the function will not return an output
Definition
A test case is a description of input/output relationship of one particular testing activity -
a set of “inputs” and expected “output(s)”.
Sample Test Cases
The following are some of the test cases for the factorial function “ int factorial (int x) ”
Input x Expected output Actual output
x = 0 1
x = 1 1
x >= 0 x!
x < 0 error
Testing principles
All tests should be traceable to their source (requirements and/or code).
Testing activity should be planned (test cases prepared) well before testing process is
exercised.
Exhaustive testing is not possible.
To be most effective, testing should be conducted by a third party – not by the developers.
— Quality Assurance team
Test case design methods
Both black-box and white-box testing are required for a complete testing process.
black-box testing
The software is assumed to be a black box (with unknown code). Test cases are derived from problem descriptions, requirements, design documents, etc.
white-box testing
Test cases are derived by examining the source code.
Black-box and White-box testing
• Test cases can be derived before the program is written.
• The objective is to ensure that the program meets (satisfies) the requirements.
• Knowledge of the application domain is necessary but not about the program or the programming language.
• Test cases can be derived only after the program is complete.
• The objective is to ensure that all parts of the logic of the program are correct.
• Knowledge of the
programming language is needed but possibly not the application domain.
White-box testing
Also called glass-box testing and structural testing
Derivation of test cases from the program’s structure.
What we would like to do...
exercise every possible execution path
What we are able to do...
exercise every possible statement at least once path coverage
statement coverage
Example – computing factorial
int factorial (int n) { int i, ret = 0;
if (n < 0) return ret;
i = n;
ret = 1;
while (i > 0) { ret = ret * i;
i = i - 1;
}
return ret;
}
Input n
Expected output n < 0 Error
n==0 1
n > 0 n!
Test cases
Flow graph notation
Also called “program graph”
Abstractly describes the control flow of the program
Consists of nodes and arrows
• Each node represents a contiguous segment of code (statement(s) or expression(s)).
• Arrows represent control flow transitions.
if if-else case
while sequence
repeat- until for
Flow graph for the factorial example
ret = 0;
(n < 0) return ret;
i = n;
ret = 1;
(i > 0)
ret = ret * i;
i = i - 1;
return ret;
int i, ret=0;
int factorial (int n) { int i, ret = 0;
if (n < 0) return ret;
i = n;
ret = 1;
while (i > 0) { ret = ret * i;
i = i - 1;
}
return ret;
}
For path coverage we must ensure that every node is executed by at least one test case.
1) number nodes 2) check test cases
ret = 0;
(n < 0) return ret;
i = n;
ret = 1;
(i > 0)
ret = ret * i;
i = i - 1;
return ret;
int i, ret=0;
1 2 3 4 5 6
7
8
Case statements n < 0 1,2,3,8 n==0 1,2,3,4,5,7 n > 0 1,2,3,4,5,6,7
Test cases
Another example
Remove an integer from an array and shift the array, adding 0 at the end.
void squeeze (int k, int[] arr) { boolean flag = false;
int index = 0, j;
while (index < arr.length && !flag) { if (arr[index] == k) {
for (j = index; j < arr.length; j++) arr[j] = arr[j+1];
arr[N] = 0;
flag = true;
} else
index = index + 1;
}
} Draw the flow graph
and generate the test cases.
What about this one?
public char reportGrade (int[] exams) { char ch = ‘ ‘;
int s,i;
s = sum (exams) / exams.length;
if (s >= 90) ch = 'A';
else if (s < 90 && s >= 75) ch = 'B';
else if (s < 75 && s >= 60) ch = 'C';
else if (s < 60 && s >= 50) ch = 'D';
else
ch = 'F';
return ch;
} public int sum (int[] exams) {
int n, total = 0;
for (n = 0; n < exams.length; n++) total = total + exams[n];
return total;
}
White-box Testing Issues
When is statement coverage impossible?
When is statement coverage impractical?
Does statement coverage ensure correctness?
Black-box testing
Black-box testing is a testing approach in which test cases are generated without knowledge of the program (source code).
also called functional testing or behavioral testing
Test cases can be derived even before coding begins.
Example: Automated teller machine (ATM)
Consider the functional requirement for a withdraw operation. Test cases
include...
withdraw an amount less than the balance
withdraw an amount equal to the balance
withdraw an amount more than the balance
withdraw an amount beyond a maximum
withdraw a negative amount (robustness testing)
Objectives of black-box testing
Identify improper implementation of functionalities.
Identify missing functionalities.
Identify errors in interfaces among the components of the software.
... including external software such as a database.
Identify performance (and other non-functional) errors.
Identify initialization and/or termination errors.
Equivalence partitioning
A black-box testing method
Partition the input domain (the set of potential input values) into mutually disjoint subsets, called equivalence classes.
Every element of a subset shares one
characteristic with all other elements in the same subset.
— As a consequence, it is sufficient to select one element from each subset, there by reduce the time for testing.
Input domain
Output domain
Equivalence partitioning does not impose any constraints on the output domain.
Software system
Equivalence partitioning example
Factorial example
• Three subsets of input (the parameter ‘n’) are negative integers
zero positive integers
Equivalence classes:
negative integers - invalid factorial input zero - base case for recursive definition of factorial
positive integers - factorial defined recursively
More examples
A function that accepts a 2-digit positive integer as input
Calculate income tax based on net income
All integers less than 10
Integers between 10 and 99 both inclusive
All integers greater than 99
Income between $0 through $33000
Income between $33001 through $150000
Income above $150000
Limitations of Equivalence Partitioning
User is responsible for ensuring the mutual exclusiveness of the partitions.
Too difficult to determine equivalence classes
Too many equivalence classes
Boundary Value Analysis
Boundary value analysis is a technique that complements the equivalence partitioning approach.
Boundary value analysis can help to partition the input domain when the input lies within a range(s) of values.
Boundary value analysis works best when parameters (if more than one) are fairly independent of each other.
Guidelines for boundary value analysis
Select the members at the boundary of each equivalence class.
Select a “middle” value.
Also select those values which are adjacent to the boundaries.
Include out of range values for robustness.
Example:
Suppose an equivalence class contains values from 120 through 175.
Test case values:
119, 120, 121, 149, 174, 175, 176
More examples
Calculate property tax based on the value of the property.
• Single variable function
Calculate the score of a competition based on the number of events participated and the score given for each such event.
• Two-variable function
X X
Ymin Ymax
Boundary value analysis for a two-variable function - f (x,y)
Boundary values for different data types
Data type Boundary cases
Integer, Double (Real) Range of values given in the problem String Empty string, String with one char,
String of maximum length, String of length greater than the maximum
Array Similar to String
File Empty file, File with one record, file of max. size, file overflow
Testing Phases
Unit testing
• Test individual components in isolation.
• Often performed by the developer.
Module testing
• Test a group of components (units) that interact with each other or a group that implements a particular functionality.
• May be a group of procedures for the imperative paradigm; a group of methods or classes in the O-O paradigm.
• Interfaces among these components must also be tested.
Testing Phases(continued)
Subsystem testing
• Test a group of modules and their interfaces.
• A subsystem may be a higher-level group of modules in the procedural paradigm or a package in the O-O paradigm.
• Interfaces between modules, particularly access restrictions across modules, must be tested along with functionalities
Integration testing
• This is a combination of module and subsystem testing.
Testing Phases(continued)
Acceptance testing
• Test the entire system as one single product.
• GUI testing falls into this category.
• Also requires testing of interfaces between external components such as a database or a device driver.
System testing
• Testing to conform that the software product meets the standards and/or specific criteria set by the customers/users.
Classifications of Testing
Black-box testing and White-box testing classify testing methods – processes to generate test cases and test data.
Stages of testing classify the portion of the software product that must be tested at any one time.
Combining these two...
— A unit can be tested both by black-box testing and by white-box testing.
— A module may undergo black-box testing and white-box testing.
Conclusion: Each classification defines a different perspective of testing
Further classifications of testing
Bottom-up testing
• Test the units first, then the modules, then the
subsystems and so on until the entire product is tested
Top-down testing
• Test the entire software product as one unit, then test its individual subsystems giving emphasis to their interfaces, then test the modules and so on until the components are broken down at the unit level.
• May require “stubs”.
Stubs - example
public static void main(String[] args) { int[] arr = new int[20];
read(arr);
sort(arr);
if (search(arr, 23))
System.out.println(“23 is in the array”);
else
System.out.println(“23 NOT in the array”);
}
public void read(int[] array) { //STUB
System.out.println (“read – to be implemented”);
}
public void sort(int[] array) { //STUB
System.out.println (“sort to be implemented”);
}
public boolean search(int[] array, int k) { //STUB return true;
}
A few more testing classifications
Desk Checks
• Developers often perform walk-throughs of their own, or colleague’s code.
Alpha testing
• Testing performed at the developer’s site.
• Generally performed by the QA team, not by the development team.
• Purpose: to check and correct errors before the release of the actual product
Beta testing
• Testing performed at the customer’s site.
• Generally conducted by the end users (may be the customer)
• Purpose: to report back to the developers any errors before product release
Regression testing
Regression testing is the act of repeating a suite of test cases after a significant change to a system.
Regression testing is an important part of software maintenance.
Documentation of test cases is essential to successful regression testing.
Dependency graphs can help to determine which components require regression testing.
Object-Oriented Software Testing
Research on O-O testing reports that testing methods proposed for
imperative approach are not adequate for O-O approach.
O-O poses additional problems due to the distinguishing characteristics such as encapsulation, inheritance and
polymorphism.
Class testing
Class testing are modules.
Methods are units.
Interactions within a class are module interfaces.
Interactions between different classes are subsystem interfaces.
Testing O-O Encapsulation
Within O-O design encapsulation specifies access restrictions to the structural and behavioral members of a class. Such encapsulation should be tested.
• Check whether all the public members are accessible from everywhere.
• Check whether all the private members are hidden outside the class.
• If language supports other access mechanisms, such as protected or class-specific scope, those features should be tested for both accessibility and non-accessibility.
Testing Inheritance
Subclasses introduce new features (attributes and methods) and can modify inherited
behaviors.
Modified behaviors must be tested in the context where they are used.
Polymorphism adds complexity (e.g., co-variance or contra-variance?)
Multiple inheritance adds complexity (What if two methods with the same signature conflict? i.e., both are inherited from different superclasses.)
Any change to a superclass requires regression testing of all subclasses.
