CS 451
Software Engineering
Winter 2009
Yuanfang Cai
Room 104, University Crossings 215.895.0298
Testing
Testing only reveals the presence of defects
Does not identify nature and location of defects
Identifying & removing the defect => role of
debugging and rework
Preparing test cases, performing testing, defects
identification & removal all consume effort
Overall testing becomes very expensive :
Levels of Testing
The code contains requirement defects, design
defects, and coding defects
Nature of defects is different for different injection
stages
One type of testing will be unable to detect the
different types of defects
Different levels of testing are used to uncover
these defects
User needs Acceptance testing
Requirement
specification System testing
Design Integration testing
Unit Testing
Different modules tested separately
Focus: defects injected during coding
Essentially a code verification technique, covered
in previous chapter
UT is closely associated with coding
Frequently the programmer does UT; coding
Integration Testing
Focuses on interaction of modules in a
subsystem
Unit tested modules combined to form
subsystems
Test cases to “exercise” the interaction of
modules in different ways
System Testing
Entire software system is tested
Focus: does the software implement the
requirements?
Validation exercise for the system with
respect to the requirements
Generally the final testing stage before the
software is delivered
May be done by independent people
Defects removed by developers
Acceptance Testing
Focus: Does the software satisfy user needs?
Generally done by end users/customer in
customer environment, with real data
Only after successful AT software is deployed
Any defects found,are removed by developers
Acceptance test plan is based on the acceptance
test criteria in the SRS
Other forms of testing
Performance testing
tools needed to “measure” performance
Stress testing
load the system to peak, load generation tools needed
Regression testing
test that previous functionality works alright
important when changes are made
Previous test records are needed for comparisons
Prioritization of testcases needed when complete test suite cannot be executed for a change
Smoke testing
Nightly buildDefect logging and tracking
A large software may have thousands of defects,
found by many different people
Often person who fixes (usually the coder) is
different from who finds
Due to large scope, reporting and fixing of
defects cannot be done informally
Defects found are usually logged in a defect
tracking system and then tracked to closure
Defect logging and tracking is one of the best
Defect logging…
A defect in a software project has a life cycle of
its own, like
Found by someone, sometime and logged along with info about it (submitted)
Job of fixing is assigned; person debugs and then fixes (fixed)
The manager or the submitter verifies that the defect is indeed fixed (closed)
Validation and Verification
Validation
: a different set of activities that
ensures that the software that has been built
is traceable to customer requirements.
Are we building the right product?
Verification
: the set of activities the ensures
that software correctly implements a specific
function.
15
Organizing Testing
Organizing Testing
17
Software Testing Strategies
How do you know when you are done testing?
There is no definitive answer to this question.
Many possible answers:
You are never done testing.
You are done testing when you run out of money or time.
Depends upon the certainty of failure you desire.
Strategic Issues
Specify product requirements in a quantifiable
manner long before testing commences
State testing objectives explicitly.
test coverage, mean time to failure, cost to find and fix defects, remaining defect density or frequency of occurrence
Understand the users of the software and
develop a profile for each user category.
Testing Strategies
Unit Testing – focuses verification effort on the smallest unit of software design. (The component or module)
Considerations:
The module interface is tested to ensure that information properly flows into and out of the program unit under test. All independent paths through the control structure are
exercised to ensure that all statements in a module have been executed at least once.
Boundary conditions are tested to ensure that the module operates properly.
21
Testing Strategies
Unit Testing
Procedure
Unit testing is normally considered as an adjunct to the coding step. Design of tests can be done before
coding begins (Agile
Testing Strategies
Unit Testing Procedure
Since each component is not a stand alone program, a bit of work must be done to perform the test.
Create a driver, usually a main program, that accepts test case data and passes it to the component to be tested. It also may print relevant results.
Stubs replace modules that may not be coded and are
subordinate to the component to be tested. A stub may do minimum data manipulation , provides verification of entry, and returns control to the module undergoing testing.
Junit Example
Java Code to Test
public class Math {
static public int add(int a, int b) { return a + b;
} }
Junit example
JUNIT test case
import junit.framework.*;
public class TestMath extends TestCase { public void testAdd() {
int num1 = 3; int num2 = 2; int total = 5; int sum = 0;
sum = Math.add(num1, num2); assertEquals(sum, total);
Integration Testing
Why is unit testing not enough? Data can be lost across an interface.
One module can have an inadvertent effect on another.
Global data structures can present problems
Individually acceptable imprecision may become magnified.
Integration testing is a systematic technique for constructing the software architecture while at the same time conducting tests to uncover errors
associated with interfacing.
The objective is to take unit tested components and build a program structure that has been dictated by
Integration Testing
There is a tendency to attempt non-incremental
integration – construct the whole program via a “big-bang” approach.
Thus to combine all components at once and test it as an entire program.
Top Down Integration is an incremental approach to construction of the software architecture. Modules are integrated by moving downward through the control hierarchy, beginning with the main control module.
Integration Testing -- Top Down
Modules subordinate to the main control module are incorporated into the structure in either a depth first or breath first manner.
Integration Testing -- Top Down
The integration process is preformed in five steps:
1. The main control module is used as a test driver, and stubs are substituted for all components directly
subordinate to the main control module.
2. Depending on the integration approach selected (i.e. depth or breath first), subordinate stubs are replaced one at a time with actual components.
3. Tests are conducted as each component is integrated.
4. On completion of each set of tests, another stub is replaced with the real component.
5. Regression testing may be conducted to ensure that new errors have not been introduced.
Bottom Up Integration
Begins construction and testing with atomic modules. Since all subordinates are always available, there is no need for stubs.
Bottom up integration follows the following steps:
1. Low-level components are combined into clusters that perform a specific software subfunction.
2. A driver (control program for testing) is written to coordinate test cases input and output.
3. The cluster is tested.
4. Drivers are removed and clusters are combined moving upward in the program structure.
Regression Testing
Each time a new module is added, the software changes.
These changes may cause problems with functions that previously worked.
Therefore, we must re-execute some subset of tests that were already conducted to ensure that changes have not propagated unintended side effects.
Any time you test, if successful you will find errors. When those errors are corrected, software changes. Therefore,
regression testing must be performed in order to ensure that those changes did not propagate unintended side effects.
Regression tests may be conducted manually – I do not recommend this.
Unit Testing in the OO Context
With OO the concept of the unit changes.
Because of the dependence of sub classes, unit testing is a bit more complicated.
In the OO context, unit testing is basically Class testing.
Cannot simply test methods in a super class and assume they are still valid in each subclass
Subclasses may re-define important attributes or subordinate methods
Integration Testing
in the OO Context
Unlike non OO programming, OO programming
does not have an hierarchical control structure.
Integrating one object at a time is also difficult
because of the direct and indirect interactions
of the components that make up the class.
There are two basic strategies:
Thread-based
Test Strategies for OO Software
Thread-based testing integrates the set of
classes required to respond to one input or
event for the system.
Each thread is integrated and tested
individually.
Regression testing is applied to ensure no side
effects occur.
Good for GUIs, other event-driven
Test Strategies for OO Software
Use-based testing begins the construction of
system by testing those classes that use very
few server classes.
After independent classes are tested, the next
layer of classes, called dependent classes are
tested.
This continues until the entire system is
constructed.
Test Strategies for OO Software
Drivers can be used to test operations at the
lowest level and for testing groups of classes.
A driver may be used to replace the user
interface.
Stubs may be used where collaboration
between classes is required but one or more of
the collaborating classes is not fully
Test Strategies for OO Software
Cluster testing is one step in integration testing
of OO software.
Here, a cluster of collaborating classes is
exercised by designing test cases that attempt
to uncover errors in the collaborations.
Validation Testing
Validation Testing
Validation testing begins at the culmination of integrations testing.
Testing focuses on the user-visible actions and user-recognizable output from the system.
Validation can be defined as:
Validation succeeds when software functions in a manner that can be reasonably expected by the customer.
Software validation is achieved through a series of tests that demonstrate conformity with software
Test Strategies for Publicly-used
Software
What to do if your product does not have one
or a few customers?
Most software product builders use a process
called alpha and beta testing to uncover errors
that only the end-user seems able to find.
The alpha test is conducted at the developer’s
site by end-users.
The beta test is conducted at the end-user
Recovery Testing
Recovery Testing – most computer-based
systems must recover from faults and resume
processing within a prespecified time.
Recovery testing is a system test that forces
the software to fail in a variety of ways and
verifies that recovery is properly preformed.
Security Testing
Any computer system that manages sensitive
information or causes actions that can improperly harm or benefit individuals is a target for improper or illegal penetration.
Lot’s of security issues.
In your code
In libraries and frameworks
In database access
In OS layers
Security Testing
During security testing, the tester plays the role(s) of individual who desires to penetrate the system.
Penetration testing requires high skill and the ability to think outside the box
Best to get an independent testing group that specializes in penetration testing
Usually it’s not the software, but people issues.
People often give out secure information. No system can truly protect against this 100%.
Stress Testing
So far all testing assumed normal program functions and performance.
Stress tests are designed to confront programs with abnormal quantity, frequency, or volume.
Multiple interrupts per second
Increased data rate
Test max memory
Performance Testing
Designed to test the run-time performance of software within the context of an integrated system.
Should be performed at all steps of the testing process.
Not until system elements are fully integrated that performance of a system can really be ascertained.
Often requires hardware and software instrumentation to measure resource utilization.
Summary
Validation and Verification
Different Types of Testing
Context