Software Engineering: A Practitioner’s Approach, 6/e Software Engineering: A Practitioner’s Approach, 6/e
Chapter 13 Chapter 13
Software Testing Strategies Software Testing Strategies
copyright © 1996, 2001, 2005
R.S. Pressman & Associates, Inc.
For University Use Only
May be reproduced ONLY for student use at the university level
when used in conjunction with Software Engineering: A Practitioner's Approach.
Any other reproduction or use is expressly prohibited.
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
2
Software Testing Software Testing
Testing is the process of exercising a Testing is the process of exercising a
program with the specific intent of finding program with the specific intent of finding errors prior to delivery to the end user.
errors prior to delivery to the end user.
Test strategy provides a road map; Test strategy provides a road map;
Steps to be conducted Steps to be conducted
When When
How much effort, time, resources How much effort, time, resources
Test strategy Test strategy
Test planning Test planning
Test case design Test case design
Test execution Test execution
Resultant data collection and evaluation Resultant data collection and evaluation
Verification and Validation Verification and Validation
V&V V&V
Testing is part of V&V Testing is part of V&V
Verification: Are we building the product right? Verification: Are we building the product right?
Software correctly implements a specific function Software correctly implements a specific function
Validation: Are we building the right product? Validation: Are we building the right product?
Software is traceable to customer requirements Software is traceable to customer requirements
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
4
What Testing Shows What Testing Shows
errors errors
requirements conformance requirements conformance
performance performance
an indication an indication
of quality
of quality
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
6
Who Tests the Software?
Who Tests the Software?
developer
developer independent tester independent tester
Understands the system Understands the system
but, will test "gently"
but, will test "gently"
and, is driven by "delivery"
and, is driven by "delivery"
Must learn about the system, Must learn about the system, but, will attempt to break it but, will attempt to break it
and, is driven by quality
and, is driven by quality
Testing Strategy Testing Strategy
unit test
unit test integration integration test test
validation validation
test test system
system
test test
Requirement -> high-order tests: validation test, Requirement -> high-order tests: validation test, System test
System test
Design -> Integration test Design -> Integration test
Code -> unit test Code -> unit test
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
8
Testing Strategy Testing Strategy
We begin by ‘ We begin by ‘ testing-in-the-small’ testing-in-the-small’ and move toward and move toward
‘ ‘ testing-in-the-large’ testing-in-the-large’
For conventional software For conventional software
The module (component) is our initial focus The module (component) is our initial focus
Integration of modules follows Integration of modules follows
For OO software For OO software
our focus when “testing in the small” changes from an our focus when “testing in the small” changes from an individual module (the conventional view) to an OO class individual module (the conventional view) to an OO class
that encompasses attributes and operations and implies that encompasses attributes and operations and implies communication and collaboration
communication and collaboration
10
Strategic Issues Strategic Issues
Specify product requirements in a quantifiable manner long before Specify product requirements in a quantifiable manner long before testing.
testing.
State testing objectives explicitly. State testing objectives explicitly.
Understand the users of the software and develop a profile for each Understand the users of the software and develop a profile for each user category.
user category.
Develop a testing plan that emphasizes “rapid cycle testing.” Develop a testing plan that emphasizes “rapid cycle testing.”
Build “robust” software that is designed to test itself Build “robust” software that is designed to test itself
Use effective formal technical reviews as a filter prior to testing Use effective formal technical reviews as a filter prior to testing
Conduct formal technical reviews to assess the test strategy and test Conduct formal technical reviews to assess the test strategy and test cases themselves.
cases themselves.
Develop a continuous improvement approach for the testing Develop a continuous improvement approach for the testing process.
process.
Unit Testing Unit Testing
module module
to be to be tested tested
test cases test cases
results results software
software
engineer
engineer
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
12
Unit Testing Unit Testing
interface interface
local data structures local data structures boundary conditions boundary conditions
independent paths independent paths
error handling paths error handling paths module
module to be to be tested tested
test cases
test cases
Unit Test Environment Unit Test Environment
Module Module
stub stub stub stub
driver driver
interface interface
local data structures local data structures boundary conditions boundary conditions independent paths independent paths error handling paths error handling paths
test cases
test cases
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
14
Integration Testing Strategies Integration Testing Strategies
Options:
Options:
• • the “big bang” approach the “big bang” approach
• • an incremental construction strategy an incremental construction strategy
Top Down Integration Top Down Integration
top module is tested with top module is tested with stubs
stubs
stubs are replaced one at stubs are replaced one at a time, "depth first"
a time, "depth first"
as new modules are integrated, as new modules are integrated, some subset of tests is re-run some subset of tests is re-run
A A
B B
C C
D D E E
F F G G
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
16
Bottom-Up Integration Bottom-Up Integration
drivers are replaced one at a drivers are replaced one at a time, "depth first"
time, "depth first"
worker modules are grouped into worker modules are grouped into builds and integrated
builds and integrated A A
B B
C C
D D E E
F F G G
cluster
cluster
Sandwich Testing Sandwich Testing
Top modules are Top modules are tested with stubs tested with stubs
Worker modules are grouped into Worker modules are grouped into builds and integrated
builds and integrated A A
B B
C C
D D E E
F F G G
cluster cluster
Top-Down: Needs creating stubs (overhead) Top-Down: Needs creating stubs (overhead)
Bottom-Up: Needs creating dirivers (overhead) Bottom-Up: Needs creating dirivers (overhead)
A Trade-off: Sandwich Testing A Trade-off: Sandwich Testing
Regression testing Regression testing
to ensure that changes do not introduce unintended to ensure that changes do not introduce unintended behavior or additional errors.
behavior or additional errors.
In context of an integration test: re-execution of some subset of In context of an integration test: re-execution of some subset of tests
tests
Any correction -> change of software Any correction -> change of software
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
18
Smoke Testing Smoke Testing
A common approach for creating “daily builds” for product A common approach for creating “daily builds” for product software
software
Smoke testing steps: Smoke testing steps:
Software components that have been translated into code are Software components that have been translated into code are integrated into a “build.”
integrated into a “build.”
A build includes all data files, libraries, reusable modules, and A build includes all data files, libraries, reusable modules, and engineered components that are required to implement one or engineered components that are required to implement one or more product functions.
more product functions.
A series of tests is designed to expose errors that will keep the A series of tests is designed to expose errors that will keep the build from properly performing its function.
build from properly performing its function.
The intent should be to uncover “show stopper” errors that have The intent should be to uncover “show stopper” errors that have the highest likelihood of throwing the software project behind the highest likelihood of throwing the software project behind schedule.
schedule.
The build is integrated with other builds and the entire product The build is integrated with other builds and the entire product (in its current form) is smoke tested daily.
(in its current form) is smoke tested daily.
The integration approach may be top down or bottom up. The integration approach may be top down or bottom up.
Identify critical modules Identify critical modules
Address several software requirements Address several software requirements
Has a high level of control (high in the program Has a high level of control (high in the program structure)
structure)
Is complex or error prone Is complex or error prone
Has definite performance requirements Has definite performance requirements
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
20
Object-Oriented Testing Object-Oriented Testing
testing strategy changes testing strategy changes
the concept of the ‘unit’ broadens due to encapsulation the concept of the ‘unit’ broadens due to encapsulation
integration focuses on classes and their execution integration focuses on classes and their execution
validation uses conventional black box methods validation uses conventional black box methods
test case design draws on conventional methods, but test case design draws on conventional methods, but also encompasses special features
also encompasses special features
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
22
OOT Strategy OOT Strategy
class testing is the equivalent of unit testing class testing is the equivalent of unit testing
operations within the class are tested operations within the class are tested
the state behavior of the class is examined the state behavior of the class is examined
integration applied three different strategies integration applied three different strategies
thread-based testing—integrates the set of classes thread-based testing—integrates the set of classes required to respond to one input or event
required to respond to one input or event
use-based testing—integrates the set of classes required use-based testing—integrates the set of classes required to respond to one use case
to respond to one use case
cluster testing—integrates the set of classes required to cluster testing—integrates the set of classes required to demonstrate one collaboration
demonstrate one collaboration
Validation Testing Validation Testing
Validation succeeds when software functions in a Validation succeeds when software functions in a manner that can be reasonably expected by the manner that can be reasonably expected by the
customer.
customer.
Reasonable expectation are defined in the Software Reasonable expectation are defined in the Software Requirement Specification
Requirement Specification
Functional Functional
Behavioral Behavioral
Performance Performance
Documentation Documentation
Usability …. Usability ….
Alpha/Beta testing Alpha/Beta testing
24
High Order Testing:
High Order Testing:
System testing System testing
System testing System testing
Focus is on system integration Focus is on system integration
Recovery testing Recovery testing
forces the software to fail in a variety of ways and verifies that forces the software to fail in a variety of ways and verifies that recovery is properly performed
recovery is properly performed
Automatic: re-initialization, check pointing, data recovery, restart Automatic: re-initialization, check pointing, data recovery, restart
By human intervention ; mean-time-to-repair By human intervention ; mean-time-to-repair
Security testing Security testing
verifies that protection mechanisms built into a system will, in verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration
fact, protect it from improper penetration
Stress testing Stress testing
executes a system in a manner that demands resources in executes a system in a manner that demands resources in abnormal quantity, frequency, or volume
abnormal quantity, frequency, or volume
Performance Testing Performance Testing
test the run-time performance of software within the context of test the run-time performance of software within the context of an integrated system
an integrated system
Debugging:
Debugging:
A Diagnostic Process
A Diagnostic Process
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
26
The Debugging Process The Debugging Process
test cases test cases
results results
Debugging Debugging
suspected suspected
causes causes identified identified
causes causes corrections
corrections regression regression
tests tests
new test new test
cases
cases
Debugging Effort Debugging Effort
time required time required to diagnose the to diagnose the symptom and symptom and determine the determine the cause
cause time required
time required
to correct the error to correct the error
and conduct and conduct
regression tests
regression tests
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
28
Symptoms & Causes Symptoms & Causes
symptom symptom
cause cause
symptom and cause may be symptom and cause may be geographically separated geographically separated
symptom may disappear when symptom may disappear when another problem is fixed
another problem is fixed cause may be due to a cause may be due to a
combination of non-errors combination of non-errors
cause may be due to a system cause may be due to a system or compiler error
or compiler error
cause may be due to cause may be due to
assumptions that everyone assumptions that everyone believes
believes
symptom may be intermittent
symptom may be intermittent
Debugging tactics Debugging tactics
Brute force Brute force
Dump the memory Dump the memory
Backtracking Backtracking
Start from presence of error, go backward in the code manually Start from presence of error, go backward in the code manually
Cause elimination Cause elimination
Cause hypothesis … Cause hypothesis …
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e
and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005
