Stan Kurkovsky
Software Engineering Software Engineering
Software Testing Software Testing
Based on Software Engineering, 7thEdition by Ian Sommerville
Objectives Objectives
•• To discuss the distinctions between validation testing and defecTo discuss the distinctions between validation testing and defect testingt testing
•
• To describe the principles of system and component testingTo describe the principles of system and component testing
•• To describe strategies for generating system test casesTo describe strategies for generating system test cases
•
• To understand the essential characteristics of tool used for tesTo understand the essential characteristics of tool used for test t automation
automation
Stan Kurkovsky
The testing process The testing process
•
• Component testing Component testing
•• Testing of individual program components;Testing of individual program components;
•
• Usually the responsibility of the component developer (except sometimes for Usually the responsibility of the component developer (except sometimes for critical systems);
critical systems);
•• Tests are derived from the developer’Tests are derived from the developer’s experience.s experience.
•• System testingSystem testing
•• Testing of groups of components integrated to create a system orTesting of groups of components integrated to create a system orsub-sub- system;
system;
•• The responsibility of an independent testing team;The responsibility of an independent testing team;
•
• Tests are based on a system specification.Tests are based on a system specification.
Component testing
System testing
Software developer Independent testing team
Testing process goals Testing process goals
•• Validation testingValidation testing
•
• To demonstrate to the developer and the system customer that theTo demonstrate to the developer and the system customer that thesoftware software meets its requirements;
meets its requirements;
•• A successful test shows that the system operates as intended.A successful test shows that the system operates as intended.
•• Defect testingDefect testing
•• To discover faults or defects in the software where its behaviorTo discover faults or defects in the software where its behavioris incorrect or is incorrect or not in conformance with its specification;
not in conformance with its specification;
•• A successful test is a test that makes the system perform incorrectly and so A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system.
exposes a defect in the system.
Design test Prepare test Run pr ogram Compare results
Test cases
Test data
Test results
Test reports
Stan Kurkovsky
Testing policies Testing policies
•
• Only exhaustive testing can show a program is free from defects.Only exhaustive testing can show a program is free from defects.
However, exhaustive testing is impossible, However, exhaustive testing is impossible,
•• Testing policies define the approach to be used in selecting sysTesting policies define the approach to be used in selecting system tests:tem tests:
•
• All functions accessed through menus should be tested;All functions accessed through menus should be tested;
•• Combinations of functions accessed through the same menu should be Combinations of functions accessed through the same menu should be tested;
tested;
•
• Where user input is required, all functions must be tested with correct and Where user input is required, all functions must be tested with correct and incorrect input.
incorrect input.
System testing System testing
•• Involves integrating components to create a system or subInvolves integrating components to create a system or sub--system.system.
•
• May involve testing an increment to be delivered to the customerMay involve testing an increment to be delivered to the customer..
•• Two phases:Two phases:
•
• Integration testingIntegration testing--the test team have access to the system source code. the test team have access to the system source code.
The system is tested as components are integrated.
The system is tested as components are integrated.
•• Release testingRelease testing--the test team test the complete system to be delivered as a the test team test the complete system to be delivered as a black
black--box.box.
Stan Kurkovsky
Integration testing Integration testing
•
• Involves building a system from its components and testing it foInvolves building a system from its components and testing it for r problems that arise from component interactions.
problems that arise from component interactions.
•• TopTop--down integrationdown integration
•
• Develop the skeleton of the system and populate it with components.Develop the skeleton of the system and populate it with components.
•• BottomBottom--up integrationup integration
•
• Integrate infrastructure components then add functional components.Integrate infrastructure components then add functional components.
•
• To simplify error localization, systems should be incrementally To simplify error localization, systems should be incrementally integrated.integrated.
T3 T2 T1
T4
T5 A
B
C
D T2 T1
T3
T4 A
B
C T1
T2
T3 A
B
Test sequence 1 Test sequence 2 Test sequence 3
Release testing Release testing
•• The process of testing a release of a system that will be distriThe process of testing a release of a system that will be distributed to buted to customers.
customers.
•
• Primary goal is to increase the supplierPrimary goal is to increase the supplier’’s confidence that the system s confidence that the system meets its requirements.
meets its requirements.
•• Release testing is usually blackRelease testing is usually black--box or functional testingbox or functional testing
•
• Based on the system specification only;Based on the system specification only;
•• Testers do not have knowledge of the system implementation.Testers do not have knowledge of the system implementation.
•• Black box testingBlack box testing
Ie Input test data
System
Inputs causing anomalous behavior
Stan Kurkovsky
Testing guidelines Testing guidelines
•
• Testing guidelines are hints for the testing team to help them cTesting guidelines are hints for the testing team to help them choose hoose tests that will reveal defects in the system
tests that will reveal defects in the system
•• Choose inputs that force the system to generate all error messages;Choose inputs that force the system to generate all error messages;
•
• Design inputs that cause buffers to overflow;Design inputs that cause buffers to overflow;
•• Repeat the same input or input series several times;Repeat the same input or input series several times;
•
• Force invalid outputs to be generated;Force invalid outputs to be generated;
•• Force computation results to be too large or too small.Force computation results to be too large or too small.
•• Use casesUse cases
•• Use cases can be a basis for deriving the tests for a system. They help Use cases can be a basis for deriving the tests for a system. They help identify operations to be tested and help design the required te
identify operations to be tested and help design the required test cases.st cases.
•• From an associated sequence diagram, the inputs and outputs to be created From an associated sequence diagram, the inputs and outputs to be created for the tests can be identified.
for the tests can be identified.
Performance testing Performance testing
•• Part of release testing may involve testing the emergent propertPart of release testing may involve testing the emergent properties of a ies of a system, such as performance and reliability.
system, such as performance and reliability.
•
• Performance tests usually involve planning a series of tests whePerformance tests usually involve planning a series of tests where the re the load is steadily increased until the system performance becomes load is steadily increased until the system performance becomes unacceptable.
unacceptable.
Stress testing Stress testing
•
• Exercises the system beyond its maximum design load. Stressing tExercises the system beyond its maximum design load. Stressing the he system often causes defects to come to light.
system often causes defects to come to light.
•• Stressing the system test failure behavior. Systems should not fStressing the system test failure behavior. Systems should not fail ail catastrophically. Stress testing checks for unacceptable loss of
catastrophically. Stress testing checks for unacceptable loss ofservice or service or data.
data.
•
• Stress testing is particularly relevant to distributed systems tStress testing is particularly relevant to distributed systems that can hat can exhibit severe degradation as a network becomes overloaded.
exhibit severe degradation as a network becomes overloaded.
Stan Kurkovsky
Component testing Component testing
•
• Component or unit testing is the process of testing individual cComponent or unit testing is the process of testing individual components omponents in isolation.
in isolation.
•• It is a defect testing process.It is a defect testing process.
•
• Components may be:Components may be:
•• Individual functions or methods within an object;Individual functions or methods within an object;
•
• Object classes with several attributes and methods;Object classes with several attributes and methods;
•• Composite components with defined interfaces used to access their Composite components with defined interfaces used to access their functionality.
functionality.
Object class testing Object class testing
•• Complete test coverage of a class involvesComplete test coverage of a class involves
•
• Testing all operations associated with an object;Testing all operations associated with an object;
•• Setting and interrogating all object attributes;Setting and interrogating all object attributes;
•
• Exercising the object in all possible states.Exercising the object in all possible states.
•• Inheritance makes it more difficult to design object class testsInheritance makes it more difficult to design object class testsas the as the information to be tested is not localized.
information to be tested is not localized.
Interface testing Interface testing
Interface types Interface types
•
• Parameter interfacesParameter interfaces
•• Data passed from one procedure Data passed from one procedure to another.
to another.
•• Shared memory interfacesShared memory interfaces
•• Block of memory is shared Block of memory is shared between procedures or functions.
between procedures or functions.
•• Procedural interfacesProcedural interfaces
•
• Sub-Sub-system encapsulates a set of system encapsulates a set of procedures to be called by other procedures to be called by other sub-sub-systems.systems.
Interface errors Interface errors
•
• Interface misuseInterface misuse
•
• A calling component calls another A calling component calls another component and makes an error in component and makes an error in its use of its interface e.g.
its use of its interface e.g.
parameters in the wrong order.
parameters in the wrong order.
•• Interface misunderstandingInterface misunderstanding
•• A calling component embeds A calling component embeds assumptions about the behavior of assumptions about the behavior of the called component which are the called component which are incorrect.
incorrect.
•• Timing errorsTiming errors
•• Objectives are to detect faults due to interface errors or invalObjectives are to detect faults due to interface errors or invalid id assumptions about interfaces.
assumptions about interfaces.
•
• Particularly important for objectParticularly important for object--oriented development as objects are oriented development as objects are defined by their interfaces.
defined by their interfaces.
Stan Kurkovsky
Test case design Test case design
•
• Involves designing the test cases (inputs and outputs) used to tInvolves designing the test cases (inputs and outputs) used to test the est the system.
system.
•• The goal of test case design is to create a set of tests that arThe goal of test case design is to create a set of tests that are effective in e effective in validation and defect testing.
validation and defect testing.
•
• Design approaches:Design approaches:
•• Requirements-Requirements-based testing;based testing;
•
• Partition testing;Partition testing;
•• Structural testing.Structural testing.
Requirements based testing Requirements based testing
•• A general principle of requirements engineering is that requiremA general principle of requirements engineering is that requirements ents should be testable.
should be testable.
•
• RequirementsRequirements--based testing is a validation testing technique where you based testing is a validation testing technique where you consider each requirement and derive a set of tests for that req
consider each requirement and derive a set of tests for that requirement.uirement.
Stan Kurkovsky
Partition testing Partition testing
•
• Input data and output results often fall into different classes Input data and output results often fall into different classes where all where all members of a class are related.
members of a class are related.
•• Each of these classes is an Each of these classes is an equivalence partitionequivalence partitionor domain where the or domain where the program behaves in an equivalent way for each class member.
program behaves in an equivalent way for each class member.
•
• Test cases should be chosen from each partition.Test cases should be chosen from each partition.
Structural testing Structural testing
•• Sometime called whiteSometime called white--box testing.box testing.
•
• Derivation of test cases according to program structure. KnowledDerivation of test cases according to program structure. Knowledge of the ge of the program is used to identify additional test cases.
program is used to identify additional test cases.
•• Objective is to exercise all program statements (not all path Objective is to exercise all program statements (not all path combinations).
combinations).
Stan Kurkovsky
Path testing Path testing
•
• The objective of path testing is to ensure that the set of test The objective of path testing is to ensure that the set of test cases is cases is such that each path through the program is executed at least onc such that each path through the program is executed at least once.e.
•• The starting point for path testing is a program flow graph thatThe starting point for path testing is a program flow graph thatshows shows nodes representing program decisions and arcs representing the f nodes representing program decisions and arcs representing the flow of low of control.
control.
•
• Statements with conditions are therefore nodes in the flow graphStatements with conditions are therefore nodes in the flow graph..
Test automation Test automation
•• Testing is an expensive process phase. Testing workbenches proviTesting is an expensive process phase. Testing workbenches provide a de a range of tools to reduce the time required and total testing cos
range of tools to reduce the time required and total testing costs.ts.
•
• Systems such as JUnit support the automatic execution of tests.Systems such as JUnit support the automatic execution of tests.
•• Most testing workbenches are open systems because testing needs Most testing workbenches are open systems because testing needs are are organization
organization--specific.specific.
•
• They are sometimes difficult to integrate with closed design andThey are sometimes difficult to integrate with closed design andanalysis analysis workbenches.
workbenches.
Stan Kurkovsky
Key points Key points
•
• Testing can show the presence of faults in a system; it cannot pTesting can show the presence of faults in a system; it cannot prove rove there are no remaining faults.
there are no remaining faults.
•• Component developers are responsible for component testing; systComponent developers are responsible for component testing; system em testing is the responsibility of a separate team.
testing is the responsibility of a separate team.
•
• Integration testing is testing increments of the system; releaseIntegration testing is testing increments of the system; releasetesting testing involves testing a system to be released to a customer.
involves testing a system to be released to a customer.
•
• Use experience and guidelines to design test cases in defect tesUse experience and guidelines to design test cases in defect testing.ting.
•• Interface testing is designed to discover defects in the interfaInterface testing is designed to discover defects in the interfaces of ces of composite components.
composite components.
•
• Equivalence partitioning is a way of discovering test cases Equivalence partitioning is a way of discovering test cases --all cases in a all cases in a partition should behave in the same way.
partition should behave in the same way.
•• Structural analysis relies on analyzing a program and deriving tStructural analysis relies on analyzing a program and deriving tests from ests from this analysis.
this analysis.
•
• Test automation reduces testing costs by supporting the test proTest automation reduces testing costs by supporting the test process with cess with a range of software tools.
a range of software tools.