Di 3.1
January 26-30, 2009, Munich, Germany ICM - International Congress Centre Munich
A Test Design Poster for Testers, Developers, and Architects
Peter Zimmerer
Copyright © Siemens AG 2009. All rights reserved.
Corporate Technology
OOP 2009
Munich, Germany
Peter Zimmerer
Principal Engineer Siemens AG, CT SE 1
Corporate Technology
Corporate Research and Technologies Software & Engineering, Development Techniques
81739 Munich, Germany [email protected]
http://www.siemens.com/research-and-development/
http://www.ct.siemens.com/
A Test Design Poster for Testers, Developers, and Architects
Contents
Introduction
Test design methods
Here: methods, paradigms, techniques, styles, and ideas to create, derive, select, generate a test case
Motivation – who cares about test design methods?
Examples and references
Problem statement
Poster Test Design Methods on One Page Guidelines and experiences
Backup – Details on test design methods
Page 3 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Introduction – Test design methods
Good test design, i.e. high-quality test cases, is very important
There are many different test design methods and techniques
Static, dynamic
Black-box, white-box, grey-box
Based on fault model, experience, exploratory
Statistical (user profiles), random (monkey) The challenge is to adequately combine these methods dependent on the given problem, domain, and requirements
This is art as well!
Black-box test design methods are often based on models – model-based testing
Some systematic methods for test design
Black-box (models, interfaces, data)
Requirements-based (traceability matrix)
Use case-based testing, scenario testing
Design by contract
Equivalence class partitioning
Classification-tree method
Boundary value analysis
State-based testing
Cause-effect graphing
Decision tables, decision trees
Combinatorial testing (n-wise)
White-box (internal structure, paths)
Control flow testing
Data flow testing
Selection, usage, and
applicability depends on the
specific domain (domain knowledge is required!)
used software technology
test requirements: required test intensity, quality criteria, risks
existing test basis:
specifications, documents, models
project factors: constraints
and opportunities
Page 5 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Who cares / should care about test design methods?
Requirements engineers Æ Requirements testing
Architects Æ Architecture testing
Developers Æ Unit testing
Test engineers Æ All the rest: system testing, …
Important: collaboration between the different stakeholders
Test levels – example V model with architecture testing
System Requirements
Architecture, Design
Unit Specification
Acceptance Testing System Testing Integration
Testing Unit
Testing User
Requirements
Code Quality Management Architecture
interrogation:
interviews, interactive workshops
Evaluation, prototyping, simulation
Architecture testing
– any testing of architecture and architectural artifacts Test case design
Analysis, reviews, previews, inspections
Load model specification
based on
Coding
Static testing
Page 7 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Test design methods ÅÆ Preventive Testing ≈ TDD
Preventive testing is built upon the observation that
one of the most effective ways of specifying something is to describe (in detail) how you would accept (test) it
if someone gave it to you.
David Gelperin, Bill Hetzel (<1990) Given any kind of specification for a product,
the first thing to develop isn't the product, but how you'd test the product.
Don’t start to build a product till you know how to test it.
Tom Gilb The act of designing tests is one of the
most effective bug preventers known.
Boris Beizer, 1983
Test design methods ÅÆ Test basis
Selection, usage, and applicability of test design methods depends on the existing test basis: specifications, documents, models
Therefore, any person who is involved in any specification activity should know about test design methods, at least a little …
Example Architect:
Specify your architecture (especially the dynamic behavior) by using the right models, formats, and notations
to provide an adequate test basis
to enable more effective and more efficient testing
BTW, how can you do this adequately without knowing anything
about test design methods???
Page 9 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Test design methods ÅÆ Risks
Test design methods address risks
ÆSee especially fault-based test design methods
Test design methods are a rich source describing what might go wrong in the product or system
Test design methods are very helpful
to identify and analyze risks w.r.t. architecture, software technologies, etc.
to built-in better design for testability (DFT)
Test design methods are an integral part of the testing strategy
BTW, how can you do this adequately without knowing anything about test design methods???
What is a Test Case?
A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. ISTQB 2007, IEEE 610
A Test Case should include
unique identification – who am I?
test goal, test purpose – why? ÅÆ link to requirements
test conditions/requirements – what? ÅÆ coverage!!!
preconditions – system state, environmental conditions
test data – inputs, data, actions
execution conditions – constraints, dependencies
expected results – oracles, arbiters, verdicts, coverage, traces
postconditions – system state, expected side effects, expected invariants, traces,
environmental conditions
Page 11 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Test design methods and test cases
Test design methods mainly help us to identify the
test data – inputs, data, actions i.e. especially the input values for the test cases
and hopefully provide us some information about the
expected results – oracles, arbiters, verdicts, coverage, traces at least to some extent
Typically test design methods alone cannot supply
preconditions – system state, environmental conditions
execution conditions – constraints, dependencies
expected results – oracles, arbiters, verdicts, coverage, traces
postconditions – system state, expected side effects, expected invariants, traces,
environmental conditions
Generally these items must be determined from the test basis
depending on the context: project, domain, requirements, constraints
Example – There are always too many test cases ...
Page 13 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Examples – Demo
Microsoft PowerPoint
Microsoft Word 2002
Test effectiveness and formal (systematic) test design
There are studies showing advantages of systematic test design.
There are also studies showing advantages of random testing.
Æ But do you really want to design your test cases only randomly?
Formal test design was almost twice as effective in defect detection per test case as compared to expert (exploratory) type testing, and much more effective compared to checklist type testing.
Bob Bartlett, SQS UK, 2006
Page 15 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Many testing books cover test design to some extent
Boris Beizer: Software Testing Techniques
Robert V. Binder: Testing Object-Oriented Systems: Models, Patterns, and Tools
Lee Copeland: A Practitioner's Guide to Software Test Design
Rick D. Craig, Stefan P. Jaskiel: Systematic Software Testing
Tim Koomen et. al.: TMap Next: For Result-driven Testing
Glenford J. Meyers: The Art of Software Testing
Torbjörn Ryber: Essential Software Test Design
James Whittaker: How to Break Software
James Whittaker, Herbert Thompson: How to Break Software Security
Standard for Software Component Testing by the British Computer Society Specialist Interest Group in Software Testing (BCS SIGIST) (see http://www.testingstandards.co.uk/)
There are many different training offerings by different providers
Test design tools are typically focused and implement only a few test design methods – some examples
ATD – Automated Test Designer (AtYourSide Consulting, http://www.atyoursideconsulting.com/): cause-effect graphing
BenderRBT (Richard Bender, http://www.benderrbt.com/):
cause-effect graphing, quick design (orthogonal pairs)
CaseMaker (Diaz & Hilterscheid, http://www.casemaker.de/):
business rules, equivalence classes, boundaries, error guessing, pairwise combinations, and element dependencies
CTE (Razorcat, http://www.ats-software.de/): classification tree editor
MaTeLo (All4tec, http://www.all4tec.net/): markov chains
Qtronic (Conformiq, http://www.conformiq.com/): state-based testing
Reactis (Reactive Systems, http://www.reactive-systems.com/):
generation of test data from, and validation of, Simulink and Stateflow models
Test Designer (Smartesting, http://www.smartesting.com/): state-based testing
TestBench (Imbus, http://www.testbench.info/, http://www.imbus.de/):
equivalence classes, work-flow / use-case-based testing
Page 17 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Problem statement
Starting from a risk-based testing strategy an adequate test design is the key for effective and efficient testing.
Automation of bad test cases is a waste of time and money!
There are many different test design methods around for a long time (perhaps too many?) and a lot of books explain them in detail.
There are different
categorizations, classifications, and dimensions
naming, interpretations, and understandings
of test design methods which does not simplify their usage … When we look into practice we can see that often there is quite limited usage of these test design methods at all.
What are the reasons behind that?
How can we overcome this and improve our testing approaches?
Poster Test Design Methods on One Page (1)
Idea:
Systematic, structured, and categorized overview about different test design methods on one page
Focus more on using an adequate set of test design methods than on using only one single test design method in depth / perfection
Focus more on concrete usage of test design methods than on defining a few perfect test design methods in detail which are not used then in the project
Focus more on breadth instead on depth
Do not miss breadth because of too much depth
Do not miss the exploratory, investigative art of testing
Page 19 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Poster Test Design Methods on One Page (2)
Effort / Difficulty / Resulting Test Intensity (5 Levels) Key
Methods, Paradigm s, Techniques, Styles, and Ideas to Create a Test Case Categorization
Black-box 3
(models, interfaces, data) 3
3 3 4 4 4 4 1 3 2 4 3 2 1 5 3 5 5 4 3 4 5
Grey-box 2
3 3 3 4
White-box Control flow-based Coverage Statements (C0), nodes 2
(internal structure, paths) (specification-based, Branches (C1), transitions, links, paths 3
model-based, Conditions, decisions (C2, C3) 4
code-based) Elementary comparison (MC/DC) 5
Interfaces (S1, S2) 4
Static metrics Cyclomatic complexity (McCabe) 4
Metrics (e.g. Halstead) 4
Read / Write access 3
Def / Use criteria 5
Positive, valid cases 1
Negative, invalid cases 3
3 5
Fault-based 2
4 3 4 3 2 2 3 1 2 2 4 5
Regression (selective retesting) 5
2 3 2 5 Communication behavior (dependency analysis)
Fault injection
Test catalog / matrix for input values, input fields State-based testing (Final State Machines) Cause-effect graphing
Normal, expected behavior Error handling
T e s t D e s i g n M e t h o d s o n O n e P a g e
Ad hoc, intuitive, based on experience, check lists Bug reports
Bug patterns: standard, well-known bug patterns or produced by a root cause analysis Systematic failure analysis (Failure Mode and Effect Analysis, Fault Tree Analysis) Dependencies / Relations between classes, objects, methods, functions Dependencies / Relations between components, services, applications, systems Special values
Standards (e.g. ISO/IEC 9126, IEC 61508), norms, (formal) specifications, claims
Retest by risk, priority, severity, criticality Retest all
User / Operational profiles: frequency and priority / criticality (Software Reliability Engineering)
Protocol based (sequence diagrams, message sequence charts)
Test patterns (e.g. by Robert Binder), Questioning patterns (Q-patterns by Vipul Kocher) Combinatorial testing (pair-wise, orthogonal / covering arrays, n-wise)
Evolutionary testing Equivalence class partitioning Features, functions, interfaces
Attack patterns (e.g. by James A. Whittaker)
Retest changed parts Error guessing
Error catalogs, bug taxonomies (e.g. by Boris Beizer, Cem Kaner)
Retest by profile, frequency of usage, parts which are often used Statistical testing (markov chains)
CRUD (Create, Read, Update, Delete) (data cycles, database operations) Requirements-based with traceability matrix (requirements x test cases) Use case-based testing (sequence diagrams, activity diagrams) Flow testing, scenario testing, soap opera testing
Random (monkey testing) Design by contract (built-in self test)
Data flow-based Decision tables, decision trees Classification-tree method
Syntax testing (grammar-based testing) Time cycles (frequency, recurring events, test dates) Domain partitioning, category-partition method Boundary value analysis
Trace-based testing (passive testing)
Invalid, unexpected behavior Exceptions Risk-based
Fault model dependent on used technology and nature of system under test
Exploratory testing, heuristics, mnemonics (e.g. by James Bach, Michael Bolton) Mutation testing
Retest parts that are influenced by the changes (impact analysis, dependency analysis)
Poster Test Design Methods on One Page (3)
Categories of test design methods are orthogonal and independent in some way but should be combined appropriately.
The selection of the used test design methods depends on many factors, for example:
Requirements of the system under test and the required quality
Requirements for the tests – quality of the tests, i.e. the required intensity and depth of the tests
Testing strategy: effort / quality of the tests, distribution of the testing in the development process
Existing test basis: specifications, documents, models
Problem to be tested (domain) or rather the underlying question (use case)
System under test or component under test
Test level / Test step
Used technologies (software, hardware)
Suitable tool support: for some methods absolutely required
Black-box 3
(models, interfaces, data) 3
3 3 4 4 4 4 1 3 2 4 3 2 1 5 3 5 5 4 3 4 5
Grey-box 2
3 3 3 4
White-box Control flow-basedCoverage Statements (C0), nodes 2
(internal structure, paths) (specification-based, Branches (C1), transitions, links, paths 3
model-based, Conditions, decisions (C2, C3) 4
code-based) Elementary comparison (MC/DC) 5
Interfaces (S1, S2) 4
Static metrics Cyclomatic complexity (McCabe) 4
Metrics (e.g. Halstead) 4
Read / Write access 3
Def / Use criteria 5
Positive, valid cases 1
Negative, invalid cases 3
3 5
Fault-based 2
4 3 4 3 2 2 3 1 2 2 4 5
Regression (selective retesting) 5
2 3 2 5 Communication behavior (dependency analysis)
Fault injection Test catalog / matrix for input values, input fields State-based testing (Final State Machines) Cause-effect graphing
Normal, expected behavior Error handling
T e s t D e s i g n M e t h o d s o n O n e P a g e
Ad hoc, intuitive, based on experience, check lists Bug reports Bug patterns: standard, well-known bug patterns or produced by a root cause analysis Systematic failure analysis (Failure Mode and Effect Analysis, Fault Tree Analysis) Dependencies / Relations between classes, objects, methods, functions Dependencies / Relations between components, services, applications, systems Special values Standards (e.g. ISO/IEC 9126, IEC 61508), norms, (formal) specifications, claims
Retest by risk, priority, severity, criticality Retest all
User / Operational profiles: frequency and priority / criticality (Software Reliability Engineering)
Protocol based (sequence diagrams, message sequence charts)
Test patterns (e.g. by Robert Binder), Questioning patterns (Q-patterns by Vipul Kocher) Combinatorial testing (pair-wise, orthogonal / covering arrays, n-wise) Evolutionary testing Equivalence class partitioning Features, functions, interfaces
Attack patterns (e.g. by James A. Whittaker)
Retest changed parts Error guessing Error catalogs, bug taxonomies (e.g. by Boris Beizer, Cem Kaner)
Retest by profile, frequency of usage, parts which are often used Statistical testing (markov chains) CRUD (Create, Read, Update, Delete) (data cycles, database operations) Requirements-based with traceability matrix (requirements x test cases) Use case-based testing (sequence diagrams, activity diagrams) Flow testing, scenario testing, soap opera testing
Random (monkey testing) Design by contract (built-in self test)
Data flow-based Decision tables, decision trees Classification-tree method
Syntax testing (grammar-based testing) Time cycles (frequency, recurring events, test dates) Domain partitioning, category-partition method Boundary value analysis
Trace-based testing (passive testing)
Invalid, unexpected behavior Exceptions Risk-based
Fault model dependent on used technology and nature of system under test
Exploratory testing, heuristics, mnemonics (e.g. by James Bach, Michael Bolton) Mutation testing
Retest parts that are influenced by the changes (impact analysis, dependency analysis)
Page 21 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Poster Test Design Methods on One Page (4)
The effort / difficulty for the test design methods or rather the resulting test intensity is subdivided into 5 levels:
1 very low, simple
2 low
3 medium
4 high
5 very high, complex
This division into levels is dependent on the factors given above for the selection of the test design methods and therefore can only be used as a first hint and guideline.
A test design method also may be used continuously from
“intuitive use” up to “100% complete use” as required.
In addition describe every test design method on one page to explain their basic message and intention.
Black-box 3
(models, interfaces, data) 3
3 3 4 4 4 4 1 3 2 4 3 2 1 5 3 5 5 4 3 4 5
Grey-box 2
3 3 3 4
White-box Control flow-basedCoverage Statements (C0), nodes 2
(internal structure, paths) (specification-based, Branches (C1), transitions, links, paths 3
model-based, Conditions, decisions (C2, C3) 4
code-based) Elementary comparison (MC/DC) 5
Interfaces (S1, S2) 4
Static metrics Cyclomatic complexity (McCabe) 4
Metrics (e.g. Halstead) 4
Read / Write access 3
Def / Use criteria 5
Positive, valid cases 1
Negative, invalid cases 3
3 5
Fault-based 2
4 3 4 3 2 2 3 1 2 2 4 5
Regression (selective retesting) 5
2 3 2 5 Communication behavior (dependency analysis)
Fault injection Test catalog / matrix for input values, input fields State-based testing (Final State Machines) Cause-effect graphing
Normal, expected behavior Error handling
T e s t D e s i g n M e t h o d s o n O n e P a g e
Ad hoc, intuitive, based on experience, check lists Bug reports Bug patterns: standard, well-known bug patterns or produced by a root cause analysis Systematic failure analysis (Failure Mode and Effect Analysis, Fault Tree Analysis) Dependencies / Relations between classes, objects, methods, functions Dependencies / Relations between components, services, applications, systems Special values Standards (e.g. ISO/IEC 9126, IEC 61508), norms, (formal) specifications, claims
Retest by risk, priority, severity, criticality Retest all
User / Operational profiles: frequency and priority / criticality (Software Reliability Engineering)
Protocol based (sequence diagrams, message sequence charts)
Test patterns (e.g. by Robert Binder), Questioning patterns (Q-patterns by Vipul Kocher) Combinatorial testing (pair-wise, orthogonal / covering arrays, n-wise) Evolutionary testing Equivalence class partitioning Features, functions, interfaces
Attack patterns (e.g. by James A. Whittaker)
Retest changed parts Error guessing Error catalogs, bug taxonomies (e.g. by Boris Beizer, Cem Kaner)
Retest by profile, frequency of usage, parts which are often used Statistical testing (markov chains) CRUD (Create, Read, Update, Delete) (data cycles, database operations) Requirements-based with traceability matrix (requirements x test cases) Use case-based testing (sequence diagrams, activity diagrams) Flow testing, scenario testing, soap opera testing
Random (monkey testing) Design by contract (built-in self test)
Data flow-based Decision tables, decision trees Classification-tree method
Syntax testing (grammar-based testing) Time cycles (frequency, recurring events, test dates) Domain partitioning, category-partition method Boundary value analysis
Trace-based testing (passive testing)
Invalid, unexpected behavior Exceptions Risk-based
Fault model dependent on used technology and nature of system under test
Exploratory testing, heuristics, mnemonics (e.g. by James Bach, Michael Bolton) Mutation testing
Retest parts that are influenced by the changes (impact analysis, dependency analysis)
Guidelines and experiences (1)
For beginners
perhaps you are confused about the many test design methods
start simple, step by step
ask for help and advice by an experienced colleague, coach or consultant For advanced, experienced testers (and developers!)
check your current approach against this poster, think twice, and improve incrementally
Use the poster as a checklist for existing test design methods
Selection of test design methods is dependent on the context!
So, you should adapt the poster to your specific needs.
Black-box 3
(models, interfaces, data) 3
3 3 4 4 4 4 1 3 2 4 3 2 1 5 3 5 5 4 3 4 5
Grey-box 2
3 3 3 4
White-box Control flow-basedCoverage Statements (C0), nodes 2
(internal structure, paths) (specification-based, Branches (C1), transitions, links, paths 3
model-based, Conditions, decisions (C2, C3) 4
code-based) Elementary comparison (MC/DC) 5
Interfaces (S1, S2) 4
Static metrics Cyclomatic complexity (McCabe) 4
Metrics (e.g. Halstead) 4
Read / Write access 3
Def / Use criteria 5
Positive, valid cases 1
Negative, invalid cases 3
3 5
Fault-based 2
4 3 4 3 2 2 3 1 2 2 4 5
Regression (selective retesting) 5
2 3 2 5 Communication behavior (dependency analysis)
Fault injection Test catalog / matrix for input values, input fields State-based testing (Final State Machines) Cause-effect graphing
Normal, expected behavior Error handling
T e s t D e s i g n M e t h o d s o n O n e P a g e
Ad hoc, intuitive, based on experience, check lists Bug reports Bug patterns: standard, well-known bug patterns or produced by a root cause analysis Systematic failure analysis (Failure Mode and Effect Analysis, Fault Tree Analysis) Dependencies / Relations between classes, objects, methods, functions Dependencies / Relations between components, services, applications, systems Special values Standards (e.g. ISO/IEC 9126, IEC 61508), norms, (formal) specifications, claims
Retest by risk, priority, severity, criticality Retest all
User / Operational profiles: frequency and priority / criticality (Software Reliability Engineering)
Protocol based (sequence diagrams, message sequence charts)
Test patterns (e.g. by Robert Binder), Questioning patterns (Q-patterns by Vipul Kocher) Combinatorial testing (pair-wise, orthogonal / covering arrays, n-wise) Evolutionary testing Equivalence class partitioning Features, functions, interfaces
Attack patterns (e.g. by James A. Whittaker)
Retest changed parts Error guessing Error catalogs, bug taxonomies (e.g. by Boris Beizer, Cem Kaner)
Retest by profile, frequency of usage, parts which are often used Statistical testing (markov chains) CRUD (Create, Read, Update, Delete) (data cycles, database operations) Requirements-based with traceability matrix (requirements x test cases) Use case-based testing (sequence diagrams, activity diagrams) Flow testing, scenario testing, soap opera testing
Random (monkey testing) Design by contract (built-in self test)
Data flow-based Decision tables, decision trees Classification-tree method
Syntax testing (grammar-based testing) Time cycles (frequency, recurring events, test dates) Domain partitioning, category-partition method Boundary value analysis
Trace-based testing (passive testing)
Invalid, unexpected behavior Exceptions Risk-based
Fault model dependent on used technology and nature of system under test
Exploratory testing, heuristics, mnemonics (e.g. by James Bach, Michael Bolton) Mutation testing
Retest parts that are influenced by the changes (impact analysis, dependency analysis)
Page 23 January 27, 2009 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology
Guidelines and experiences (2)
Pick up this poster and
give it to every developer and tester in your team or
put it on the wall in your office or
make it the standard screensaver or
desktop background for all team members or
even use the testing on the toilet approach by Google (see http://googletesting.blogspot.com/) …
The poster increases visibility and importance of test design methods, especially also for developers to improve unit testing
The poster facilitates a closer collaboration of testers and developers: you have something to talk about ...
Black-box 3
(models, interfaces, data) 3
3 3 4 4 4 4 1 3 2 4 3 2 1 5 3 5 5 4 3 4 5
Grey-box 2
3 3 3 4
White-box Control flow-basedCoverage Statements (C0), nodes 2
(internal structure, paths) (specification-based, Branches (C1), transitions, links, paths 3
model-based, Conditions, decisions (C2, C3) 4
code-based) Elementary comparison (MC/DC) 5
Interfaces (S1, S2) 4
Static metrics Cyclomatic complexity (McCabe) 4
Metrics (e.g. Halstead) 4
Read / Write access 3
Def / Use criteria 5
Positive, valid cases 1
Negative, invalid cases 3
3 5
Fault-based 2
4 3 4 3 2 2 3 1 2 2 4 5
Regression (selective retesting) 5
2 3 2 5 Communication behavior (dependency analysis)
Fault injection Test catalog / matrix for input values, input fields State-based testing (Final State Machines) Cause-effect graphing
Normal, expected behavior Error handling
T e s t D e s i g n M e t h o d s o n O n e P a g e
Ad hoc, intuitive, based on experience, check lists Bug reports Bug patterns: standard, well-known bug patterns or produced by a root cause analysis Systematic failure analysis (Failure Mode and Effect Analysis, Fault Tree Analysis) Dependencies / Relations between classes, objects, methods, functions Dependencies / Relations between components, services, applications, systems Special values Standards (e.g. ISO/IEC 9126, IEC 61508), norms, (formal) specifications, claims
Retest by risk, priority, severity, criticality Retest all
User / Operational profiles: frequency and priority / criticality (Software Reliability Engineering)
Protocol based (sequence diagrams, message sequence charts)
Test patterns (e.g. by Robert Binder), Questioning patterns (Q-patterns by Vipul Kocher) Combinatorial testing (pair-wise, orthogonal / covering arrays, n-wise) Evolutionary testing Equivalence class partitioning Features, functions, interfaces
Attack patterns (e.g. by James A. Whittaker)
Retest changed parts Error guessing Error catalogs, bug taxonomies (e.g. by Boris Beizer, Cem Kaner)
Retest by profile, frequency of usage, parts which are often used Statistical testing (markov chains) CRUD (Create, Read, Update, Delete) (data cycles, database operations) Requirements-based with traceability matrix (requirements x test cases) Use case-based testing (sequence diagrams, activity diagrams) Flow testing, scenario testing, soap opera testing
Random (monkey testing) Design by contract (built-in self test)
Data flow-based Decision tables, decision trees Classification-tree method
Syntax testing (grammar-based testing) Time cycles (frequency, recurring events, test dates) Domain partitioning, category-partition method Boundary value analysis
Trace-based testing (passive testing)
Invalid, unexpected behavior Exceptions Risk-based
Fault model dependent on used technology and nature of system under test
Exploratory testing, heuristics, mnemonics (e.g. by James Bach, Michael Bolton) Mutation testing
Retest parts that are influenced by the changes (impact analysis, dependency analysis)
Summary
There exist many different methods for adequate test design.
When looking into practice often these test design methods are used only sporadically and in a non-systematical way.
The poster Test Design Methods on One Page
containing a systematic, structured, and categorized overview about test design methods will help you to really get them used in practice in your projects
Do not miss breadth because of too much depth
This will result in better and smarter testing for testers, developers, and architects
Black-box 3
(models, interfaces, data) 3
3 3 4 4 4 4 1 3 2 4 3 2 1 5 3 5 5 4 3 4 5
Grey-box 2
3 3 3 4
White-box Control flow-basedCoverage Statements (C0), nodes 2
(internal structure, paths) (specification-based, Branches (C1), transitions, links, paths 3
model-based, Conditions, decisions (C2, C3) 4
code-based) Elementary comparison (MC/DC) 5
Interfaces (S1, S2) 4
Static metrics Cyclomatic complexity (McCabe) 4
Metrics (e.g. Halstead) 4
Read / Write access 3
Def / Use criteria 5
Positive, valid cases 1
Negative, invalid cases 3
3 5
Fault-based 2
4 3 4 3 2 2 3 1 2 2 4 5
Regression (selective retesting) 5
2 3 2 5 Communication behavior (dependency analysis)
Fault injection Test catalog / matrix for input values, input fields State-based testing (Final State Machines) Cause-effect graphing
Normal, expected behavior Error handling
T e s t D e s i g n M e t h o d s o n O n e P a g e
Ad hoc, intuitive, based on experience, check lists Bug reports Bug patterns: standard, well-known bug patterns or produced by a root cause analysis Systematic failure analysis (Failure Mode and Effect Analysis, Fault Tree Analysis) Dependencies / Relations between classes, objects, methods, functions Dependencies / Relations between components, services, applications, systems Special values Standards (e.g. ISO/IEC 9126, IEC 61508), norms, (formal) specifications, claims
Retest by risk, priority, severity, criticality Retest all
User / Operational profiles: frequency and priority / criticality (Software Reliability Engineering)
Protocol based (sequence diagrams, message sequence charts)
Test patterns (e.g. by Robert Binder), Questioning patterns (Q-patterns by Vipul Kocher) Combinatorial testing (pair-wise, orthogonal / covering arrays, n-wise) Evolutionary testing Equivalence class partitioning Features, functions, interfaces
Attack patterns (e.g. by James A. Whittaker)
Retest changed parts Error guessing Error catalogs, bug taxonomies (e.g. by Boris Beizer, Cem Kaner)
Retest by profile, frequency of usage, parts which are often used Statistical testing (markov chains) CRUD (Create, Read, Update, Delete) (data cycles, database operations) Requirements-based with traceability matrix (requirements x test cases) Use case-based testing (sequence diagrams, activity diagrams) Flow testing, scenario testing, soap opera testing
Random (monkey testing) Design by contract (built-in self test)
Data flow-based Decision tables, decision trees Classification-tree method
Syntax testing (grammar-based testing) Time cycles (frequency, recurring events, test dates) Domain partitioning, category-partition method Boundary value analysis
Trace-based testing (passive testing)
Invalid, unexpected behavior Exceptions Risk-based
Fault model dependent on used technology and nature of system under test
Exploratory testing, heuristics, mnemonics (e.g. by James Bach, Michael Bolton) Mutation testing
Retest parts that are influenced by the changes (impact analysis, dependency analysis)