Learning Objective
After completing this chapter, you will be able to:
List different levels of testing
Unit Testing
Unit testing: Isn't that some annoying requirement that we're going to ignore? Many developers get very nervous when you mention unit tests. Usually this is a vision of a grand table with every single method listed, along with the expected results and pass/fail date. It's important, but not relevant in most programming projects.
The unit test will motivate the code that you write. In a sense, it is a little design document that says, "What will this bit of code do?" Or, in the language of object oriented programming, "What will these clusters of objects do?"
The crucial issue in constructing a unit test is scope. If the scope is too narrow, then the tests will be trivial and the objects might pass the tests, but there will be no design of their interactions.
Certainly, interactions of objects are the crux of any object oriented design. Likewise, if the scope is too broad, then there is a high chance that not every component of the new code will get tested.
The programmer is then reduced to testing-by-poking-around, which is not an effective test strategy.
Need for Unit Test
How do you know that a method doesn't need a unit test? First, can it be tested by inspection? If the code is simple enough that the developer can just look at it and verify its correctness then it is simple enough to not require a unit test. The developer should know when this is the case. Unit tests will most likely be defined at the method level, so the art is to define the unit test on the methods that cannot be checked by inspection. Usually this is the case when the method involves a cluster of objects. Unit tests that isolate clusters of objects for testing are doubly useful, because they test for failures, and they also identify those segments of code that are related.
People who revisit the code will use the unit tests to discover which objects are related, or which objects form a cluster. Hence: Unit tests isolate clusters of objects for future developers. Another good litmus test is to look at the code and see if it throws an error or catches an error. If error handling is performed in a method, then that method can break. Generally, any method that can break is a good candidate for having a unit test, because it may break at some time, and then the unit test will be there to help you fix it.
The danger of not implementing a unit test on every method is that the coverage may be
incomplete. Just because we don't test every method explicitly doesn't mean that methods can get away with not being tested. The programmer should know that their unit testing is complete when the unit tests cover at the very least the functional requirements of all the code. The careful
programmer will know that their unit testing is complete when they have verified that their unit tests cover every cluster of objects that form their application.
Life Cycle Approach to Testing
Testing will occur throughout the project lifecycle i.e., from Requirements till User Acceptance Testing.
The main Objective to Unit Testing are as follows :
To execute a program with the intent of finding an error.;
To uncover an as-yet undiscovered error ; and
Prepare a test case with a high probability of finding an as-yet undiscovered error.
Levels of Unit Testing
UNIT
100% code coverage
INTEGRATION
SYSTEM
ACCEPTANCE
MAINTENANCE AND REGRESSION
Concepts in Unit Testing:
The most 'micro' scale of testing;
To test particular functions or code modules.
Typically done by the programmer and not by testers.
As it requires detailed knowledge of the internal program design and code.
Not always easily done unless the application has a well-designed architecture with tight code;
Types of Errors detected
The following are the Types of errors that may be caught
Error in Data Structures
Performance Errors
Logic Errors
Validity of alternate and exception flows
Identified at analysis/design stages
Unit Testing – Black Box Approach
Field Level Check
Field Level Validation
User Interface Check
Functional Level Check
Unit Testing – White Box Approach
STATEMENT COVERAGE
DECISION COVERAGE
CONDITION COVERAGE
MULTIPLE CONDITION COVERAGE (nested conditions)
CONDITION/DECISION COVERAGE
PATH COVERAGE
Unit Testing – Field Level Checks
Null / Not Null Checks
Unit Testing – Field Level Validations
Test all Validations for an Input field
Date Range Checks (From Date/To Date’s)
Date Check Validation with System date
Unit Testing – User Interface Checks
Readability of the Controls
Tool Tips Validation
Ease of Use of Interface Across
Tab related Checks
User Interface Dialog
GUI compliance checks
Unit Testing - Functionality Checks
Screen Functionalities
Field Dependencies
Auto Generation
Algorithms and Computations
Normal and Abnormal terminations
Specific Business Rules if any..
Unit Testing - Other Measures
Function coverage
Loop coverage
Race coverage
Execution of Unit Tests
Design a test case for every statement to be executed.
Select the unique set of test cases.
This measure reports whether each executable statement is encountered.
Also known as: line coverage, segment coverage and basic block coverage.
Basic block coverage is the same as statement coverage except the unit of code
measured is each sequence of non-branching statements.
Unit Testing Flow :
Advantage of Unit Testing
Can be applied directly to object code and does not require processing source code.
Performance profilers commonly implement this measure.
Disadvantage of Unit Testing
Insensitive to some control structures (number of iterations)
Does not report whether loops reach their termination condition
Statement coverage is completely insensitive to the logical operators (|| and &&).
Method for Statement Coverage
Design a test-case for the pass/failure of every decision point
Select unique set of test cases
This measure reports whether Boolean expressions tested in control structures (such as the if-statement and while-statement) evaluated to both true and false.
The entire Boolean expression is considered one true-or-false predicate regardless of whether it contains logical-and or logical-or operators.
Additionally, this measure includes coverage of switch-statement cases, exception handlers, and interrupt handlers.
Also known as: branch coverage, all-edges coverage, basis path coverage, decision-path testing.
"Basis path" testing selects paths that achieve decision coverage.
Advantage:
Simplicity without the problems of statement coverage
Disadvantage
This measure ignores branches within boolean expressions which occur due to short-circuit operators.
Method for Condition Coverage:
Test if every condition (sub-expression) in decision for true/false
Select unique set of test cases.
Reports the true or false outcome of each Boolean sub-expression, separated by logical-and and logical-or if they occur.
Condition coverage measures the sub-expressions independently of each other.
Reports whether every possible combination of boolean sub-expressions occurs.
As with condition coverage, the sub-expressions are separated by logical-and and logical-or, when present.
The test cases required for full multiple condition coverage of a condition are given by the logical operator truth table for the condition.
Disadvantage:
Tedious to determine the minimum set of test cases required, especially for very complex Boolean expressions
Number of test cases required could vary substantially among conditions that have similar complexity
Condition/Decision Coverage is a hybrid measure composed by the union of condition coverage and decision coverage.
It has the advantage of simplicity but without the shortcomings of its component measures
This measure reports whether each of the possible paths in each function have been followed.
A path is a unique sequence of branches from the function entry to the exit.
Also known as predicate coverage. Predicate coverage views paths as possible combinations of logical conditions
Path coverage has the advantage of requiring very thorough testing
Function Coverage:
This measure reports whether you invoked each function or procedure.
It is useful during preliminary testing to assure at least some coverage in all areas of the software.
Broad, shallow testing finds gross deficiencies in a test suite quickly.
Loop Coverage
This measure reports whether you executed each loop body zero times, exactly once, twice and more than twice (consecutively).
For do-while loops, loop coverage reports whether you executed the body exactly once, and more than once.
The valuable aspect of this measure is determining whether while-loops and for-loops execute more than once, information not reported by others measure.
Race Coverage
This measure reports whether multiple threads execute the same code at the same time.
Helps detect failure to synchronize access to resources.
Useful for testing multi-threaded programs such as in an operating system.
Integration Testing
Integration testing is the testing of a partially integrated application to identify defects involving the interaction of collaborating components.
Objectives
The typical objectives of integration testing are to:
Determine if components will work properly together.
Identify defects that are not easily identified during unit testing.
Kinds of Integration Testing
Integration testing includes the following kinds of testing:
Commercial Component Integration
Commercial component integration testing is the integration testing of multiple commercialoff- the-shelf (COTS) software components to determine if they are not interoperable (i.e., if they contain any interface defects).
Software Integration
Software integration testing is the incremental integration testing of two or more integrated software components on a single platform to produce failures caused by interface defects.
System Integration
System integration testing is the integration testing of two or more system components.
Specifically, system integration testing is the testing of software components that have been distributed across multiple platforms (e.g., client, web server, application server, and database
server) to produce failures caused by system integration defects (i.e., defects involving distribution and back-office integration).
Guidelines
The iterative and incremental development cycle implies that integration testing is regularly performed in an iterative and incremental manner.
Integration testing must be automated if adequate regression testing is to occur.
System Testing
For most organizations, software and system testing represents a significant element of a project's cost in terms of money and management time. Making this function more effective can deliver a range of benefits including reductions in risk, development costs and improved 'time to market' for new systems.
Systems with software components and software-intensive systems are more and more complex everyday. Industry sectors such as telecom, automotive, railway, and aeronautical and space, are good examples. It is often agreed that testing is essential to manufacture reliable products.
However, the validation process does not often receive the required attention. Moreover, the validation process is close to other activities such as conformance, acceptance and qualification testing.
The difference between function testing and system testing is that now the focus is on the whole application and its environment . Therefore the program has to be given completely. This does not mean that now single functions of the whole program are tested, because this would be too redundant. The main goal is rather to demonstrate the discrepancies of the product from its requirements and its documentation. In other words, this again includes the question, ``Did we build the right product?'' and not just, ``Did we build the product right?'' However, system testing does not only deal with this more economical problem, it also contains some aspects that are orientated on the word ``system'' . This means that those tests should be done in the environment for which the program was designed, like a mulituser network or whetever. Even security guide lines have to be included. Once again, it is beyond doubt that this test cannot be done completely, and nevertheless, while this is one of the most incomplete test methods, it is one of the most important.
A number of time-domain software reliability models attempt to predict the growth of a system's reliability during the system test phase of the development life cycle. In this paper we examine the results of applying several types of Poisson-process models to the development of a large system for which system test was performed in two parallel tracks, using different strategies for test data selection.
We will test that the functionality of your systems meets with your specifications, integrating with which-ever type of development methodology you are applying. We test for errors that users are likely to make as they interact with the application as well as your application’s ability to trap errors gracefully. These techniques can be applied flexibly, whether testing a financial system,
ecommerce, an online casino or games testing.
System Testing is more than just functional testing, however, and can, when appropriate, also encompass many other types of testing, such as:
Security
Load/stress
Performance
Browser compatibility
Localisation
Need for System Testing
Effective software testing, as a part of software engineering, has been proven over the last 3decades to deliver real business benefits including:
These benefits are achieved as a result of some fundamental principles of testing, for example, increased independence naturally increases objectivity. Your test strategy must take into consideration the risks to your organisation, commercial and technical. You will have a personal interest in its success in which case it is only human for your objectivity to be compromised.
System Testing Techniques
Goal is to evaluate the system as a whole, not its parts
Techniques can be structural or functional
Techniques can be used in any stage that tests the system as a whole (acceptance,
installation, etc.)
Techniques not mutually exclusive
Structural techniques
Stress testing - test larger-than-normal capacity in terms of transactions, data, users, speed, etc.
Execution testing- test performance in terms of speed, precision, etc.
Recovery testing - test how the system recovers from a disaster, how it handles corrupted data, etc.
Operations testing - test how the system fits in with existing operations and procedures in the user organization
Compliance testing - test adherence to standards
Security testing - test security requirements
Functional techniques
Requirements testing - fundamental form of testing - makes sure the system does what it’s required to do
Regression testing - make sure unchanged functionality remains unchanged error-handling testing - test required error-error-handling functions (usually user error)
Manual-support testing - test that the system can be used properly - includes user documentation
Intersystem handling testing - test that the system is compatible with other systems in the environment
Control testing - test required control mechanisms
Parallel testing - feed same input into two versions of the system to make sure they
Produce the same output
Unit Testing
Goal is to evaluate some piece (file, program, module, component, etc.) in isolation
Techniques can be structural or functional
In practice, it’s usually ad-hoc and looks a lot like debugging
More structured approaches exist
Functional techniques
Input domain testing - pick test cases representative of the range of allowable input, including high, low, and average values
Equivalence partitioning - partition the range of allowable input so that the program is expected to behave similarly for all inputs in a given partition, then pick a test case from each partition
Boundary value - choose test cases with input values at the boundary (both inside and outside) of the allowable range
Syntax checking - choose test cases that violate the format rules for input
Special values - design test cases that use input values that represent special situations
Output domain testing - pick test cases that will produce output at the extremes of the output domain
Structural techniques
Statement testing - ensure the set of test cases exercises every statement at least once
Branch testing - each branch of an if/then statement is exercised
Conditional testing - each truth statement is exercised both true and false
Expression testing - every part of every expression is exercised
Path testing - every path is exercised (impossible in practice)
Error-based techniques
Basic idea is that if you know something about the nature of the defects in the code, you can estimate whether or not you’ve found all of them or not
Fault seeding - put a certain number of known faults into the code, then test until they are all found
Mutation testing - create mutants of the program by making single changes, then run test cases until all mutants have been killed
Historical test data - an organization keeps records of the average numbers of defects in the products it produces, then tests a new product until the number of defects found approaches the expected number
SUMMARY
Testing irrespective of the phases of testing should encompass the following:
Cost of Failure associated with defective products getting shipped and used by customer is enormous
To find out whether the integrated product work as per the customer requirements
To evaluate the product with an independent perspective
To identify as many defects as possible before the customer finds
To reduce the risk of releasing the product
Hence the system Test phase should begin once modules are integrated enough to
perform tests in a whole system environment. System testing can occur in parallel with integration test, especially with the top-down method.
Test your Understanding
1). Unit testing is predominantly a. white-box oriented b. black-box oriented
c. both black-and-white-box oriented d. none of the above
2). In general, unit testing is performed by:
a. an independent test group b. the software engineer c. SQA
d. The customer
Answers:
1) a 2) b