Learning Objectives:
After studying this chapter, you should be able to:
• Describe how system testing differs from unit and integration testing.
• Classify tests as function testing, performance testing, acceptance testing or installation testing.
• Understand the purposes and roles of function testing, performance testing, acceptance testing, and installation testing.
• Define software reliability, maintainability and availability.
• Describe different techniques for measuring reliability, maintainability and availability.
• List the different types of test documentation and know what items belong in test documentation.
• Understand the special problems associated with testing safety-critical systems.
• Describe the principles of Cleanroom and how it differs from conventional testing.
Summary:
This chapter looks at the system testing process: its purpose, steps,
participants, techniques and tools. The chapter describes the principles of system testing, including reuse of test suites and data, and the need for careful configuration management. The concepts intro duced include function testing, performance testing, acceptance testing and installation testing. The chapter examines the special needs of testing object -oriented systems. Several test tools are described, and the roles of test team members are discusse d. The reader is introduced to software reliability modeling. The issues of
reliability, maintainability and availability are discussed. The chapter describes how to use the results of testing to estimate the likely
characteristics of the delivered prod uct. Several types of test documentation are described.
Testing the system is very different from unit and integration testing. When unit testing components, the developer has complete control over the testing process. The developer creates the test dat a, designs the test cases, and runs the tests. When integrating components, the developer sometimes works
individually, but often collaborates with a small part of the test or
development team. However, when testing a system, the developer works with the entire development team, coordinated and directed by the test team leader.
The objective of unit and integration testing is to ensure that the code
implements the design properly. In system testing, however, the objective is to ensure that the system does what the customer wants it to do. Test procedures should be thorough enough to exercise system functions to everyone's
satisfaction: the user, customer, and developer.
The steps involved in system testing include function testing, performance testing, acceptance testing, and installation testing. Each step has a
different focus. Function testing checks that the integrated system performs its functions as specified in the requirements. Performance testing compares the integrated components with the n onfunctional system requirements.
Acceptance testing assures the customers that the system they requested is the
system that was built for them. Installation testing allows users to exercise system functions and document additional problems that result i n the actual operating environment.
Often, a system is tested in stages or pieces. System testing must also take into account the several different system configurations that are being
developed. A system configuration is a collection of system componen ts
delivered to a particular customer. During testing, configuration management, the control of system differences to minimize risk and error, is especially important. Configuration management helps to coordinate efforts among the testers and developers.
Techniques such as Cleanroom require a great deal of team planning and
coordination, in developing the box structures and in designing and running the statistical tests. And the activities involved in acceptance testing require close collaboration with customers and users; as they run tests and find problems, the team must quickly determine the cause so that corrections can allow testing to proceed. Thus, whereas some parts of development are solitary, individual tasks, testing the system is a collabora tive, group task.
Exercises:
1. How does system testing differ from unit and integration testing?
2. Explain the purposes and roles of function testing, performance testing, acceptance testing, and installation testing.
3. What is the difference between verification and validation? Which types of testing address verification? Which types of testing address validation?
4. Describe the principles of Cleanroom and how it differs from conventional testing.
5. Read the press release and failure report for the Ariane-5 Flight 501. An electronic copy of the failure report is available at
http://www.esrin.esa.it/htdocs/tidc/Press/Press96/ariane5rep.html. The joint ESA/CNES press release is avail able at
http://www.esrin.esa.it/htdocs/tidc/Pres/Press96/pres19.html. What kinds of tests might have exposed the problems that caused each of the failures?
Answer Guidelines:
1. The emphasis for unit and integration testing is to make sure the code implements the design properly. With system testing, the focus is shifted to the customer. System testing looks to verify that the system
implements the requirements properly.
For more details on the differences, re-read Section 9.1.
2. The purpose and roles of the different types of testing are presented throughout the chapter. For each type of system testing mentioned,
describe the purpose and role. Explain when in the system testing pr ocess each type of test should occur. Describe how and why each type of test is performed.
3. A verified system implies that the system operates the way the designers intended it to operate. A validated system implies that the system meets the customers' expectations. The various types of unit and integration tests focus on verification. System testing focuses on
validation. Review the descriptions of the types of unit and integration tests and the types of system testing to determine which tests contr ibute to verification and validation.
Section 9.1 contains more information on verification and validation.
4. Cleanroom reflects the ideas used in the manufacturing of chips. The goal is to keep faults at a minimum. For software, the goals are to certify software before unit testing and to produce as few faults as possible.
With Cleanroom, verification replaces unit testing. Cleanroom also makes use of statistical testing.
Section 9.9 describes the principles and the advantages and drawbacks to the Cleanroom process in greater detail. Use this information to compare the Cleanroom process against traditional testing.
5. The most obvious failure from the Ariane-5 flight was the explosion of the space rocket itself. The failure report describes additional fault s and failures that contributed to the explosion. Use the descriptions of the types of testing in this chapter to determine which tests may have
uncovered the faults. Explain how the testing would have uncovered the fault. Be sure to consider whether or not the type of test you describe would have been feasible.