System testing is the most misunderstood and most difficult testing pro- cess. System testing isnot a process of testing the functions of the complete system or program, because this would be redundant with the process of function testing. Rather, as shown in Figure 6.3, system testing has a
particular purpose: to compare the system or program to its original objec- tives. Given this purpose, consider these two implications:
1. System testing is not limited to systems. If the product is a program, system testing is the process of attempting to demonstrate how the program, as a whole, fails to meet its objectives.
2. System testing, by definition, is impossible if there is no set of writ- ten, measurable objectives for the product.
In looking for discrepancies between the program and its objectives, focus on translation errors made during the process of designing the external specification. This makes the system test a vital test process, be- cause in terms of the product, the number of errors made, and the sever- ity of those errors, this step in the development cycle usually is the most error prone.
It also implies that, unlike the function test, the external specification cannot be used as the basis for deriving the system test cases, since this would subvert the purpose of the system test. On the other hand, the objectives document cannot be used by itself to formulate test cases, since it does not, by definition, contain precise descriptions of the program’s external interfaces. We solve this dilemma by using the program’s user documentation or publications—design the system test by analyzing the objectives; formulate test cases by analyzing the user documentation. This has the useful side effect of comparing the program to its objectives and to the user documentation, as well as comparing the user documentation to the objectives, as shown in Figure 6.4.
Figure 6.4 illustrates why system testing is the most difficult testing pro- cess. The leftmost arrow in the figure, comparing the program to its objec- tives, is the central purpose of the system test, but there are no known test- case design methodologies. The reason for this is that objectives state what a program should do and how well the program should do it, but they do not state the representation of the program’s functions. For instance, the objectives for the DISPLAYcommand specified in Chapter 4 might have read as follows:
A command will be provided to view, from a terminal, the contents of main storage locations. Its syntax should be consistent with the syn- tax of all other system commands. The user should be able to specify 120 The Art of Software Testing
C06 08/25/2011 12:12:59 Page 121
a range of locations, via an address range or an address and a count. Sensible defaults should be provided for command operands.
Output should be displayed as multiple lines of multiple words (in hexadecimal), with spacing between the words. Each line should contain the address of the first word of that line. The command is a ‘‘trivial’’ command, meaning that under reasonable system loads, it should begin displaying output within two seconds, and there should be no observable delay time between output lines. A programming error in the command processor should, at the worst, cause the com- mand to fail; the system and the user’s session must not be affected. The command processor should have no more than one user- detected error after the system is put into production.
Given the statement of objectives, there is no identifiable methodology that would yield a set of test cases, other than the vague but useful guide- line of writing test cases to attempt to show that the program is in- consistent with each sentence in the objectives statement. Hence, a different approach to test-case design is taken here: Rather than describing a methodology, distinct categories of system test cases are discussed. Be- cause of the absence of a methodology, system testing requires a substantial FIGURE 6.4 The System Test.
amount of creativity; in fact, the design of good system test cases requires more creativity, intelligence, and experience than are required to design the system or program itself.
Table 6.1 lists 15 categories of test cases, along with a brief description. We discuss the categories in turn here. We don’t claim that all 15 catego- ries apply to every program, but to avoid overlooking something, we rec- ommend that you explore all of them when designing test cases.
TABLE 6.1 15 Categories of Test Cases
Category Description
Facility Ensure that the functionality in the objectives is implemented.
Volume Subject the program to abnormally large volumes of data to
process.
Stress Subject the program to abnormally large loads, generally
concurrent processing.
Usability Determine how well the end user can interact with the program.
Security Try to subvert the program’s security measures.
Performance Determine whether the program meets response and
throughput requirements.
Storage Ensure the program correctly manages its storage needs, both
system and physical.
Configuration Check that the program performs adequately on the
recommended configurations. Compatibility/
Conversion
Determine whether new versions of the program are compatible with previous releases.
Installation Ensure the installation methods work on all supported
platforms.
Reliability Determine whether the program meets reliability specifications
such as uptime and MTBF.
Recovery Test whether the system’s recovery facilities work as designed.
Serviceability/ Maintenance
Determine whether the application correctly provides
mechanisms to yield data on events requiring technical support.
Documentation Validate the accuracy of all user documentation.
Procedure Determine the accuracy of special procedures required to use or
maintain the program. 122 The Art of Software Testing
C06 08/25/2011 12:13:1 Page 123