Testing Process Models

Traditionally, efforts to improve quality have centered around the end of the product development cycle by emphasizing the detection and correction of defects (Kshirasagar Naik,2008). It is the common practice of the Waterfall software development process, in which testing concerns are postponed to the end of the development cycle, after the implementation has started, or even after it has ended.

Notwithstanding, by waiting until this late in the process, testing ends up being compressed, in which not enough time and budget resources remain, problems with previous stages have been solved by taking time and money from testing phase, and testers do not have enough time

2.2. SOFTWARE TESTING

to plan for testing. Instead of planning and designing tests, the developers have time only to run tests, usually without being guided by a process, but rather in an ad hoc fashion (Ammann and Offutt,2008).

An approach for testing to enhancing quality should start at the beginning of a project, long before any program code has been written, encompassing all phases of a product development process. Testing should occur at different places and times during the development life cycle - from the requirements analysis to the final delivery of the product to the customer. The requirements documentation is verified first; then, in the later stages of the project, testing can concentrate on ensuring the quality of the application code. Expensive reworking is minimized by eliminating requirements-related defects early in the project’s life, prior to detailed design or coding work.

The idea of integrating testing activities into the earlier software life cycle phases is illustrated by the V-model (Figure2.4). The V-model gives equal weight to development and testing rather than treating testing as an afterthought (Goldsmith and Graham,2002).

Figure 2.4 Testing in the V-model.

There are other models for testing process that can be combined with the ones aforementioned It is the case of Spiral Testing, which work with the idea of software increments, and prototypes evolving into applications (Mathur,2009). In this model, testing activities evolve over time and with the prototype, increasing their sophistication as the project evolves. It can thus be applied to any incremental development process.

2.2. SOFTWARE TESTING

Testing Levels

As mentioned above, in a quality-driven development process, testing activities should be performed along the whole life cycle, considering all phases, in order to organize the testing process as well as to find major problems earlier in the development and thus avoiding resources waste. Early identification of defects is by far the best means of reducing their ultimate cost.

The testing activities mentioned are then expressed as testing levels. A different level of testing accompanies each distinct software development activity, consequently the information for each test level is typically derived from the associated development activity.

The idea behind splitting testing into levels is to build code and test it in pieces and gradually put together into larger and larger portions, in order to avoid surprises when the entire product is linked together (Patton,2005).

The common testing levels are following described:

• Unit Testing: It is designed to assess the units produced by the implementation phase and is the “lowest" level of testing. Unit testing has as goal the capability to ensure that each individual software unit is functioning according to its specification (Burnstein, 2002). Most companies make unit testing the responsibility of the programmer. It is straightforward to package unit tests together with the corresponding code through the use of tools such as JUnit for Java classes (Ammann and Offutt,2008). Each unit test must run independently of all other units as well as unit tests must be able to run in any order.

In addition, one test must not depend on some side effect caused by a previous test, e.g. a member variable being left in a certain state.

• Module Testing: It is designed to assess individual modules in isolation, including how the component units interact with each other and their associated data structures. As with unit testing, most software development organizations make module testing the responsibility of the programmer (Ammann and Offutt,2008). It is possible to describe a process merging module and unit testing objectives in a single level.

• Integration Testing: As the units and/or modules are tested and the low-level bugs are fixed, they are then integrated and integration testing occurs. It is designed to assess whether the interfaces between units (or modules) in a given subsystem have consistent assumptions and communicate correctly. Integration testing must assume that modules work correctly (Ammann and Offutt,2008).

• System Testing: Its purpose is to compare the system to its original objectives. It assumes

2.2. SOFTWARE TESTING

that the pieces work individually, and asks if the system works as a whole (Myers and Sandler,2004). This level of testing usually looks for design and specification problems, identified at Analysis phase. It is a very expensive place to find lower-level faults and is usually not done by the programmers, but by a separate testing team (Ammann and Offutt, 2008).

• Acceptance Testing: It is designed to determine whether the completed software in fact meets customers’ requirements, gained through Requirements Analysis phase. Acceptance testing probes whether the software does what the users want. It must involve users or other individuals who have strong domain knowledge (Ammann and Offutt,2008).

Testing Coverage

In an ideal world, we would want to test every possible permutation of a program. In most cases, however, this simply is not possible (Ammann and Offutt,2008;Craig and Jaskiel,2002;

Myers and Sandler,2004;Patton,2005). The features and attributes of a simple application may result in millions of permutations that could potentially be developed into test cases. Even a seemingly simple program can have hundreds or thousands of possible input and output combinations. Obviously, creating test cases for all of these possibilities is impractical. Even if a large number of test cases are created, they generally still represent only a tiny fraction of the possible combinations. Several other combinations may still exist. As said byCraig and Jaskiel (2002), in most cases, what is tested in a system is much more important than how much it is tested.

Since we cannot test with all inputs, it is necessary to determine a testing coverage criteria.

Coverage criteria are used to decide the what aforementioned, i.e. which test inputs to use, providing suitable stopping rules for testing. A formal coverage criteria give test engineers ways to decide what test inputs to use during testing, making it more likely that the tester will find problems in the program and providing greater assurance that the software is of high quality and reliability (Ammann and Offutt,2008).

Coverage criteria can be defined in terms of test requirements. These can be described with respect to a variety of software artifacts, including the source code, design components, specification modeling elements, or even descriptions of the input space. Testers have to understand the testing coverage required by a project, and it should be detailed in the test plan document. Furthermore, coverage criteria act as basis for test case design. As an example, in some projects there may be contractual agreements that list all functional requirements to be