Development and Derivation of Test Cases

Figure4.13 depicts the dependency between test objective and test case when variability is considered (Wübbeke,2008). In (A) the component is variable as a whole, and in (B) only part of the component is variable. They will turn, respectively, into (A’) and (B’), the former as a new test case, which is variable as a whole, and the latter, a test case in which only the corresponding part of the test case is variable.

4.6. MANAGING TEST ASSETS VARIABILITY WITHIN RIPLE-TE

Figure 4.12 The metamodel for tests.

Hence, the challenge is how to optimally build test case that take into consideration variability aspects so that reuse the parts of test cases will emerge easily.

Figure 4.13 Dependency between test objective and test cases considering variability

As an example of this possible flow, we represent in Figure 4.14a use case in a form of an activity diagram, which depicts its set of steps. In this diagram, the white diamond below step A depicts a variation point in which there is one mandatory feature, depicted by step B and an optional feature, depicted by E. The black diamond below step E is also representing a variation point, which has two variants, denoted as F and G. These are alternative variants, so that only one can be instantiated in a product. Figure4.15depicts the three possible scenarios:

(1) [A-B-C-D]; (2) [A-B-C-D-E-F]; e (3) [A-B-C-D-F-G].

This is the typical case presented in Figure4.13, in which only part of a use case varies. In this case, it is not necessary to create different use cases to represent the variability but rather reuse part of it. Returning to the metamodel, this representation is enabled since, according to the model specification, every step in a use case can make reference to a subflow. Thus, in order to test these three possible scenarios, we could create the test case for scenario (1), and then

4.6. MANAGING TEST ASSETS VARIABILITY WITHIN RIPLE-TE

Figure 4.14 Example activity diagram including variability

Figure 4.15 Example activity diagram showing possible scenarios

reuse the flow of this scenario and the results as well for the remaining scenarios (2) and (3).

It is based on the control-flow coverage criteria, which is based on graph representation. We applied such a representation, but evolve it considering peculiar aspects of variability.

The same idea could be applied to the sequence diagrams, artifact built in the design phase.

We can represent variation points and variants by using a stereotype tag, so that scenarios are likely to be easily identified. Next step, hence, is to create the test assets, according to this representation. The reuse strategy is also feasible.

Considering that a use case can generate several test cases, the strategy to represent each

4.7. CHAPTER SUMMARY

step in a use case and enable it to be linked to a variation point enables the building of test cases which also consider the variation points. This way, several test cases can be instantiated from a use case, that have represented the variation points. Variability is preserved in the core asset test artifacts to facilitate reuse. This strategy enables that every change in the use case and/or the variation points and variants be propagated to the test cases and their steps (variable or not), due to the traceability represented.

In summary, test cases are derived from use scenarios, that can emerge from both use cases (depicted as activity diagrams) and design models (sequence diagrams), hence we can apply this strategy in every level that comprises the RiPLE-TE.

4.7 Chapter Summary

This Chapter presented the proposed process for testing in Software Product Lines, named RiPLE-TE. Throughout this chapter we presented the process roles and attributions, work products, activities, and associated steps. This proposal was raised in order to meet a gap found in the literature which does not include detailed approaches for conducting testing in SPL projects, but instead only parts of approaches are available. As SPL is mainly concerned with variability management, we presented how test assets could be built in a SPL environment, in which variability takes place in many artifacts. UML is used to represented variability in the process metamodel. Traceability and evolution management issues are also handled.

Adaptations can be made on the basis of the guidelines we proposed along this Chapter, in order to meet some organization specific goals. As an example, we do not include exploratory testing issues when executing test cases. However, it is possible that a kind of exploratory testing should be included in the context of the RiPLE-TE process. It is an optional step for the execution activities, but strongly relevant. As testers gain knowledge on the domain and the products under test, they are capable enough to ask the right questions to the software. However, a drawback is that with exploratory tests, how testers encountered defects are commonly not recorded, going against the SPL principles where assets should be tracked and recorded in a repository for future reuse.

We have not performed any comparative analysis to existing approaches, since the level of details we provide in our process is not provided by any other, hence such an analysis might not have relevance, due to the scope in which it would be conducted.

Next Chapter present an initial experimental study on the basis of this proposal, performed as a way to understand the behaviour od a process for Testing in SPL.

5

Experimental Evaluation

Software development has been characterized from its origins by the need of empirical facts tested against reality, which provide evidence of the advantages or disadvantages of the different methods, techniques or tools used in the development of software systems (Juristo et al., 2002). Wohlin et al.(2000) states in his well accepted textbook on experimentation in software engineering that the only real evaluation of a process or process improvement proposal is to have people using it, since the process is just a description until it is used by people. In fact, the path from subjectivity to objectivity is paved by testing or empirical comparison with reality (Juristo et al.,2004).

Besides, while nearly all computer scientists have strong intuition about the link between unit testing and its effectiveness in overall software development, we are not aware of any systematic studies examining this test level in SPL projects in terms of effectivity and effort saving as well. Thus, we conducted two experimental studies in order to evaluate our proposal.

The same project was applied in both experiments, but in different environments. The replication was motivated by the desire of achieving more significance and confidence on the results.

This Chapter describes and motivates the design of the experiments and discusses the threats to validity. The experimental study was conceived and structured based on the concepts of experimental software engineering and the evaluation of software engineering methods and tools provided byWohlin et al.(2000).

Following the experiment process defined in (Wohlin et al.,2000), we firstly defined the purpose of the experiment and discussed the context in which the both experiments took place in Section5.1. Then Section5.2details the first experiment, comprising subsections: Section5.2.1, which reports on the experiment planning and design; Section5.2.2, detailing the project used in this experimental study; Section 5.2.3, which outlines the preparation and execution of the experiment; and Section 5.2.4, which presents the analysis of the collected data. Then,