Qualitative Analysis

In the quantitative analysis, the idea was to evaluate mainly the metrics Feasibility and Visualization, but other aspects as the training provided and the difficulties faced by the subjects are also addressed. This analysis was based on the questionnaire presented in AppendixB.3.

Feasibility. Three subjects agreed with the statement that "CodeScoping can support the execution of a SPL scoping process." and one subject (ID 4) strongly agreed. Then, we can assume that 100% of the subjects believe that the tool can supports the scoping process execution. This result indicates that the null hypothesis H0c: µsupport in the process<80%

can berejected.

On the other hand, the subjects reported that a long time still was spent on source code analysis to confirm the similarity ratio provided by the tool, since they do not have confidence in the results presented because it was their first use. In this sense, they suggested that the tool should ensure the precision similarity comparison, letting the source code analysis just for exceptional cases.

Visualization. Regarding the proposed visualization provided by CodeScoping, two subjects (IDs 2, 3) agreed and one subject (ID 4) strongly agreed with the statement "The visualization proposed in CodeScoping is very useful to perform the scope analysis of a product line.". They reported that the visualization is simple and facilitate the source code analysis. However, the subject (ID 1) neither agrees nor disagrees of the statement, reporting that did not think it is intuitive the use of squares to represent the source files. Thus, we can assume that 75% of the subjects consider the visualization useful to perform the scoping analysis, therefore, this result didnot reject the null hypothesis H0d: µvisualization usefulness<80%.

The participant (ID 4) also commented that would like to see the visualization applied

6.6. ANALYSIS AND INTERPRETATION to real systems of high complexity, since he believes that new visualizations will have to be developed and embedded into the tool.

Training. The training provided about CodeScoping was based on documents de-scribing its features and its analysis workflow. Three subjects (75%) agreed that the training was effective and sufficient and one subject (ID 3) remained neutral. Moreover, they highlighted that the tool is simple and intuitive. However, all participants reported that a live training would be better, since the learning curve would be shorter and the features would best explained.

6.6.3 Lessons Learned

In order to replicate this experiment, some important aspects should be considered, mainly the ones seen as limitations in this initial experiment.

Training. In this experiment, the training was composed of documents explaining how to perform the experiment and the tool features. Although most of the subjects have held that the training was sufficient, some doubts were solved by online chat. In addition, we observed in the scoping analysis with CodeScoping that some features were not correctly configured due to mistakes in tool usage. Thus, we believe that a training in real time about the experiment and the tool should be performed before the experiment execution.

Experiment objects. Toy systems were used in the experiment, due to the difficulty to find real systems suitable for the tool test. Thus, the problem of analyzing features of complex systems may not have been depicted in the experiment, which facilitated the manual analysis (ad-hoc).

Experiment design. Due to the number of subjects available to perform the ex-periment, we choose to compare the same subjects using the two approaches, with CodeScoping and after without tool support. If only one set of systems were used for analysis, the experiment would be affected by the maturation effect, when the subjects learn along the experiment (Wohlin et al.,2000). In order to avoid this effect, two sets of systems were used. Despite the set of systems have similar complexities, the ideal design would be that the same systems were analyzed by two groups of subjects, each of them using a different approach.

6.7. CHAPTER SUMMARY

6.7 Chapter Summary

In this chapter was presented a preliminary experiment to evaluate the tool developed in this work: CodeScoping. The experiment was conducted with four scope analysts and all phases involved in the experiment were detailed (definition, planning, operation, analysis and interpretation).

The experiment results showed that the proposed tool is not yet ready for real use and needs to be evolved. In contrast, the experiment subjects believe that the new proposed approach is promising, and the information precision provided by the tool should be a priority to convey confidence to its users. In addition, the items pointed in the lessons learned should be addressed and the experiment should be replicated more times to ensure more concrete conclusions.

Next chapter presents the concluding remarks and future work of this dissertation.

Conclusion 7

The extractive adoption model enables a very quickly transition from single software development to a product line paradigm (Krueger, 2002), enabling organizations to take advantage of benefits, such as reduction of development costs and time-to-market, enhancement of quality and increasing of customer satisfaction. Since existing software artifacts from one or more existing products are extracted and reengineered to serve as the core asset inputs for the product line, this model is typically used in practice by big companies, such as, Bosch Gasoline Systems, Nokia Mobile Phones, Nokia Networks and Philips Medical Systems (Linden et al.,2007).

In this context, the product line scope is defined on analysis of the available documen-tation of existing products and based on knowledge from domain experts, who are the only people who know in detail the products and its features that will compose the product line.

In contrast, this may be considered an effort-intensive task since much time is invested in workshops and interviews with the domain experts (John,2010). Moreover, often, the domain experts do not have time to share their knowledge and the documentation of existing products is inexistent or outdated.

Given the existence and relevance of the problem, and the constant search for cutting costs in adoption of software product lines, it was developed CodeScoping, a tool to support the SPL scoping process based on legacy products source code. We believe that the source code of existing products contains information about variability and commonality, and can be used to guide the product line scope definition.

For the tool development, an empirical study (focus group) was conducted with scope analysts in an industrial setting and an analysis of the literature in different areas, such software product lines, software visualization, software maintenance and software reengineering. In addition, the tool was evaluated through a formal experiment, allowing the study can be replicated in the future.

7.1. RESEARCH CONTRIBUTIONS

7.1 Research Contributions

This work has three main contributions: (i) afocus group study, in which an empirical study was conducted in an industrial setting to characterize the SPL scoping process; (ii) theCodeScoping, the proposed tool which was developed to support the scope analysis in product lines; and finally (iii) apreliminary experiment, which evaluate the feasibility of the proposed tool. Each one is following detailed: