Literature Review Method

that a summary of each area was presented.

These studies can be considered good sources of information on this subject. In order to develop our work, we considered every mentioned study, since they bring relevant information.

However, we have noticed that important aspects were not covered by them in an extent that should be possible to map out the current status of research and practice of the area. Thus, we categorized a set of important research areas under SPL testing, focusing on aspects addressed by the studies mentioned before as well as the areas they did not addressed, but are directly related to SPL practices, in order to perform critical analysis and appraisal. In order to accomplish our goals in this work, we followed the guidelines for mapping studies development presented in Budgen et al.(2008). We also included threats mitigation strategies in order to have the most reliable results.

We believe our study states current and relevant information on research topics that can complement others previously published. By current, we mean that, as the number of studies published has increased rapidly, as shown in Figure3.4, it justifies the need of more up to date empirical research in this area to contribute to the community investigations.

3.3 Literature Review Method

The method used in this research is a Systematic Mapping Study (henceforth abbreviated to as

’MS’) (Budgen et al.,2008;Petersen et al.,2008). A MS provides a systematic and objective procedure for identifying the nature and extent of the empirical study data that is available to answer a particular research question (Budgen et al.,2008).

While a Systematic Review is a mean of identifying, evaluating and interpreting all avail-able research relevant to a particular question (Kitchenham and Charters, 2007), a MS in-tends to ’map out’ the research undertaken rather than to answer detailed research question (Budgen et al.,2008;Petersen et al.,2008). A well-organized set of good practices and proce-dures for undertaking MS in the software engineering context is defined in (Budgen et al.,2008;

Petersen et al.,2008), which establishes the base for this mapping study. It is worthwhile to highlight that the importance and use of MS in the software engineering area is increasing (Afzal et al.,2008;Bailey et al.,2007;Budgen et al.,2008;Condori-Fernández et al.,2009;Juristo et al.,2006;Kitchenham,2010;Petersen et al.,2008;Pretorius and Budgen,2008), showing the relevance and potential of the method. Nevertheless, of the same way as systematic reviews (Bezerra et al.,2009;Chen et al.,2009;Lisboa et al.,2010;Moraes et al.,2009;Souza Filho et al.,2008), we need more MS related to software product lines, in order to evolve the field

3.3. LITERATURE REVIEW METHOD

with more evidence (Kitchenham et al.,2004).

A MS comprises the analysis of primary studies that investigate aspects related predefined research questions, aiming at integrating and synthesizing evidence to support or refute particular research hypotheses. The main reasons to perform a MS can be stated as follows, as defined by Budgen et al.(2008):

• To make an unbiased assessment of as many studies as possible, identifying existing gaps in current research and contributing to the research community with the reliable synthesis of the data;

• To provide a systematic procedure for identifying the nature and extent of the empirical study data that is available to answer research questions;

• To map out the research that has been undertaken;

• To help to plan new research, avoiding unnecessary duplication of effort and error;

• To identify gaps and clusters in a set of primary studies, in order to identify topics and areas to perform more complete systematic reviews.

The experimental software engineering community is working towards the definition of standard processes for conducting mapping studies. This effort can be checked out in the initial study fromPetersen et al.(2008), which describes how to conduct systematic mapping studies in software engineering. The paper provide a well defined process which serves as a starting point for our work. We merged ideas from (Petersen et al.,2008) with good practices defined in the guidelines published by (Kitchenham and Charters,2007). This way, we could apply a process for mapping study including good practices of conducting systematic reviews, making better use of the both techniques.

This blending process enabled us to include topics not covered byPetersen et al.(2008) in their study, such as:

• Protocol. This artifact was adopted from systematic review guidelines. Our initial activity in this study was to develop a protocol, i.e. a plan defining the basic mapping study procedures. Searching in the literature, we noticed that some studies created a protocol (e.g. (Afzal et al.,2009)), while other do not (e.g. (Condori-Fernández et al., 2009;

Petersen et al.,2008)). Even though this is not a mandatory artifact, as mentioned by Petersen et al.(2008), authors who created a protocol in their studies encourage the use this artifact as being important to evaluate and calibrate the mapping study process.

3.3. LITERATURE REVIEW METHOD

• Collection Form. This artifact was also adopted from systematic review guidelines and its main purpose is to help the researchers in order to collect all the information needed to address the review questions, study quality criteria and classification scheme.

• Quality Criteria. The purpose of quality criteria is to evaluate the studies, as a means of weighting their relevance against others. Quality criteria are commonly used when performing systematic literature reviews. The quality criteria were evaluated independently by two researchers, hopefully reducing the likelihood of erroneous results.

Some elements, as proposed byPetersen et al.(2008) were also changed and/or rearranged in this study, such as:

• Phasing mapping study. As can be seen in Figure3.1, the process was explicitly split into three main phases: 1 - Research Directives, 2 - Data Collection and 3 - Results. It is in line with systematic reviews practices (Kitchenham and Charters,2007), which defines planning, conducting and reportingphases. Phases are named differently from what is defined for systematic reviews, but the general idea and objective for each phase was followed. In the first, the protocol and the research questions are established. This is the most important phase, since the research goal is satisfied with answers to these questions.

The second phase comprises the execution of the MS, in which the search for primary studies is performed. This consider a set of inclusion and exclusion criteria, used in order to select studies that may contain relevant results according to the goals of the research.

In third phase, the classification scheme is developed. The results of a meticulous analysis performed with every selected primary study is reported, in a form of a mapping study.

All phases are detailed in next sections.

Figure 3.1 The Systematic Mapping Process (adapted fromPetersen et al.(2008)).