Ivan do Carmo Machado

(1)

Pós-Graduação em Ciência da Computação

“RiPLE-TE: A Software Product Lines Testing Process”

By

Ivan do Carmo Machado

M.Sc. Dissertation

Universidade Federal de Pernambuco [email protected] www.cin.ufpe.br/~posgraduacao

RECIFE, AUGUST/2010

(2)

Universidade Federal de Pernambuco Centro de Informática

Pós-graduação em Ciência da Computação

Ivan do Carmo Machado

“RiPLE-TE: A Software Product Lines Testing Process”

Trabalho apresentado ao Programa de Pós-graduação em Ciência da Computação do Centro de Informática da Univer- sidade Federal de Pernambuco como requisito parcial para obtenção do grau de Mestre em Ciência da Computação.

A M.Sc. Dissertation presented to the Federal University of Pernambuco in partial fulfillment of the requirements for the degree of M.Sc. in Computer Science.

Advisor: Silvio Romero de Lemos Meira Co-Advisor: Eduardo Santana de Almeida

RECIFE, AUGUST/2010

(3)

Machado, Ivan do Carmo

RiPLE-TE: a software product lines testing process / Ivan do Carmo Machado. - Recife: O Autor, 2010.

xiii, 163 folhas : il., fig., tab.

Dissertação (mestrado) Universidade Federal de Pernambuco. CIN. Ciência da Computação, 2010.

Inclui bibliografia, anexo e apêndice.

1. Ciência da Computação. 2. Engenharia de software. 3.

Reuso de software. I. Título.

004 CDD (22. ed.) MEI2010 – 0123

(4)

(5)

To my beloved family.

(6)

Acknowledgements

First and foremost, I would like to thank my greatest teacher of all: God, for providing me this opportunity and granting me the capability to proceed successfully. I could never have done this without the faith I have in you.

I would like to gratefully acknowledge the supervision of Dr. Eduardo Almeida during this work. He provided me with many helpful suggestions and encouragement during the course of this work as well as the challenging research that lies behind it. I also wish to express my appreciation to Dr. Silvio Meira, for accepting me as M.Sc. Student. Special Gratitude goes to the rest of the teaching staff of the CIn/UFPE, for their valuable classes.

My sincere thanks are extended to Dr. Manoel Mendonça, from DMCC/UFBA, and his students, for their help during the execution of the experimental study. I would like to thank the Brazilian National Research Council (CNPq). Without their grant, this M.Sc. would not have been possible.

I cordially thank my friends and colleagues that I have met during my journey in Recife. I thank my housemates Bruno, Jonatas, Iuri and Leandro for their good friendship and for being the surrogate family during the years I stayed there. I want to express my deep thanks to my colleague Paulo Anselmo for taking intense academic interest in this study as well as providing valuable suggestions that improved the quality of this study. My cordial thanks to Danuza, Ednaldo, Flavio, Heberth, Hernan, Marcela, Ricardo, Thiago, Vanilson and Vinicius, for the insightful discussions, offering valuable advice and support. Postgraduates of the RiSE Research Group are thanked for numerous stimulating discussions. Saturdays will not be the same any longer!

I would like to express my heartiest thanks to Edna Telma for her understanding, endless patience and encouragement when it was most required, and for never letting me feel that I was away from my family. My cordial thanks to her family members for their kindness and affection, specially to Francisco Joaquim, for supporting me during these years I lived in Recife.

I also want to thanks to Benedita (Dio) for opening her wonderful home to me and making me feel so comfortable there. She provided me a perfect place to finish my work while I was in Salvador.

I cannot finish without thanking my family. I am forever indebted to my parents, Serafim and Joselice, my brother Ivo and his wife, Flavia, for their material and spiritual support in all aspects of my life. I love them and appreciate the efforts they have put into giving me the life I have now. Thank you so much!

(7)

Agradecimentos

Em primeiro lugar, agradeço ao meu maior mestre de todos: Deus, por dar vida a esta oportunidade e conceder-me a capacidade de continuar a minha jornada com êxito. Meu Pai, eu não teria realizado este sonho sem a fé que eu tenho em Ti.

Gostaria de agradecer a orientação do Professor Eduardo Almeida. Muitas sugestões úteis e incentivo durante o decorrer deste trabalho, que mostrou-se bastante desafiador, a mim foram concedidos. Expressar também o meu agradecimento ao Professor Silvio Meira, por ter me aceito como Estudante de Mestrado em uma das melhores instituições de ensino e pesquisa do Brasil. Agradeço ainda ao demais docentes do CIn/UFPE com quem tive a oportunidade de trabalhar, por suas valiosas aulas e discussões.

Meus sinceros agradecimentos ao Prof. Dr. Manoel Mendonça, do DMCC/UFBA, e aos seus alunos, por sua ajuda durante a execução do estudo experimental. Gostaria de agradecer ao Conselho Nacional de Pesquisa (CNPq), sem o apoio financeiro concedido por esta agência, não seria possível realizar este trabalho.

Agradeço de coração a meus amigos e colegas que conheci durante a minha breve “viagem”

ao Recife. Agradeço aos colegas que dividiram comigo o espaço de moradia, Bruno, Jonatas, Iuri e Leandro, e puderam propiciar um ambiente divertido durante os anos em que por lá permaneci. Quero manifestar o meu agradecimento a Paulo Anselmo, por participar diretamente do desenvolvimento deste projeto, por interesses transversais, bem como proporcionar valiosas sugestões que objetivavam a melhoria na qualidade deste trabalho. Meu cordial agradecimento a Danuza, Ednaldo, Flavio, Heberth, Hernan, Marcela, Ricardo, Thiago, Vanilson e Vinicius, que têm suas pontinhas de contribuição nesta pesquisa. Aos demais membros do RiSE eu também extendo os meus agradecimentos, por tantos momentos, especialmente os sábados, de valiosas conversas!

Gostaria de expressar meus mais sinceros agradecimentos a minha namorada Edna Telma, por sua compreensão, paciência e encorajamento quando a mim mais foi exigido, e por não deixar-me sentir a ausência de minha família. Meu cordial agradecimento a seus familiares, especialmente a Francisco Joaquim, por apoiar-me durante os anos do meu Mestrado.

Eu também quero agradecer a Benedita (Dio), por abrir as portas de sua maravilhosa casa, por fazer-me sentir tão confortável. Ela me proporcionou o ambiente perfeito, em harmonia e aconchego, para a conclusão do meu trabalho, enquanto morador da cidade de Salvador.

Não posso terminar sem agradecer a minha família. Fui, Sou e Serei eternamente grato aos meus pais, Serafim e Joselice, meu irmão Ivo e sua esposa, Flávia, por seu apoio incondicional, seja espiritual ou material, em todos os aspectos da minha vida. Saibam que é imenso o meu amor por vocês! Aprecio e agradeço, de coração, os esforços que vocês têm feito para possibilitar a vida que tenho agora! Sou orgulhoso por ter esta família! Muito Obrigado Meu Deus! Muito obrigado!

(8)

We are built to conquer environment, solve problems, achieve goals, and we find no real satisfaction or happiness in life without obstacles to conquer and goals to achieve.

—MAXWELL MALTZ (1899-1975)

(9)

Resumo

Linhas de Produtos de Software (SPL) pode ser considerada uma estratégia eficiente para o reuso de software. SPL oferece significativos benefícios econômicos para as empresas, tais como redução de custos, melhoria da qualidade e, redução do tempo de entrega de produtos.

SPL baseia-se no reuso sistemático de artefatos, através da exploração de commonalities (pontos em comum), e o gerenciamento de variabilities (pontos de variação), entre os produtos, desenvolvidos sob uma arquitetura comum.

Em SPL, atenção especial deve ser dada à qualidade dos artefatos produzidos. Em termos de garantia de qualidade, enquanto que no desenvolvimento tradicional, um programa é dito válido se pudermos garantir que ele irá funcionar corretamente, no contexto de SPL, para este mesmo cenário ser garantido, é necessário ter confiança de que qualquer instância de produto irá funcionar corretamente. Reforça-se, então, a atenção necessária mencionada anteriormente. Da mesma forma, é maior também o esforço necessário para tratar aspectos de garantia de qualidade em projetos de SPL. No entanto, a entrega de softwares com qualidade é fundamental e, talvez, a principal prática a se adotar, para que seja possível experimentar os benefícios mencionados.

Neste contexto, a entrega de produtos com qualidade deve contar com o apoio de processos bem definidos, para o estabelecimento e coordenação das atividades relacionadas. Assim, testes, como ainda a técnica de garantia de qualidade mais conhecida e aplicada, exige uma atenção especial, devido a sua característica conhecida de ser uma atividade deveras custosa. Testes em SPL é complexo e oneroso, podendo tornar-se um gargalo em projetos de SPL.

Assim, esta dissertação descreve um processo para apoiar as atividades de testes em projetos de SPL. Estabelecemos este processo com o objetivo de fornecer às organizações uma estratégia de redução de esforço na condução de atividades de teste em projetos de SPL. O processo é parte do projeto RiPLE, um esforço para a construção de um framework para SPL, que engloba o conjunto das disciplinas que compõem o ciclo de vida de desenvolvimento. Nossa pesquisa fundamenta-se em um systematic mapping study, realizado com o objetivo de fornecer a base teórica sobre o tema de investigação, bem como identificar tópicos de pesquisa a explorar.

Esta dissertação apresenta ainda uma avaliação inicial da proposta, conduzida através da realização de estudos experimentais, objetivando coletar e reunir evidências sobre a eficácia da proposta, bem como compreender, na prática, como a atividade de testes em SPL pode ser melhorada, no sentido de alcançar os benefícios e metas da SPL.

Palavras-Chave: Linhas de Produtos, Teste de Software, Reuso de Software, Processo de Software, Estudo Experimental.

(10)

Abstract

Software Product Lines (SPL) can be considered an efficient approach to intra-organizational reuse of software. SPL delivers significant economic benefits for organizations, such as reduced cost and improved quality and time-to-market. It is based upon the systematic reuse of artifacts, through exploiting commonalities and managing variabilities among products, that are established under a common architecture.

In SPL, special attention regarding quality of produced artifacts is required. In terms of quality assurance, whereas in single system development, a program is said validated if we have confidence that it will operate correctly, in SPL it is required to have confidence that any derived instance will operate correctly. It reinforces the attention aforementioned. Likewise, it also increases the effort required to deal with quality assurance in SPL projects. However, providing software with quality is fundamental and perhaps the major practice that organizations should adopt in order to experience the real SPL benefits. Thus, to provide quality products in a SPL, one should count on well-defined support processes, to establish and coordinate the activities.

Hence, testing, as still the most known and widely applied technique for quality assurance, requires such a special attention, due to its characteristic of being a resource consuming activity.

Testing a SPL is complex and costly and often becomes a bottleneck in product line projects.

Thus, this dissertation describes a process for supporting testing activities in SPL projects.

We established this process intended to provide a way for organizations to save effort when performing testing activities in a SPL environment. The proposed process is part of the RiPLE project, an effort to build a framework for SPL development, that encompasses the whole set of disciplines that comprise its life cycle. We based our research on a systematic mapping study, conducted in order to provide background on the research topic as well as to identify rooms for improvements.

In addition, this dissertation also presents an initial evaluation of our proposed process, in which experimental studies were conducted aiming to gather evidences about the proposal effectiveness, as well as to understand, by applying it in practice, how testing in software product lines can be improved upon, towards achieving software product line goals.

Keywords: Software Testing, Software Product Lines, Software Reuse, Software Process, Experimental Study.

(11)

List of Figures

1.1 RiSE Labs influences. . . 5

1.2 RiSE Labs projects. . . 6

2.1 Essential Product Line Activities . . . 13

2.2 Core Asset Development . . . 14

2.3 Product Development . . . 15

2.4 Testing in the V-model. . . 20

3.1 The Systematic Mapping Process (adapted from Petersen et al. (2008)). . . 31

3.2 Stages of the selection process. . . 37

3.3 Primary studies filtering categorized by source. . . 38

3.4 Distribution of primary studies by their publication years. . . 39

3.5 Amount of Studies vs. sources. . . 40

3.6 Distribution of papers according to classification scheme. . . 51

3.7 Distribution of papers according to intervention. . . 52

3.8 Visualization of a Systematic Map in the Form of a Bubble Plot. . . 52

4.1 RiPLE-TE main flow. . . 60

4.2 Interaction among testing levels and SPL phases, in the context of the RiPLE-TE. 62 4.3 RiPLE-TE Core Asset testing workflow . . . 66

4.4 Levels of Test Planning . . . 67

4.5 Technical Reviews . . . 69

4.6 Testing workflow . . . 71

4.7 Unit Test execution steps . . . 75

4.8 State transition diagram of test case result . . . 77

4.9 RiPLE-TE Product testing workflow . . . 82

4.10 RiPLE Metamodel for assets variability management . . . 88

4.11 The metamodel core UML profile. . . 89

4.12 The metamodel for tests. . . 91

4.13 Dependency between test objective and test cases considering variability . . . . 91

4.14 Example activity diagram including variability. . . 92

4.15 Example activity diagram showing possible scenarios . . . 92

5.1 RiSE Chair Products . . . 106

(17)

5.2 RiSE Chair Feature Model used in the experiment. . . 107

5.3 BoxPlot of defects found by groups, including outliers. . . 114

5.4 BoxPlots of Scores from defects found. . . 117

5.5 Distribution of RiPLE-TE effectiveness. . . 119 5.6 Boxplot with the distribution of subjects in (A) first and (B) second experiments.124

(18)

List of Tables

3.1 List of Research Strings . . . 35

3.2 Quality Criteria . . . 41

3.3 Research Type Facet . . . 43

3.4 Research Questions (RQ) and primary studies. . . 54

5.1 Experiment Training and Execution Agenda . . . 102

5.2 Subjects divided into groups . . . 109

5.3 Subjects’ Profile from Group 1 - Ad-hoc fashion . . . 109

5.4 Subjects’ Profile from Group 2 - RiPLE-TE . . . 110

5.5 Final grouping of subjects . . . 111

5.6 Subjects Expertise, calculated through S_{X P}formula. . . 112

5.7 Distribution of Subjects considering the expertise coefficient . . . 113

5.8 Amount of Designed Test Cases . . . 113

5.9 Amount of Defects Found . . . 113

5.10 Test Case Effectiveness . . . 114

5.11 Defects found by groups . . . 115

5.12 Difficulty and Severity of defects found . . . 116

5.13 Amount of defects found in terms of Difficulty and Severity . . . 116

5.14 Results from the t-test applied to Test Score - Experts . . . 116

5.15 Results from the t-test applied to Test Score - Non-Experts . . . 118

5.16 Results from the t-test applied to Test Score - Both groups . . . 118

5.17 Test Coverage . . . 118

5.18 Subjects’ Profile in the Experimental Study . . . 123

5.19 Subjects Expertise, calculated through SX Pformula - 2nd exp. . . 124

5.20 Distribution of Subjects considering the expertise coefficient - 2nd exp. . . 124

5.21 Amount of Designed Test Cases - 2nd exp.. . . 125

5.22 Amount of Defects Found - 2nd exp. . . 125

5.23 Test Case Effectiveness - 2nd exp. . . 125

5.24 Amount of defects found in terms of Difficulty and Severity - 2nd exp. . . 126

5.25 Scores - 2nd exp. . . 126

5.26 Test Coverage - 2nd. exp. . . 126

A.1 List of Journals . . . 148

(19)

A.2 List of Conferences . . . 149

A.3 Primary Studies Quality Score . . . 150

B.1 Consent Form . . . 156

B.2 Error Reporting Form . . . 157

(20)

Acronyms

CBD Component-based development

C.E.S.A.R. Recife Center For Advanced Studies and Systems EPF Eclipse Process Framework

GQM Goal Question Metric OMG Object Management Group

RiSE Reuse in Software Engineering Labshttp://labs.rise.com.br RiPLE RiSE Product Line Engineering Framework

SCM Software Configuration Management SPEM Software Process Engineering Metamodel SPL Software Product Lines

(21)

1

Introduction

In the software engineering field, researchers and practitioners have been searching for methods, techniques and tools that would allow for improvements in costs, time-to-market and quality (Almeida et al.,2007), particularly due to the challenges of increasing complexity and size of software systems. Software reuse is a key aspect for software organizations interested in these gains (Mili et al.,2001), since a set of reusable assets is used to solve recurring problems instead of performing the same activities over and over again (Almeida et al.,2007). However, when the reuse initiative is based on an approach focusing on small-scale, ad-hoc reuse (i.e.

typically restricted to the code level), these benefits may not be perceived (Linden et al.,2007).

On the other hand, approaches that enables systematic assembling and configuring software parts to be reused across various products, can enable organizations to experience the real software reuse benefits. In this sense, the software product line engineering (SPL), an innovative, growing concept in software engineering, was developed towards a systematic and prescribed way to achieve reuse (Klaus Pohl and van der Linden,2005). SPL is a planned, systematic and pro-active reuse strategy, through exploiting the similarities within a set of products. SPL can enable organizations to achieve significant reductions in terms of development and maintenance costs and time to market as well (Clements and Northrop,2001;Klaus Pohl and van der Linden, 2005), and remarkable quantitative improvements in productivity, product quality and customer satisfaction (Northrop,2002), thus addressing the problems aforementioned.

In general, the characteristic that distinguishes SPL from previous efforts in software development is predictive reuse versus opportunistic software reuse. Rather than put general software components into a library in hopes that opportunities for reuse will arise, SPL only call for software artifacts to be created when reuse is predicted in one or more products in a well defined product line (Krueger,2006).

With the growing acceptance of SPL by industry (Kang et al.,2007;Mansell,2006;Northrop,

(22)

1.1. MOTIVATION

2002;Pohl and Metzger,2006;Weiss,2008), effective and efficient quality assurance techniques and mechanisms are required. Therefore, testing, the most important and widely applied quality assurance mechanism (Crnkovic,2002), deserves attention. The SPL approach has a deep effect on the overall development process, which in turn calls for changes in the testing processes as parts of the overall process. Testing in software product lines is hence the focus of this dissertation, which reports on our work of investigating the state-of-the-art in this field and providing a systematic process for testing, in order to maximize the benefits of systematic reuse achieved by the SPL approach.

This Chapter contextualizes the focus of this dissertation and starts by presenting its motivation in Section1.1and a clearer definition of the problem in Section1.2. An overview of the proposed solution is presented in Section ??. Section1.3describes some related aspects that are not directly addressed by this work. Finnaly, Section1.5sketches the remainder structure of this dissertation.

1.1 Motivation

The software industry is increasingly faced with a growing demand for customized products, in order to meet a wide variety of customer specific needs. It has led to increase in the product diversity the companies have to deal with. If scale and time were considered, throughout the years, companies that have diverse implementations might probably re-develop assets that had already been developed. As a consequence, companies waste workforce and budget.

SPL is in frame with this urging need of software companies, which requires more and more meeting their customer requirements and handling with a growing diversity of products. The SPL practice, offers a systematic reuse approach which is focused on improving the development time-to-market, cost, quality, portfolio scale and scope and other business drivers, such as customer satisfaction (Northrop and Clements,2007). An important principle behind SPL is that, among the different needs, i.e. diverse projects implementations, there might be large amounts of common parts, which means a potential for high levels of reuse of development efforts between different projects.

Organizations have achieved these improvements when working in the SPL approach, as can be seen in (Weiss,2008). Companies of all types and sizes have discovered that a product line strategy, when skillfully implemented, can produce many benefits, and ultimately give the organizations a competitive edge (Northrop and Clements,2007).

While in the economic respect, SPL promises to return many benefits, at the same time

(23)

1.1. MOTIVATION

there are many aspects that need to be addressed to make the economic expectation come true.

There are considerable barriers to widespread product line practice. Even though the maturity of techniques and mechanisms for implementing SPL is considerable, additional problems are faced when adapting these practices to a specific organizational context (Mansell,2006).

Developing software for a product line is extremely complex due to multiple intertwined products, features and production deadlines. Moreover, the two phases of SPL development, consisting of core asset and product development requires related but distinct treatments. Such treatment does not stop at development but should extend also to testing (Kang et al.,2007).

There is a lack of an overall reasoning about testing in software product lines, regarding standard methodologies that can be generalize testing practices to varying scenarios and organizations (Lamancha et al.,2009) as well as techniques that address the problems directly rising from scale, reuse and variability (Tevanlinna et al.,2004).

Testing is an essential activity in software engineering. It is an important process that is performed to support quality assurance (Bertolino,2007;Harrold,2000). Product quality is surely a business driver for SPL practices. However, testing is still a very labor-intensive task, and effective testing requires a large number of tests to be handled. It is an expensive activity, which may consume up to 50 percent (or even more in some projects) of software development costs (Kolb and Muthig,2003). The same holds true in the SPL context (Kauppinen and Taina, 2003;McGregor,2001b;Tevanlinna et al.,2004).

It is important to point out that testing has become more critical and complex for product lines since quality issues in an artifact can have high impact on the numerous products in the product line that depends on it (Kolb and Muthig,2003). It may indicate that the effort required to testing in SPL is higher than in traditional software development, since several instances will use the same artifact, but we cannot guarantee that the artifact tested in one instance will necessarily behave accordingly in all instances. It is rather necessary to assess the quality of all derived products that will use a specific asset, instead of assessing the quality in only one product. In the context of a SPL project, which encompasses several artifacts, it is reinforced the seek for strategies that attempt to reduce such an effort.

In this context, a testing process contributes to a large extent to the success or failure of a product line effort and the potential benefits of product lines can be lost if the testing process does not take into account the issues specific to product lines. For that, there is a growing interest, by the SPL community, in developing effective and efficient methods, techniques and processes to handle testing in the product line context (Kang et al.,2007;Tevanlinna et al., 2004). These should considerate the both phases of SPL development as well as its interaction

(24)

1.2. SCOPE

with the other disciplines that comprise the SPL development life cycle, thus covering a gap of the existing approaches, which usually do not handle testing in the whole life cycle, but rather address specific points. Moreover, the systematic mapping study conducted in order to establish a basis for this study, as will be further detailed in Chapter3, pointed out some urging topics, regarding processes for SPL testing, which deserve special attention, such as quality attributes testing considering variations in quality levels among products, management of traceability between development and test artifacts, and the management of variability. These are topics which intends to compose an effective process focusing on reducing the effort required by testing activities, as discussed above.

Therefore, the focus of this dissertation is to provide a systematic process for testing in product lines, in order to maximize the benefits of systematic reuse and thus reduce the overall effort in testing activities in the SPL context. Both software product lines and software testing, as well as the relationship among these practices are further discussed in Chapter2.

1.2 Scope

Motivated by the challenges presented in the previous section, the goal of the work described in this dissertation can be stated as follows:

This work defines a testing process for software product lines which focuses on effort reduction through the systematic reuse of test artifacts by exploiting commonalities and managing variability among products. The process guides the SPL testing activities by providing tasks, inputs, outputs, roles and guidelines for test assets development and management.

In order to achieve the goal stated above, the RiPLE-TE - RiSE Product Line Engineering Testing Process is proposed. This process was proposed with basis in a mapping study on software product lines testing, which will be presented in Chapter 3. Different from other approaches in the literature, which only focuses on specific aspects on testing, e.g. considering integration testing or system testing (Kang et al.,2007;Reuys et al.,2006), this process works considering the both SPL phases, core asset and product development, in which testing levels are distributed. Moreover, the relationship with other life cycle disciplines is represented.

Next it is presented the context in which this work is inserted.

(25)

1.2. SCOPE

1.2.1 Context

This dissertation is part of the RiSE Labs¹(Almeida et al.,2004), formerly called RiSE Project, whose goal is to develop a robust framework for software reuse in order to enable the adoption of a reuse program. RiSE Labs is influenced by a series of areas, such as software measurement, architecture, quality, environments and tools, and so on, in order to achieve its goal. The influence areas are depicted in Figure1.1.

Figure 1.1 RiSE Labs influences.

Based on these areas, the RiSE Labs is divided in several different projects related to software reuse, as shown in Figure1.2:

• RiSE Framework: It involves reuse processes (Almeida et al.,2005), component certifi- cation (Alvaro et al.,2006) and reuse adoption and adaptation processes (Garcia et al., 2008).

• RiSE Tools: Research focused on software reuse tools, such as the Admire Environment (Mascena et al.,2006), the Basic Asset Retrieval Tool (B.A.R.T) (Santos et al.,2006),

1

(26)

1.2. SCOPE

Figure 1.2 RiSE Labs projects.

which was enhanced with folksonomy mechanisms (Vanderlei et al.,2007), semantic layer (Durão,2008), facets (Mendes,2008) and data mining (Martins et al.,2008), the Legacy InFormation retrieval Tool (LIFT) (Brito et al., 2008), the Reuse Repository System (CORE) (Melo et al.,2008), and the Tool for Domain Analysis (ToolDAy) (Lisboa,2008;

Lisboa et al.,2007).

• RiPLE: Stands for RiSE Product Line Engineering Process and aims at developing a methodology for Software Product Lines, composed of scoping (Moraes,2010), requirements engineering (Neiva,2009), design, implementation, test (Neto,2010), and evolution management (Oliveira,2009).

• SOPLE: Development of a methodology for Software Product Lines based on services, with some idea of the RiPLE (Medeiros et al.,2009).

• BTT: Research focused on tools for detection of duplicated change requests (Cavalcanti, 2009;Cavalcanti et al.,2009).

• MATRIX: Investigates the area of measurement in reuse and its impact in quality and productivity.

• Exploratory Research: Investigates new research directions in software engineering and its impact on reuse.

(27)

1.3. OUT OF SCOPE

• CX-Ray: Focused on understanding with empirical data of C.E.S.A.R², its processes and practices in software development, including reuse.

1.3 Out of Scope

This work is part of a broader context, as mentioned in the previous Section, in which a complete framework for software product lines development has been designed. Thus, some SPL topics are not described in this work, and some specific issues regarding testing are not considered in the scope of this dissertation. The following issues are not discussed in this work:

• Regression testing - In software development, many products go through several releases, and can be ported to many platforms. Every line of code written or modified offers opportunity for defects to creep in. Regression testing is the way to catch a large class of these bugs quickly and efficiently. It focuses on ensuring that everything that used to work still works after evolution or bug fixing activities. This continues to hold in case of SPL, strictly due to the nature of evolution of core assets. Even so, regression testing is not the focus here, since another member of our Research Group worked towards defining an approach for regression testing in the SPL context (Neto,2010). In the future this work will combine with that approach in order to have defined a complete process for testing for SPL.

• Correctitude of test cases - In line with the previous point, we believe that a meticulous analysis on existing techniques of correctitude analysis of documents, thereby involving test cases, should be performed. It has been proved that earlier semantic and syntactic corrections of documents can save much effort, because of the capability of finding and correcting errors earlier in the development cycle is cheaper that postponing such a task.

It is even more suitable in the context of SPL Testing, since reusable artifacts play the essential role of such a process. Thereby, avoiding errors earlier can save considerable effort along the time. However, we have not find related research reporting the use of tools and/or methodologies related to such a topic, i.e. we have no evidence on the effectiveness of this technique. In addition, we believe that additional effort that would be required to analyze this strategy would not be worthwhile to this work at this moment.

2http://www.cesar.org.br/

(28)

1.4. STATEMENT OF CONTRIBUTIONS

1.4 Statement of Contributions

As a result of the work presented in this dissertation, a list of contributions may be enumerated:

• The realization of a systematic mapping study on software product lines testing, which can provide research community with the state-of-the-art in the field, comprising current and relevant information, extracted from a formalized evaluation, analysis and synthesis process.

• The design of the RiPLE-TE process, developed within the context of the RiSE Frame- work for Software Product Line Engineering (RiPLE), which encompasses activities for the test discipline. The RiPLE-TE provides a systematic way to handle testing in SPL projects. It includes variability management concerns, a key point in the software product lines engineering.

• The definition, planning, operation and analysis of an experimental study, aimed at evaluating the proposed process. It was conducted once and then replicated in a slightly different environment in order to have extensive feedbacks on the adoption of the process through observing their outcomes. It was set up and thus documented in a way that enable opportunities for replication.

In addition, the following paper, reporting on the mapping study we conducted was accepted by Elsevier Information and Software Technology Journal (IST):

• Silveira Neto, P. A. M., Machado, I. C., McGregor, J. D., Almeida, E. S. and Meira, S. R.

L. Software Product Lines Testing: A Systematic Mapping Study. Elsevier Information and Software Technology Journal, 2011. To appear.

The same amount of work was performed by the first two authors, wherein the choose of the order of authoring does not represent the importance of their participation.

The following paper was also accepted for publication:

• Silveira Neto, P. A. M., Machado, I. C., Cavalcanti, Y. C., Almeida, E. S., Garcia, V. C. and Meira, S. R. L. A Regression Testing Approach for Software Product Lines Architectures. Brazilian Symposium on Software Components, Architectures and Reuse, in conjunction with Brazilian Conference on Software: Theory and Practice. Salvador-BA, Brazil. 2010 To appear.

(29)

1.5. ORGANIZATION OF THIS THESIS

1.5 Organization of this Thesis

The remainder of this dissertation is organized as follows:

• Chapter2 reviews the essential topics used throughout this work: Software Product Lines and Software Testing. An initial discussion on the relation between Testing and SPL is also presented.

• Chapter3 presents a mapping study on software product line testing, performed with the goal of map out the existing approaches towards understanding the state of the art and practice in this field, serving as a basis for our research work.

• Chapter4 describes the proposed process for testing in software product lines, presenting the roles associated, activities and disciplines involved, and the key concepts of the process.

• Chapter5 describes the experiments performed in order to evaluate the proposed process.

It details the purpose of the experiments and discuss the context in which they took place, besides reporting on their planning, operation, analysis, interpretation and packaging.

• Chapter6 provides a set of conclusions based on the this work, discussing our contributions and limitations of the process, presents the related work, and outline directions for future work.

• AppendixA presents the information sources from where primary studies were extracted, to serve as basis for the mapping study analysis, as reported in Chapter3. It also brings the scores from the analysis, based on the established quality criteria.

• AppendixB details the material used in the experimental studies, reported in Chapter5.

(30)

2

Foundations on Software Product Lines and Testing

Software is everywhere. It comes in a large range of devices, not only as desktops or laptops, its commonplace, but also in home electronics, airplanes, automobiles, security systems, medical and business environment, and many other situations and devices. Human society depend heavily on computers, and consequently on software, in almost every situation, such as their transportation systems, commerce, governance, utilities and more. Software increasingly becomes the key asset for modern, competitive products. No matter how simple or complex, no matter how large or small, there is hardly any modern product without software (Linden et al., 2007).

In face of the urging characteristic of software systems, which are becoming increasingly intensive and complex, with an even increasing need of products diversity associated to the market needs of developing software in shorter time and at a lower cost, Software Product Lines (SPL) is emerging as a practical and important paradigm in software development (Northrop, 2002). Based on the systematic and planned reuse of previous development efforts among a set of similar products, the SPL approach enables organizations not only to reduce development and maintenance costs but also to achieve impressive gains in productivity and time-to-market.

Yet, it is based on the observation that most of new software systems are similar to already existing ones. They are either different versions or next releases of the same baseline product (Trendowicz and Punter,2003). However, SPL quality demands are of high importance and continuously rise, specially due to the its characteristic of supporting to large-scale systems, that requires high confidence products. Thus, testing, as still the most effective way for quality assurance, requires special attention, since the literature reports on that it is more critical and complex for product lines than for traditional single software systems (Kolb and Muthig,2003).

(31)

2.1. SOFTWARE PRODUCT LINES (SPL)

This Chapter describes background information relevant to this research. It is organized as follows: Section2.1introduces concepts regarding Software Product Lines; Software Testing is introduced in Section2.2; Section2.3sketches the relationship between the previously addressed concepts; Chapter Summary is addressed in Section2.4.

2.1 Software Product Lines (SPL)

The software industry has continuously been challenged to improve its engineering practice aiming at delivering products in a faster and more reliable way, since the way that goods are produced has changed significantly in the course of time. Formerly goods were handcrafted for individual customers. Gradually, the number of people who could afford to buy various kinds of products increased (Pohl et al.,2005). Furthermore, it is important to point out that no two customers are identical. Each has unique requirements that can only be completely satisfied by a custom solution. However, software organizations must target large homogeneous markets in order to keep development costs within reason, essentially treating their customers as if they are identical.

This reason demonstrates the consequent competitiveness software development industry has experienced. It has increasingly become a concern for companies of all sizes and in all markets to develop strategies to meet the needs stated above.

A growing number of software development organizations are adopting strategies that emphasize proactive reuse, interchangeable components, and multi-product planning cycles to construct high-quality products faster and cheaper (McGregor et al.,2002), thereby meeting customer needs. The SPL engineering seamlessly address these strategies.

SPL engineering is emerging as a viable and important development paradigm allowing companies to realize improvements in time to market, cost, productivity, quality, and other business drivers (Denger and Kolb,2006;Pohl et al.,2005). Software product lines can also enable rapid market entry and flexible response, and provide a capability for mass customization (Northrop, 2002). By definition, a SPL is a set of software-intensive systems that share a common, managed set of features satisfying the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way (Clements and Northrop,2001).

In SPL, products are developed from a common set of assets, in contrast to being developed separately, from scratch, or in an arbitrary fashion (Clements and Northrop,2001). In addition, in constrast with what is customary, “design reuse” in SPL does not simply mean taking an

(32)

existing design, copying it and modifying it to some particular need, but rather the development of a model (or set of models) that can be reused in a disciplined fashion. SPL techniques explicitly consolidate and capitalize on commonality throughout the product line. They formally manage and control the variations among the products in the product line. New solutions are developed by assembling partial solutions and/or by configuring generic ones. Only the unique features of each solution, represented by product variations, must be specified because the common ones can be safely assumed. It eliminates duplication of effort in the engineering processes.

Yet, it has gained increasing attention by industry over recent years (Linden et al.,2007). It has proven successful in a variety of settings, including large and small organizations in business, industry, and governmental sectors, and across a variety of domains (Käkölä and Dueñas,2006), ranging from mobile phones to satellite ground station systems (Weiss,2008).

As real world motivation, organizations use product line practices to (Northrop and Clements, 2007): achieve large scale productivity gains, improve time to market, maintain market presence, sustain unprecedented growth, achieve greater market agility, compensate for an inability to hire, enable mass customization, get control of diverse product configurations, improve product quality, increase customer satisfaction, increase predictability of cost, schedule, and quality.

Ideas common to most successful product line efforts encompasses, besides exploring commonality among products to strategically reuse software artifacts, also the adoption of architecture -centric development and a two-tiered organizational structure (McGregor et al., 2002).

The SPL development begins by identifying the domain, deciding a set of products comprising the product line and then proceeds by identifying what requirements are common to all products (commonality), and what differentiate them (variability). Commonality and variability concepts are key prerequisites for software product lines (Klaus Pohl and van der Linden,2005).

Architectureis also a key concept of software product line engineering. It determines how products are derived efficiently from the core assets. To allow the derivation of several different products, a product line architecture has to deal with variation. The architecture’s support for variation determines the scope of the product line (Rommes and America,2006), so that it can be configured to match a diverse range of requirements.

Moreover, SPL requires an organization to involve two types of development activities, one responsible for developing reusable assets (core asset development) and the other responsible for the development of products that use those assets (product development). These, taken together with management, are the essential activities underlying SPL.

(33)

2.1.1 SPL Essential Activities

As mentioned, software product lines engineering involves three essential activities, as shown in Figure2.1(Clements and Northrop,2001): core asset development, product development, and management. Each of these activities is essential, as is the blending of all three. The rotating arrows indicate not only that core assets are used to develop product but also that revisions of existing core assets or even new core assets might, and most often do, evolve out of product development. There is a strong feedback loop between the core assets and the products.

Moreover, strong management at multiple levels is needed throughout. Management oversees core asset and product development. Management orchestrates all activities and processes needed to make the three essential activities work together. Each activity is next detailed:

Figure 2.1 Essential Product Line Activities

Core asset development

Also known as domain engineering, it forms the basis for the software product line. The goal of this activity is to define the commonality and the variability of the product line, and thus for establishing the reusable artifacts and then a production capability for products (Pohl et al.,2005). It also involves the evolution of the assets in response to product feedback, new

(34)

market needs, and so on (Clements and Northrop,2001). Figure2.2illustrates the core asset development activity along with its outputs and influential contextual factors.

The activity of core asset development is iterative. The rotating arrows in Figure2.2suggest that there is no one-way causal relationship from inputs to outputs, they rather affect each other.

For example, expanding the product line scope (one of the outputs) may admit whole new classes of systems to examine as possible sources of legacy assets (part of the context). Correspondingly, a production constraint may lead to restrictions on the product line architecture (an output).

This restriction, in turn, will determine which preexisting assets (another contextual factor) are candidates for reuse or mining (Northrop,2002).

Core assets include, but are not limited to, the architecture and its documentation, specifica- tions, software components, tools such as component or application generators, performance models, schedules, budgets, test plans, test cases, work plans, and process descriptions (Clements and Northrop,2001). Although it may be possible to create core assets that can be used across the products without any adaptations, in many cases some adaptations are required to make core assets usable in the broader context of a product line. Variation mechanisms used in core assets help to control the required adaptations and to support the differences among the software products (Bachmann and Clements,2005). These adaptations should be planned before development and made easy for the product development team to use without putting at risk existing properties of the core assets.

Figure 2.2 Core Asset Development

(35)

Product development

Also known as application engineering, this activity creates individual products by reusing the core assets, gives feedback to core asset development, and evolves the products. Figure2.3 illustrates the product development activity along with its outputs and influential contextual factors.

Product development uses the core assets, in accordance with the production plan, to produce products that meet their respective requirements, as defined in the product line scope, all artifacts derived from core assets development, as previously shown in Figure2.2.

As in Figure 2.2, the rotating arrows in Figure 2.3 indicate iteration and involved rela- tionships. For example, the existence and availability of a particular product may well affect the requirements for a subsequent product. As another example, building a product that has previously unrecognized commonality with another product already in the product line will create pressure to update the core assets and provide a basis for exploiting that commonality for future products (Northrop,2002). Moreover, this activity has an obligation to give feedback on any problems or deficiencies encountered with the core assets, in order to keep the core asset base in accordance with the products.

Figure 2.3 Product Development

Management

It includes technical and organizational management, where technical management is responsible for requirement control and the coordination between core asset and product development. Not only technical aspects, here represented by the development of core assets and products, are

(36)

2.2. SOFTWARE TESTING

considered when developing SPL, but managerial and organizational activities as well.

The common set of assets and the plan for how they are used to build product do not just materialize without planning, and they certainly do not come free. They require organizational foresight, investment, planning and direction. They require strategic thinking that looks beyond a single product. The disciplined use of the assets to build products does not just happen either.

Management must direct, track, and enforce the use of the assets. Software product lines are as much about business practices as they are about technical practices (Clements and Northrop, 2001).

Although organizations are different, in terms of nature of their products, market or mission, business goals, organizational structure, culture and policies, software process disciplines, and so on, these essential activities apply in every situation, since they represent the highest level of generality which involves the most important aspects regarding SPL development. In general, organizations accomplish this division of responsibility in a variety of ways. Some organizations have teams dedicated to each role and others use the same people for both (McGregor et al., 2002). It depends on the strategy, budget, staff availability, among other aspects. In fact, it is valid to mention that there is no “first" activity, i.e. in some contexts, existing products are mined for core assets, whereas in others, core assets may be developed or procured for future use.

2.2 Software Testing

The role of software which is, more and more, part of our lives, both economically and socially, and is used more often to perform critical tasks, has imposed pressure for software professionals to focus on quality issues, since it is necessary to avoid that people, i.e. customers, have experience with software that do not work as expected. It can be reachable by building reliable software.

Although many factors affect the engineering of reliable software, including, of course, careful design and sound process management, testing is the primary method that the industry uses to evaluate software under development (Ammann and Offutt,2008).

Testing is an important process that is performed to support quality assurance (Harrold, 2000). According to Software Engineering Institute (SEI)¹ in its report on software testing (McGregor,2001b), testing is one approach to validating and verifying the artifacts produced in software development. A more detailed definition can be stated as testing is designed to make sure computer code does what it was designed to do and that it does not do anything unintended.

1SEI - Software Engineering Institute at Carnegie Melon University - www.sei.cmu.edu.

(37)

Software should be predictable and consistent, offering no surprises to users (Myers and Sandler, 2004). Moreover, testing helps us to measure the quality of software in terms of the number of defects found, the tests performed, and the system covered by the tests. It can be performed for both the functional and non-functional software requirements and characteristics.

Testing activities support quality assurance by gathering information about the nature of the software being studied. These activities consist of designing test cases, executing the software with those test cases, and examining the results produced by those executions. They, together, can reduce the risk of failure in the real environment.

Since the focus is to deliver high-quality products to customers, software researchers realized that it was important to integrate testing activities within the context of a software development process (Burnstein,2002), including testing into the whole software life cycle so that confidence in the software can be acquired.

Following the role of process in software testing is addressed and then process models and techniques are described.

2.2.1 Testing Processes

In a high-level definition, a process is a set of actions, observations, and decisions taken to achieve some desired outcome, in which some activities can happen in parallel, and some are sequential. The activities are related by the desired outcomes but do not necessarily have techniques or required skills in common (Black,2003). If we instantiate such a definition for the context of software engineering, we can consider a process as the set of methods, practices, standards, documents, activities, policies, and procedures that software engineers use to develop and maintain a software system and its associated artifacts, such as project and test plans, design documents, code, and manuals (Burnstein,2002).

Thus, if we think about a test process, we can elaborate on this definition by considering three main aspects: economic, technical and managerial.

Economic aspects are related to the reality that resources and time are available to the testing group on a limited basis, as described byBurnstein(2002). In many cases, complete testing is not feasible due to these economic contraints. Testing coverage criteria are used to decide which test inputs to use so that testing continues to be an effective approach. A little bit more on testing coverage is addressed in Section2.2.2.

Technical aspects of testing include the techniques, methods, measurements, and tools used to guarantee that the software under test is as defect-free and reliable as possible for the conditions and constraints under which it must operate.

(38)

Regarding managerial aspect, since we advocate that testing is a process, a process must be managed. In short, it means that an organizational policy for testing must be defined and documented, which include their procedures and steps. The testing activity must be planned, and the process should have associated quantifiable goals that can be measured and monitored.

A test process should specify and separate the testing characteristics, which represents the why we testquestion; the testing stages, delineating when we test; and also testing techniques, responsible for answering the question how we test. It enables to organize the activity of revealing defects, confirming conformance to specified requirements and evaluating quality against criteria such as business risk and business confidence. A well planned, designed, built and executed test process can result in delivery of projects on time, to cost, meet the required quality and deliver the required scope more often and more predictably. The expected business benefits and return on investment from the project can be achieved through applying a test process.

It clearly denotes the importance of such a process adoption by an organization, which in short allows a team to better execute its testing projects, working in a guided environment with timely, accurate, credible information (Black,2003).

A test process depends upon the organizational needs. We can not consider a standard process to be applicable to every one. We can only state that a process should accomplish the three aspects as aforementioned and furthermore it should be in constant optmization, being able to absorb improvements from both internal and external stakeholders. Besides, personnel should clearly understand their roles and responsibilities. They should be trained in the process, hence considering these elements we have just described. Next we bring on some information regarding the role of test engineer, which is responsible for dealing with test in an organization and then regarding the artifacts used and/or generated by test processes.

Activities of a Test Engineer

According to Ammann and Offutt(2008), a test engineer is a professional who is in charge of one or more technical test activities. The tasks and responsibilities include designing test inputs, producing test case values, running test scripts, analyzing results, and reporting results to developers and managers.

A sample flow performed by a test engineer consists of firstly design tests by creating test requirements. These requirements are then transformed into actual values and scripts that are ready for execution. These executable tests are run against the software, and the results are evaluated to determine if the tests reveal a non-compliance in the software.

(39)

These activities may be carried out by one person or by several, and the process is monitored by a test manager. A test manager is in charge of one or more test engineers. Test managers set test policies and processes, interact with other managers on the project, and otherwise help the engineers do their work (Ammann and Offutt,2008). Moreover, with every project the test project manager learns new concepts that will improve the test processes, which should undergo continuous process improvement.

In many organizations, the developer assumes the role of test engineers, being responsible for both coding and testing tasks. In other, in which testing activities are separated from the development ones, the role of test engineer is more visible. For the purpose of this work, we use the term tester to refer to every person who works with testing activities, which in a broader view can be included in the group of test engineers. Further in Section4.2we expand on the roles and responsibilities of test engineers considering testing in a SPL.

Testing Artifacts

According to IEEE Standard for Software Test Documentation (IEEE,1998), the following are considered the basic artifacts to be applied in the testing activity, that is basically composed by sets of test cycles: test plan, test design specification, test case specification, test procedure specificationand test report. These are considered the most relevant artifacts and they all are applicable to any context, although other artifacts can be produced as well.

A good practice, specially when striving for reducing effort, indicates that tests must be repeatable and reusable, thus testing artifacts are to be produced considering such aspects. The expertise gathered in various projects should be documented so that the knowledge can be reused in new projects.

2.2.2 Testing Process Models

Traditionally, efforts to improve quality have centered around the end of the product development cycle by emphasizing the detection and correction of defects (Kshirasagar Naik,2008). It is the common practice of the Waterfall software development process, in which testing concerns are postponed to the end of the development cycle, after the implementation has started, or even after it has ended.

Notwithstanding, by waiting until this late in the process, testing ends up being compressed, in which not enough time and budget resources remain, problems with previous stages have been solved by taking time and money from testing phase, and testers do not have enough time

(40)

to plan for testing. Instead of planning and designing tests, the developers have time only to run tests, usually without being guided by a process, but rather in an ad hoc fashion (Ammann and Offutt,2008).

An approach for testing to enhancing quality should start at the beginning of a project, long before any program code has been written, encompassing all phases of a product development process. Testing should occur at different places and times during the development life cycle - from the requirements analysis to the final delivery of the product to the customer. The requirements documentation is verified first; then, in the later stages of the project, testing can concentrate on ensuring the quality of the application code. Expensive reworking is minimized by eliminating requirements-related defects early in the project’s life, prior to detailed design or coding work.

The idea of integrating testing activities into the earlier software life cycle phases is illustrated by the V-model (Figure2.4). The V-model gives equal weight to development and testing rather than treating testing as an afterthought (Goldsmith and Graham,2002).

Figure 2.4 Testing in the V-model.

There are other models for testing process that can be combined with the ones aforementioned It is the case of Spiral Testing, which work with the idea of software increments, and prototypes evolving into applications (Mathur,2009). In this model, testing activities evolve over time and with the prototype, increasing their sophistication as the project evolves. It can thus be applied to any incremental development process.

(41)

Testing Levels

As mentioned above, in a quality-driven development process, testing activities should be performed along the whole life cycle, considering all phases, in order to organize the testing process as well as to find major problems earlier in the development and thus avoiding resources waste. Early identification of defects is by far the best means of reducing their ultimate cost.

The testing activities mentioned are then expressed as testing levels. A different level of testing accompanies each distinct software development activity, consequently the information for each test level is typically derived from the associated development activity.

The idea behind splitting testing into levels is to build code and test it in pieces and gradually put together into larger and larger portions, in order to avoid surprises when the entire product is linked together (Patton,2005).

The common testing levels are following described:

• Unit Testing: It is designed to assess the units produced by the implementation phase and is the “lowest" level of testing. Unit testing has as goal the capability to ensure that each individual software unit is functioning according to its specification (Burnstein, 2002). Most companies make unit testing the responsibility of the programmer. It is straightforward to package unit tests together with the corresponding code through the use of tools such as JUnit for Java classes (Ammann and Offutt,2008). Each unit test must run independently of all other units as well as unit tests must be able to run in any order.

In addition, one test must not depend on some side effect caused by a previous test, e.g. a member variable being left in a certain state.

• Module Testing: It is designed to assess individual modules in isolation, including how the component units interact with each other and their associated data structures. As with unit testing, most software development organizations make module testing the responsibility of the programmer (Ammann and Offutt,2008). It is possible to describe a process merging module and unit testing objectives in a single level.

• Integration Testing: As the units and/or modules are tested and the low-level bugs are fixed, they are then integrated and integration testing occurs. It is designed to assess whether the interfaces between units (or modules) in a given subsystem have consistent assumptions and communicate correctly. Integration testing must assume that modules work correctly (Ammann and Offutt,2008).

• System Testing: Its purpose is to compare the system to its original objectives. It assumes

(42)

that the pieces work individually, and asks if the system works as a whole (Myers and Sandler,2004). This level of testing usually looks for design and specification problems, identified at Analysis phase. It is a very expensive place to find lower-level faults and is usually not done by the programmers, but by a separate testing team (Ammann and Offutt, 2008).

• Acceptance Testing: It is designed to determine whether the completed software in fact meets customers’ requirements, gained through Requirements Analysis phase. Acceptance testing probes whether the software does what the users want. It must involve users or other individuals who have strong domain knowledge (Ammann and Offutt,2008).

Testing Coverage

In an ideal world, we would want to test every possible permutation of a program. In most cases, however, this simply is not possible (Ammann and Offutt,2008;Craig and Jaskiel,2002;

Myers and Sandler,2004;Patton,2005). The features and attributes of a simple application may result in millions of permutations that could potentially be developed into test cases. Even a seemingly simple program can have hundreds or thousands of possible input and output combinations. Obviously, creating test cases for all of these possibilities is impractical. Even if a large number of test cases are created, they generally still represent only a tiny fraction of the possible combinations. Several other combinations may still exist. As said byCraig and Jaskiel (2002), in most cases, what is tested in a system is much more important than how much it is tested.

Since we cannot test with all inputs, it is necessary to determine a testing coverage criteria.

Coverage criteria are used to decide the what aforementioned, i.e. which test inputs to use, providing suitable stopping rules for testing. A formal coverage criteria give test engineers ways to decide what test inputs to use during testing, making it more likely that the tester will find problems in the program and providing greater assurance that the software is of high quality and reliability (Ammann and Offutt,2008).

Coverage criteria can be defined in terms of test requirements. These can be described with respect to a variety of software artifacts, including the source code, design components, specification modeling elements, or even descriptions of the input space. Testers have to understand the testing coverage required by a project, and it should be detailed in the test plan document. Furthermore, coverage criteria act as basis for test case design. As an example, in some projects there may be contractual agreements that list all functional requirements to be

Ivan do Carmo Machado

“RiPLE-TE: A Software Product Lines Testing Process”

Ivan do Carmo Machado

Universidade Federal de Pernambuco Centro de Informática

Pós-graduação em Ciência da Computação

Ivan do Carmo Machado

“RiPLE-TE: A Software Product Lines Testing Process”

RECIFE, AUGUST/2010

Acknowledgements

Agradecimentos

Resumo

Abstract

Table of Contents

List of Figures

List of Tables

Acronyms

1

Introduction

1.1 Motivation

1.2 Scope

1.2.1 Context

1.3 Out of Scope

1.4 Statement of Contributions

1.5 Organization of this Thesis

2

Foundations on Software Product Lines and Testing

2.1 Software Product Lines (SPL)

2.1.1 SPL Essential Activities

2.2 Software Testing

2.2.1 Testing Processes

2.2.2 Testing Process Models