AIP Conference Proceedings 2016, 020032 (2018); https://doi.org/10.1063/1.5055434 2016, 020032 © 2018 Author(s).
Regression testing – A protocol for
systematic literature review
Cite as: AIP Conference Proceedings 2016, 020032 (2018); https://doi.org/10.1063/1.5055434 Published Online: 27 September 2018
Bakr Ba-Quttayyan, Haslina Mohd, and Fauziah Baharom
ARTICLES YOU MAY BE INTERESTED IN
Preface: The 3rd International Conference on Applied Science and Technology (ICAST’18) AIP Conference Proceedings 2016, 010001 (2018); https://doi.org/10.1063/1.5055402
Identification of criteria affecting software project monitoring task of Agile Kanban method AIP Conference Proceedings 2016, 020021 (2018); https://doi.org/10.1063/1.5055423
A process for developing an instrument to measure the persuasion perspectives of parents using PMCOM app
Regression Testing – A Protocol for Systematic Literature
Review
Bakr Ba-Quttayyan
1, a)Haslina Mohd
1, b)and Fauziah Baharom
1, c)1Human-Centered Computing Research (HCCR) Lab, School of Computing, College of Arts and Sciences, Universiti Utara Malaysia, 06010, Sintok, Kedah, Malaysia
a)Corresponding author: [email protected] b)[email protected]
Abstract. Regression testing is a costly but important maintenance activity which performed to revalidate modified
software to make sure that changes did not adversely affect software behavior. Several studies have been carried out regarding regression testing domain in the literature. However, these studies need to be classified, summarized and ordered in a systematic manner to spot on the current state of the art of regression testing field. Therefore, this paper will carry out a systematic literature review (SLR) protocol to illustrate the process of preparation and undertaking a systematic review later. In this protocol, a manual search has been conducted to obtain related studies from four various sources where those studies were classified under journal, conference proceeding, and book chapter categories within specific criteria. Consequently, the initial search resulted in 1261 unfiltered studies, which then decreased according to certain predefined criteria up to 246 selected relevant studies. As a result, this protocol will behave as a guideline towards conducting SLR in the near future.
INTRODUCTION
Software needs to be modified based on change demands mostly obtained from users within the aim of improving software functionality by adding new features, modifying existing features, or eliminating others. However, these changes naturally lead to introduce new faults [1,2], and validating software after these modifications to ensure there are no faults were introduced is what called a "regression testing" [3]. Such "regress", i.e.: failure or faults, is necessary to be uncovered after modifications to ensure the quality of modified system under test (SUT) [4]. Regression testing practice increased due to the growth in software demand on the software industry around the world. In software industry, studies indicate that more than 50% of software maintenance cost is related to testing activities. Moreover, regression testing activities alone costs up to 80% of those costs, i.e.: more than one-third of the software production total costs [5–9]. However, this process is getting costly, and time consumed especially when size and complexity of systems increased [10]. Increasing cost of regression testing leads the researchers and practitioners in this field looking for further ways not only to decrease the cost of regression testing, but also to increase its effectiveness and performance in terms of ensuring high quality achievements.
Formally, IEEE standard glossary of software engineering terminology defines regression testing as follows:
"Selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements." [6,11,12]. In other words, regression testing
can be defined as a "software maintenance task performed on a modified program to instill confidence that changes
are correct and have not adversely affected the unchanged portions of the program" [4]. According to [13], regression
testing problem is described as:
"Given program P, its modified version P', and test set T used previously to test P, find a way, making use of
In the past, regression testing performed as a type of retesting the modified software within the existing test suites. However, this way is a very costly and time consumed [1]. After that, tester start to optimize test suites by selecting test cases randomly, unfortunately this way is not effective in accordance with coverage and fault detection ability [15]. In 1981, the first systematic approach for selecting test cases to perform regression testing effectively was proposed by Fischer, Raji and Chruscicki [16,17], and this improvement establishes a new trend towards regression testing effectiveness called "selective retest approach" or "regression test selection approach" as others prefer. In 1997, another approach of regression testing was introduced by proposing [18] their technique, establishing by that a new regression testing direction, which specifically called "test case prioritization approach" [19].
Regression testing is an important part of software testing which specifically applied during software project development and maintenance [11,20], and it has many published studies in the literature. To the best of our knowledge, However, these studies need to be classified, summarized and ordered to spot on the current state of regression testing field, since there is no general systematic literature review has been conducted to highlight regression testing domain in terms of approaches, frameworks, models and tools. To address these issues, this paper presents a systematic literature review protocol that studying the regression testing systematically. The details of the Regression Testing Systematics Literature Review (SLR) protocols are discussed in detail in the next section. In addition, the results section highlights the initial findings of the SLR. This paper ends with a summary and future study directions of the regression testing.
METHOD
In order to conduct a systematic literature review in any software engineering discipline, Kitchenham et al. [21] provided comprehensive guidelines to facilitate this process and these guidelines were used in this review. Thus, our review process consists of three phases: planning, conducting and reporting the review. For the first stage, we formulated a set of research questions and prepared a review protocol to be a basis throughout data collection process. Next, we conducted a manual search through various sources and selected relevant papers based on the protocol. Accordingly, the selected papers were read thoroughly, and data were extracted in a tabular form as stated in data extraction section. Finally, we analyzed results and prepared them for reporting.
Research Questions. This review addresses two main questions and each question has sub-questions. The questions
were emphasized as:
1. How many regression testing articles published up to 30th June 2016? How many regression testing SLRs?
How many regression testing Reviews / Surveys? How many regression testing Primary Studies?
2. What are the highlighted core concepts of regression testing in these published articles? Approaches. Process. Frameworks. Models. Tools.
Search Process
The search process is the approach of how primary and secondary studies will be obtained. We conducted a manual search using specific terms in specific databases as following:
x Terms of Search. The terms that were used in this search are those related to regression testing keywords
generally, with the aim of including many published papers in this domain. Thus, the search string was formulated in general keywords as: [“Regression Testing” OR “Regression Test”].
x Search Databases. We conducted a manual search in various sources and our study were based on to the
following databases:
1. IEEE Xplore: <http://ieeexplore.ieee.org/Xplore/home.jsp>. 2. ACM digital library: <https://dl.acm.org/>.
3. Science Direct: <http://www.sciencedirect.com/>. 4. Wiley Online Library: <http://onlinelibrary.wiley.com/>.
Paper Selection
To select relevant studies throughout the search process, selection criteria must be defined. Table 1 presents our inclusion and exclusion criteria for this review.
TABLE 1. Paper Selection Criteria
In cl us ion Cri teri
a. Published articles in the following themes were included:9 Systematic Literature Reviews (SLRs) articles of Regression Test/Testing.
9 Review or Survey articles of Regression Test / Testing. 9 Primary Studies of Regression Test / Testing.
Ex clusio n Cr ite ria.
Articles with the following issues were excluded: 2
2 Articles have not addressed “Regression Test” or “Regression Testing” in the main title.
2 Articles have not written in English.
2 Not completed articles (only abstract or part of the article). 2 Workshop articles.
Data Extraction
We organized the data obtained from selected studies in a tabular form as emphasized in Table 2:
TABLE 2. Data Extraction Form
Item Description
1. Publication Details:
x ID Publication identifier. x Title The title of the study. x Author(s) The writers of the study.
x Publication year The year when the primary study was published.
x Publication Type Determine the type of publication (e.g.: journal, conference, etc.). x Source The source where paper found.
x Study Type Determine type of study (e.g.: primary, SLR, etc.).
2. Highlighted Concepts:
x Approaches The existing approaches used in regression testing.
x Process The different processes where used during regression testing. x Frameworks The existing frameworks applied for regression testing. x Models The existing models built for regression testing. x Tools The outcome or used tools during regression testing.
INITIAL RESULTS
The initial results of conducting this review were presented in two different levels: results of the conducted search process and results initial data extraction.
1. Search Process. Conducting the search process has been staged in three phases which emphasized as: x Initial Stage – the initial search performed through specified sources has obtained 1264 papers in general.
This amount of papers was filtered in the next phase.
x Title Exclusion Stage – in this stage, we excluded studies that did not demonstrate the specified keywords
in the main title. The obtained studies after this step minimized to 287 papers.
x Full-Text Exclusion – at the end of this phase, the full-text of the remaining 287 papers was read and a
further 41 studies were excluded by consensus. To be obvious, we found some papers were published in two incorporated databases, i.e.: IEEE and ACM, so this were considered and highlighted in our review. Table 3 presents final selected studies in relation with their sources.
TABLE 3. Search Process Summary of Relevant Articles
# Database Initial Search Second Stage Final Stage #Articles % #Articles % #Articles %
1. IEEE Xplore 174 13.79 129 44.95 124 50.41 2. ACM Digital Library 331 26.25 102 35.54 73 29.67 3. Science Direct 420 33.31 32 11.15 29 11.79 4. Wiley Online Library 336 26.65 24 8.36 20 8.13
Final Total 1261 100 287 100 246 100
The overall search process has illustrated in Fig. 1 as bellow.
FIGURE 1. Search Process Stages
2. Initial Data Extraction. In this review, we organized the selected studies in to forms: historical and publication type form. Historically, the first research paper has been published in regression testing was in 1981 where placed in ACM database. Table 4 presents the historical growth of publishing in regression testing till this review conducted. In this table, we can see that there is an increased growth of research in the last ten years. Moreover, most of these studies published in IEEE Xplore and ACM, while the rest in Science Direct and Wiley respectively.
TABLE 4. Papers Classification According to Publish Year Year IEEE Xplore ACM Science Direct Wiley Total
1981 - 1 - - 1 1989 1 - - - 1 1990 - - 1 1 2 1991 1 - - - 1 1993 3 1 - - 4 1994 3 - - - 3 1995 - - 1 - 1 1996 1 1 1 - 3 1997 1 1 - - 2 1998 3 3 1 - 7 1999 1 2 - - 3 2000 - 2 - - 2 2001 3 3 1 1 8 2002 2 3 1 - 6 2003 2 3 1 1 7
Year IEEE Xplore ACM Science Direct Wiley Total 2004 2 2 - - 4 2005 4 3 - 1 8 2006 1 3 1 1 6 2007 5 3 1 2 11 2008 4 9 - 1 14 2009 4 3 2 2 11 2010 14 2 1 1 18 2011 15 1 1 3 20 2012 12 4 3 1 20 2013 18 4 2 - 24 2014 6 11 3 1 21 2015 16 6 2 3 27 2016 2 2 6 1 11 Total 124 73 29 20 246
Furthermore, each selected study takes a form of journal, conference or book chapter type; while most of these studies takes the form of conference proceeding, only 3 of them were published as book chapters as well as they were published only in science direct database as illustrated in Table 5.
TABLE 5. Papers Classification According to Publish Type
Journals Conference Proceedings Book Chapters Total
IEEE Xplore 12 112 - 124 ACM 15 58 - 73 Science Direct 26 - 3 29 Wiley 20 - - 20 Total 73 170 3 246
CONCLUSION
Regression testing is a costly but necessary activity which performed to conform the changes made to software were not adversely affected its quality. In this paper, we conducted a protocol for a comprehensive systematic literature review to explore different areas in regression testing domain. As a result, we selected 246 studies out of 1261. Furthermore, initial findings were presented. The researchers currently continue analyzing and extracting data, and achieving our research questions will be the expected outcome of this review.
REFERENCES
1. S. Eghbali and L. Tahvildari, IEEE Trans. Softw. Eng. 42, 1178 (2016).
2. Q. Yi, Z. Yang, J. Liu, C. Zhao, and C. Wang, Proc. - Int. Conf. Softw. Eng. 1, 257 (2015).
3. H.K.N. Leung and L. White, in Proceedings. Conf. Softw. Maint. - 1989 (IEEE Comput. Soc. Press, 1989), pp. 60–69.
4. N. Chauhan, Software Testing: Principles and Practices (Oxford University Press, 2010).
5. 0.RZDOF]HZVNL0.UDZF]\N(/HZDĔVNDDQG:$EUDPRZLF]LQInt. Conf. Bus. Inf. Syst. (Springer, Cham, 2017), pp. 120–131.
6. R. Kazmi, D.N.A. Jawawi, R. Mohamad, and I. Ghani, ACM Comput. Surv. 50, 29:1 (2017). 7. E. Engström and P. Runeson, in (2010), pp. 3–16.
8. M. Rava and W.M.N. Wan-Kadir, Int. J. Softw. Eng. Its Appl. 10, 221 (2017).
9. C. Tao, B. Li, X. Sun, and Y. Zhou, in 2010 Asia Pacific Softw. Eng. Conf.(2010), pp. 290–299. 10. H. Do, in Adv. Comput. (Elsevier, 2016), pp. 53–77.
11. S. Nayak, C. Kumar, and S. Tripathi, Arab. J. Sci. Eng. 1 (2017). 12. I.S.C. IEEE Committee, IEEE Std 610.12-1990 169, 1 (1990).
13. G. Rothermel and M.J.J. Harrold, in Proc. 16th Int. Conf. Softw. Eng.(1994), pp. 201–210.
14. T.. Yu, X.. Qu, M.. Acharya, and G.. c Rothermel, Proc. - IEEE 6th Int. Conf. Softw. Testing, Verif. Validation, ICST 2013 292 (2013).
15. A. Janes, in 2017 IEEE Int. Conf. Softw. Testing, Verif. Valid. Work. (IEEE, 2017), pp. 38–39. 16. K. Fischer, F. Raji, and A. Chruscicki, in Proc. Natl. Telecommun. Conf. (1981), pp. 1–6. 17. G. Rothermel and M.J. Harrold, Empir. Softw. Eng. 2, 178 (1997).
18. W.E.E. Wong, J.R.R. Horgan, S. London, and H. Agrawal, in Proc. Eighth Int. Symp. Softw. Reliab. Eng.(1997), pp. 264–274.
19. D. Hao, L. Zhang, and H. Mei, Front. Comput. Sci. 10, 769 (2016).
20. M.J. Harrold, in Proc. Conf. Futur. Softw. Eng. - ICSE ’00 (ACM Press, New York, New York, USA, 2000), pp. 61–72.
21. S. Keele, others, and S.E. Group, Guidelines for Performing Systematic Literature Reviews in Software
