SEQUENCE AND SEQUENCE-LESS T-WAY
TEST SUITE GENERATION STRATEGY
BASED ON THE ELITIST FLOWER
POLLINATION ALGORITHM
ABDULLAH NASSER MOHAMMED
ABDULLAH
Doctor of Philosophy
SUPERVISOR’S DECLARATION
We hereby declare that I have checked this thesis and in our opinion, this thesis is adequate in terms of scope and quality for the award of the degree of Doctor of Philosophy (Computer Science).
_______________________________ (Supervisor’s Signature)
Full Name : PROR DR. KAMAL ZUHAIRI BIN ZAMLI Position : Professor
Date : 8 March 2018
_______________________________ (Co-supervisor’s Signature)
Full Name : DR. ABDULRAHMAN AHMED AL-SEWARI Position : Senior Lecturer
STUDENT’S DECLARATION
I hereby declare that the work in this thesis is based on my original work except for quotations and citation which have been duly acknowledged. I also declare that it has not been previously or concurrently submitted for any other degree at Universiti Malaysia Pahang or any other institutions.
_______________________________ (Student’s Signature)
Full Name : ABDULLAH NASSER MOHAMMED ABDULLAH ID Number : PCC14003
SEQUENCE AND SEQUENCE-LESS T-WAY TEST SUITE GENERATION STRATEGY BASED ON THE ELITIST FLOWER POLLINATION ALGORITHM
ABDULLAH NASSER MOHAMMED ABDULLAH
Thesis submitted in fulfillment of the requirements for the award of the degree of
Doctor of Philosophy
Faculty of Computer Systems and Software Engineering UNIVERSITI MALAYSIA PAHANG
ii
ACKNOWLEDGEMENT
I would like to take this opportunity with great pleasure to acknowledge and extend my heartfelt gratitude to the following persons who have made the successful completion of this PhD project possible.
First and foremost, my deepest gratitude goes to the lord – Allah s.w.t. and his messenger Prophet Muhammad (PBUH). Secondly, my gratitude goes to my father and mother and all family members. Special thanks go to my lovely wife Amani for her support, encouragement, quiet patience and unwavering love. You are my inspiration – this thesis is for both of you.
Thirdly, I would like to thank my supervisor, Prof Kamal Zuhairi Zamli and my co-supervisor Dr Abdulrahman Alsewari, for their advice, guidance and patience throughout the process until the completion of the project. Under their supervisions, I am well trained to work independently. Thanks for the endless vital encouragement and support given. My thanks also go to the staff of the Faculty of Computer Systems and Software Engineering, UMP, especially those members of my doctoral committee for their input, valuable discussions and accessibility.
Last but not least, I would like to thank my fellow PhD students: Yazan Alsariera, Fadhl Hujainah and Hasneezaliza for the friendship and support.
v TABLE OF CONTENT ACKNOWLEDGEMENT ii ABSTRAK iii ABSTRACT iv TABLE OF CONTENT v
LIST OF TABLES viii
LIST OF FIGURES ix LIST OF ABBREVIATIONS xi INTRODUCTION 1 1.1 Overview 1 1.2 Problem Statement 4 1.3 Research Questions 7 1.4 Aims and Objectives 8 1.5 Research Contribution 8 1.6 Research Activities 9 1.6.1 Literature Review 9 1.6.2 Design and Implementation 9 1.6.3 Benchmarking and Analysis 10
1.7 Research Scope 11
1.8 Structure of Thesis 11
LITERATURE REVIEW 13
2.1 Meta-heuristic Algorithms 13 2.2 Search Based Software Engineering 15 2.3 Test Suite Generation Techniques 16 2.3.1 Random Testing 17 2.3.2 Partition Testing 17 2.3.3 Cause and Effect Graphing 18 2.3.4 Combinatorial T-way Testing 20
2.4 T-way Test Suite Generation 20
vi
2.4.2 Sequence T-way Test Suite Generation 24 2.4.3 Theoretical Background of T-way Testing 28 2.5 Applications of T-way Testing 30
2.6 T-way Test Suite Generation Strategies 31
2.6.1 Existing Sequence-less T-way Test Suite Generation Strategies 31 2.6.2 Existing Sequence T-way Test Suite Generation Strategies 44
2.7 Gap Analysis 49
2.8 Summary 55
DESIGN AND IMPLEMENTATION OF THE eFPA
STRATEGY 57
3.1 Flower Pollination Algorithm 57 3.1.1 Flower Pollination Characteristics 58 3.1.2 Basic Form of Flower Pollination Algorithm 59 3.1.3 Variants of Flower Pollination Algorithm 61 3.2 Elitist Flower Pollination Algorithm Design 63 3.2.1 Interaction Elements Generation Algorithm 67 3.2.2 Test Suite Generation algorithm 69
3.3 Parameter Tuning 72
3.3.1 Switch and Elitism Probabilities 73 3.3.2 Pollen Size and Maximum Iteration 75 3.4 eFPA Implementation 77 3.4.1 eFPA Functions 78 3.4.2 eFPA User Interface 80
3.5 Summary 83
EXPERIMENTS, RESULTS AND DISCUSSIONS 84
4.1 Experiments 84
4.1.1 Comparison eFPA versus FPA 88 4.1.2 Benchmarking with Sequence T-way Strategies 90 4.1.3 Benchmarking with Existing Sequence-less T-way Strategies 95 4.1.4 Statistical Analysis 104
4.2 Discussion 110
vii
CONCLUSION AND FUTURE WORK 113
5.1 Overview 113
5.2 Future Work 115
REFERENCES 117
APPENDIX A RESEARCH MAPPING 128
APPENDIX B SIMPLE EXAMPLE 129
viii
LIST OF TABLES
Table 2.1 CEG Decision Table for Printer Example 19 Table 2.2 Air Asia Ticketing System Options 21 Table 2.3 List of Events in Air Asia Booking System 21 Table 2.4 Software System Configuration 23 Table 2.5 Exhaustive Testing 23 Table 2.6 Summary of Gap Analysis Findings 51 Table 2.7 Summary of Comparison of Existing Meta-Heurisitic based t-way
Strategies 52
Table 3.1 Describe eFPA’s parameters 70 Table 3.2 Average Test Suite for CA(N; 2, 46) and SCA(N; 3,9) with varied
values of sp = 0.1 to 0.9 73 Table 3.3 Averages Test Suite for CA(N; 2, 46) and SCA(N; 3,9) with varied
values of 𝑒𝑝 =[0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.4, and 0.5] 75 Table 3.4 Averages Test Suite for CA(N; 2, 46) with sp = 0.8 76 Table 3.5 Averages Test Suite for SCA (N; 3,9) with sp = 0.8 76 Table 3.6 Inputs line Arguments 79 Table 4.1 Parameters for Existing Sequence-less Strategies 87 Table 4.2 Comparison with Existing Sequence T-way Strategies for
Experiment 1 92
Table 4.3 Comparison with Existing Sequence-based T-way Strategies for
Experiment 2 93
Table 4.4 Comparison with Existing Sequence T-way Strategies for
Experiment 3 94
Table 4.5 Comparison with Existing Strategies for Experiment 4 97 Table 4.6 Comparison with Existing Strategies for Experiment 5 101 Table 4.7 Comparison with Existing Strategies for Experiment 6 102 Table 4.8 Comparison with Existing Strategies for Experiment 7 103 Table 4.9 Friedman test for Experiment 1 to 3 107 Table 4.10 Post-hoc Wilcoxon Rank-Sum Tests for Experiment 1 to 3 107 Table 4.11 Friedman Test for Experiment 4 108 Table 4.12 Post-hoc Wilcoxon Rank-Sum Tests for Experiment 4 108 Table 4.13 Friedman Test for Experiment 5 till 7 109 Table 4.14 Post-hoc Wilcoxon Rank-Sum Tests for Experiment 5 till 7 109
ix
LIST OF FIGURES
Figure 1.1 Software Failure Examples 2 Figure 1.2 Software Testing Life Cycle (STLC) Phases 3 Figure 1.3 Air Asia Ticket Booking Website 5 Figure 1.4 Research Activities 9 Figure 2.1 Search-Based Techniques Categories 16 Figure 2.2 Random Testing Example 17 Figure 2.3 Equivalence Partitioning Example 18 Figure 2.4 The CEG for Printer Example 19 Figure 2.5 Revisiting the Air Asia Ticket Booking Website 21 Figure 2.6 Illustration of T-way Combinations that are Covered by
Sequence-less Test Suite 24 Figure 2.7 Exhaustive Test Suite for 4 Events 26 Figure 2.8 Sequence 2-way Test suite for Four Events 26 Figure 2.9 Illustration of t-way Combinations are Covered by Sequence Test
Suite 27
Figure 2.10 Generic Meta-Heuristic Algorithms for Test Suite Generation 33 Figure 2.11 Hill Climbing Search Mechanism. 34 Figure 2.12 SA Test Case Generation Pseudocode 35 Figure 2.13 Tabu Search and Simulated Annealing Search Mechanisms 35 Figure 2.14 TS Test Case Generation 36 Figure 2.15 Population-based Search Mechanisms such GA, PSO and CS. 37 Figure 2.16 GA Test Case Generation Pseudocode 38 Figure 2.17 ACA Test Case Generation Pseudocode 39 Figure 2.18 PSO Test Suite Generation Algorithm Pseudocode 41 Figure 2.19 HS Test Suite Generation Algorithm Pseudocode 42 Figure 2.20 CS Test Suite Generation Algorithm Pseudocode 44 Figure 2.21 T-SEQ Test Suite Generation Algorithm Pseudocode 45 Figure 2.22 EDIST-SA Test Suite Generation Algorithm Pseudocode 47 Figure 2.23 BA Test Suite Generation Algorithm Pseudocode 48 Figure 3.1 Flower Pollination Methods 59 Figure 3.2 Flower Pollination Algorithm Pseudocode 61 Figure 3.3 Sequence and Sequence-less Test Case Repersentation 64 Figure 3.4 Elitism Feature 66 Figure 3.5 Graphical Representation of eFPA Design 67
x
Figure 3.6 Sequence-less Interaction Elements for CA (N; 2, 23 31) 68 Figure 3.7 Sequence Interaction Elements for SCA(N;4,3) 69 Figure 3.8 Pseudocode of eFPA strategy 71 Figure 3.9 Graphical Representation of Averages Test Suite for CA(N; 2, 46)
and SCA(N;3,9) with Varying Values of 𝑠𝑝 74 Figure 3.10 Graphical Representation of Averages Test Suite for CA(N; 2, 46)
and SCA(N;3,9) with varying values of 𝑒𝑝 75 Figure 3.11 Graphical representation of averages test suite for CA(N; 2, 46)
and SCA(N; 3,9) 77 Figure 3.12 eFPA implementation steps 78 Figure 3.13 eFPA Implementation User Interface 81 Figure 3.14 eFPA’s Final Test Suite Results for CA(121,2,105) 82 Figure 3.15 eFPA’s Final Test Suite Results for SCA(6,3,4) 83 Figure 4.1 Comparison of eFPA versus FPA for CA(N; 2, 105) 88
Figure 4.2 Comparison of eFPA versus FPA for SCA(N; 20, 3) 89 Figure 4.3 Time Complexity of the Original FPA, and eFPA 89
xi
LIST OF ABBREVIATIONS
ACA Ant Colony Algorithm
AETG Automatic Efficient Test Generator
ASP Answer Set Programing
BA Bees Algorithm
BFPA Binary Flower Pollination Algorithm
CPU Central Processing Unit
CS Cuckoo Search
CTEXL Classification-Tree Editor eXtended Logics
DC Direct Construction
DDA Deterministic Density Algorithm
EDIST-SA Event Driven Input Sequence Testing based on Simulated
Annealing
eFPA Elitism Flower Pollination Algorithm
FPA Flower Pollination Algorithm
G_MIPOG Grid_MIPOG
GA Genetic Algorithm
HC Hill Climbing
HSS Harmony Search strategy
IEGA Interaction Elements Generation Algorithm
IPO Input Parameter Order
IPOG Input Parameter Order Generalized
ITCH IBM’s Intelligent Test Case Handler
JDK Java Development Kit
LAHC Late Acceptance Hill-Climbing
mAETG Modified Automatic Efficient Test Generator
MC_MIPOG Multi_Core MIPOG
MIPOG Modified Input Parameter Order Generalized
MOFPA Multi-objective Flower Algorithm for Optimization
MPI Message Passing Interface
PC Personal Computer
PICT Pairwise Independent Combinatorial Testing
RAM Random Access Memory
xii
SA Simulated Annealing
SCAT Sequence Covering Array Generator
SPL Software product line
STLC Software Testing Life Cycle
SUT System Under Test
TConfig Test Configuration
T-SEQ T-SEQ Algorithm
TSGA Test Suite Generation Algorithm
TVG Test Vector Generator
U Upper bound Strategy
