3.4.1
Hou et al. 2008
Hou et al. [53] argued that quota constraint (e.g., the upper limit of the number of requests that a user can send to a Web service during a certain time range), may delay fault exposure and the subsequent debugging. Hence, they proposed a quota-constrained test-case prioritization techniques for service compositions to maximize requirement coverage testing.
The general idea behind the proposed prioritization techniques is to divide the regression testing (and thus test cases) into several time slots according to the request quotas. Thus, for each time slot, test cases are selected and then prioritized with the aim of maximizing the testing coverage per slot. After selecting test cases for each time slot, test cases are sorted using traditional test case prioritization techniques. After each test selection and prioritization per time slot, updated information regarding remaining selectable test cases, requirement coverage, and request quota are considered for scheduling test cases for the next time slot.
Hou et al. performed an experimental study (using a Travel Agent System) on their proposed techniques and compared them with the random technique, the traditional total technique, and the traditional additional technique. The results showed their prioritized test cases using their techniques are more ef-
fective for achieving total or additional branch coverage and exposing faults with the constraint of request quotas.
3.4.2
Mei et al. 2009 and 2011
Mei et al. [73, 74] presented black-box test case prioritization approach for single Web services in a service composition using WSDL-tag coverage. The approach reorders test cases based their coverage of WSDL-Tags (both ascend- ing and descending).
Focusing on business process service compositions, in [77] Mei et al. argue that industrial service compositions are typically written in languages such as WS-BPEL, and are integrated with workflow steps and Web services via XPath and WSDL. Faults in these artefacts may cause the service composition to extract wrong data from messages, thereby leading to failures in service compositions. To this end, Mei et al. propose a family of test case prioritization techniques based on multilevel coverage model. The multilevel coverage model captures the business process, XPath, and WSDL.
Thus, Mei et al. first order test cases according to the business process coverage. To resolve ties in case of similar process coverage, the coverage of the XPath expressions are considered. To resolve ties in case of similar XPath coverage, the coverage of the WSDL schema elements are considered.
Mei et al. present extensive evaluation of their contributions by performing experiments on a set of WS-BPEL applications.
3.4.3
Chen et al. 2010
Chen et al. [26] argue that when a service evolves, service interactions and change impacts are two key points in service composition’s regression testing. To this end, Chen et al. proposed a test case prioritization approach based on impact analysis of BPEL processes. In their approach, change impact analysis is performed on a weighted dependence propagation graph constructed from BPEL processes. Both data and control dependencies are considered in their analysis. The authors give weights to different impacted parts. The weight is defined as difference to the changed part. The executions of test cases are scheduled according to the impact weight.
Chen et al. demonstrate the applicability of their proposed approach by applying it on Automated Teller Machine (ATM) example.
3.4.4
Zhai et al. 2010 and 2014
Focusing on location-aware service compositions, Zhai et al. [110] suggest to incorporate service selection in test case prioritization in order to reduce the number of service invocations. Additionally, Zhai et al. propose a suite of metrics and test case prioritization techniques tailored to testing location- aware services.
The idea behind incorporating service selection in testing is to maintain – during testing process – a blacklist which contains faulty services. These faulty services are detected in previous test case executions and therefore should not be used by the service selection process. That is, if a service is found to be faulty, it should not be considered by the service selection approach. Discarding faulty services from the set of candidate service considered by the selection process reduces the number of services that need to be tested.
The test case prioritization idea is based on modelling test cases as a se- quence of Global Positioning System (GPS) locations. Consequently, several location-based metrics can be applied to the test cases and the results are used for scheduling the test cases. The proposed metrics are classified into
two groups: (1) input-guided and (2) Point-of-Interest (POI) aware. The
input-guided techniques are based on the observation that the more diverse a sequence is, the more effective they will be in fault detection. The input-guided techniques thus prioritize a test suite in descending order of the diversity of locations in test cases. The POI-aware techniques are based on the observation that test cases that are closer to POIs or cover more POIs are more effective in fault detection and thus should be assigned highest priorities.
In [109], Zhai et al. present an extended evaluation using a controlled
experiment, measuring the cost-effectiveness, and also comparing the proposed techniques in terms of several criteria.
3.4.5
Nguyen et al. 2011
Nguyen et al. [84] present an approach for prioritizing test cases for audit testing of service compositions using information retrieval (IR) techniques. In their approach, test cases are prioritized based on their relevance to the service change, based on matching service change description with the code portions
exercised by the relevant test cases. The matching is performed using IR
techniques.
The proposed approach assumes the availability of service change descrip- tions. Service change descriptions are used to define queries which are then matched – using IR techniques – with identifier documents. Identifier docu- ments, each associated with a test case, are extracted by analysing past exe- cution traces of test cases. The result of the match is used to search for the most relevant test cases for each change. Search results form a ranked list are used to prioritize the test cases.
Nguyen et al. perform a case study using the internally developed eBayfinder service composition, to evaluate the effectiveness of their IR-based prioritiza- tion, and compared it with coverage-based prioritization.
3.4.6
Evaluation of RTP Contributions
Table 3.4 provides a characterization of the reviewed RTP contributions by answering the questions used in the evaluation framework presented in Section 3.1.
Table 3.4: Evaluation of RTP Contributions
Research Contribution Section Question
1 (SO) 2 (DB) 3 (CA) 4 (MC) 5 (CIC) 6 (EP) 7 (RS) Hou et al. 2008 3.4.1 + − ◦ − − − − Mei et al. 2009 and 2011 (black box) 3.4.2 − − ◦ − − − ◦ Mei et al. 2009 (white box) 3.4.2 + − ◦ − − − − Chen et al. 2010 3.4.3 + − − − − − − Zhai et al. 2010 and 2014 3.4.4 + + − − − − + Nguyen et al. 2011 3.4.5 + − − − − − ◦
The results show that there are efforts for RTP for services and service compositions. However, existing approaches for service compositions are not
applied at runtime during the operation of the service composition. There- fore, these approaches do not consider the potential coverage obtained from runtime monitoring of service compositions. Additionally, the coverage-based approaches do not use execution plans of service composition.