Complexity Analysis of the Extended Algorithm

In this section, we perform asymptotic analysis for the runtime of the extended graph-walk algorithm. In order for our extended graph-walk algorithm to be applicable for service compositions at runtime, the runtime of the algorithm is a relevant aspect. To support timely adaptation activities, the algorithm should be efficient when applied to service compositions at runtime.

Rothermel and Harrold [94] have analysed the runtime of their graph-walk algorithm using the big-Oh notation. Therefore, to facilitate benchmarking with the original algorithm, we adopt the same notation for our analysis.

The runtime of the extended graph-walk algorithm depends on the runtime of the calls to COMPARE plus the runtime of the other operations for initializing the sets. However, since the initialisations have constant time, the runtime of the algorithm is bounded by the runtime of the calls to COMPARE. An up- per bound on the number of calls to COMPARE is reached when assuming that

COMPARE can be called with each pair of nodes N and N0 in the control-flow

graphs G and G0, respectively.

To estimate the runtime of COMPARE, we initially assume that COMPARE does not include the extension to examine modifications in service bindings. Under this assumption, the runtime of COMPARE is similar to the runtime of the orig- inal algorithm of Rothermel and Harrold [94]. Rothermel and Harrold analyse the runtime of COMPARE as follows. A call to COMPARE results in at most two calls to LEquivalent, resulting in either a set union operation or examination

of at most two successors of N and N0. The set union operation is bounded by

the number of execution traces (|ET |) in the examined set of execution traces ET . The runtime of the function LEquivalent is bounded by the number of characters in the compared texts; a constant k, the maximum text length, for practical reasons [94]. Thus, the runtime of COMPARE is bounded by O(k |ET |) for some constant k.

Therefore, given a pair of control-flow graphs G and G0 containing n and

n0 nodes, respectively, and given a set of |ET | execution traces, if COMPARE is

called for each pair of nodes (N ∈ G, N0 ∈ G0_{), the runtime of the algorithm}

nodes N and N0 from G and G0 applies only to graphs G and G0 for which the multiply-visited-node condition holds. In these cases, the runtime of the

graph-walk algorithm is O(|ET | (min{n, n0})).

To add the cost of the extension to the runtime of the algorithm, we assume that the number of bindings in a single node is bounded by a constant c; a maximum number of bindings for all nodes.

Each call to COMPARE results in at most b lookups on the table DBI for the modified bindings of the examined node. This results in a runtime bounded to c for the worst case (assuming all bindings of the node are modified). Based on the value of c, additional set union operations are invoked leading to a runtime of |ET | c. It follows that these additional steps are bounded by |ET | c. Thus,

the time required by the algorithm in the worst case is O(|ET | n n0c). When

the multiply-visited-node condition does not hold, the expression |ET | n n0

becomes |ET | (min{n, n0}) leading to a worst case is of O(|ET | (min{n, n0_{} c)}

for the extended algorithm.

To conclude, the runtime of the algorithm is linear to the number of control- flow graph nodes when the other bounding parameters are hold constant. The runtime of the algorithm is linear to the number of bindings when the other bounding parameters are hold constant. The runtime of the algorithm is linear to the number of execution traces when the other bounding parameters are hold constant.

5.3

Summary

For computing coverage of service composition, valid execution traces are used. In this chapter we defined invalid execution traces. Additionally, we presented our algorithm for determining invalid execution traces. The algorithm extends an existing graph-walk algorithm of Rothermel and Harrold for safe regression test case selection to account for modifications in service bindings.

Coverage Criteria

Traditional cover criteria are based on a static model of the program code structure (i.e., control-flow graph), which maps to a concrete implementation of the program. For instance, if the nodes in the control-flow graph are used to represent the functions of a program, then each node represents one concrete implementation of one function in the program. However, dynamic binding implies that the control-flow graph generated from the service composition code can have a large number of possible realizations, each having different implementation and behaviour. For example, an abstract service can have more than one candidate service to be bound during service execution.

The coverage criteria can be fulfilled using only one realization of the service composition. However, the remaining potential realizations remain uncovered and thus may contain undetected faults. Achieving adequate coverage for all potential realizations using the traditional coverage criteria can require a large number of test cases. Additionally, the cost of testing a service composition can be very high, since each test case can require the invocation of the service from the service provider [20]. Moreover, some of the candidate concrete ser- vices and/or combinations of the candidate services may never be used when executing the service composition. From quality assurance perspective, not covering the unused (combination of) candidate services should not be of con- cern. If these unused (combination of) candidate services would be considered, the achieved coverage might be perceived superfluous. Moreover, redundant testing might be initiated.

The above limitations of traditional coverage criteria call for the definition 69

of a set of coverage criteria more appropriate for service composition. The coverage criteria proposed in this chapter consider the actual execution plans of a service composition as reference for the coverage assessment. Execution plans specify the binding of concrete services to the abstract services defined in the service composition and thus define the combination of concrete services for a service composition [100, 108, 5]. Considering execution plans guaranties that only the combinations of concrete services actually used for realizing the service composition are considered.

6.1

Intra-plan and Inter-plan Coverage Criteria

Often, different execution plans for the same service composition are defined for different user groups with varying end-to-end requirements, such as overall performance or availability. The binding between the abstract services and the concrete services can be defined at design-time, but often the binding is delayed until deployment or even runtime [91]. Additionally, re-binding of concrete services might occur at runtime in case of adaptation [35]. Considering execution plans allows us to define the following two types of coverage criteria: • Intra-plan coverage criteria focus on the coverage within one execution plan. For example, coverage of all service composition’s operations de- fined in one execution plan.

• Inter-plan coverage criteria consider coverage across all execution plans. For example, coverage of service composition’s operations defined in all execution plans.

Traditional coverage criteria consider only a whole service composition. An abstract service of a service composition may have multiple bindings defined in different execution plans. It would then be helpful to have coverage criteria focusing on coverage of one abstract service considering execution plans. This scope of coverage is especially helpful for critical abstract services whose cover- age might be more important than for other services. Thus, we define separate coverage criteria for one abstract service and for a whole service composition considering execution plans:

• The local coverage criteria focus on coverage of one abstract service of a service composition.

• The global coverage criteria focus on a whole service composition. Considering execution plans and different coverage scopes leads us to define four new coverage criteria: (1) intra-plan-local, (2) inter-plan-local, (3) intra- plan-global, and (4) inter-plan-global.

Figure 6.1 provides an overview of the criteria.

Figure 6.1: Our Proposed Coverage Criteria