The Simulation-based Evaluation Process

With the absence of publicly available data sets about web services’ prices, bundling of web services and requests for composite web services, we based our evaluation on

96

conducting simulations. The proposed simulation-based evaluation process is presented in Fig 4.2.

Fig 4.2. The evaluation process

The objective of the simulation is to analyze the performance of the proposed auction- based composite service selection mechanisms (single and simultaneous) in allocating web service offers to one or more composite service requests. The performance metrics have been defined based on the important criteria in this problem domain which are: (1) the success rate of the mechanism in allocating service offers to requests, (2) the final cost of composite service provisioning, and (3) the time to find the optimal allocations for the requests.

However, due to the limited empirical research on composite service selection with bundling or composite service selection in the presence of multiple requests, the design of the simulation, in terms of the data generation model and seeding of the parameters, was a challenging for our study.

We initially designed the experiments to evaluate the proposed “single auction” mechanism (to be discussed in Chapter 5). Based on the experience from these experiments, we revised the design of the experiments and improved it with more realistic scenarios and seeding data for the evaluation of the proposed “simultaneous auction” mechanism (to be discussed in Chapter 6).

97

In our experiments, the two essential elements of the simulations are the data generation model and the seeding of the simulation’s parameters. For the data generation model, we initially used an existing package, CATS suite (Leyton-Brown et al. 2000), that generates combinatorial bids for combinatorial auctions. However due to the limitations of this package, later we decided to design our own data generation model to have full control over the data set to be able to accommodate the specific requirements of our problem domain, that is, the (multiple) composite service selection problem. Nevertheless, our designed data generation model is based on the data distributions applied in combinatorial auctions literature. The final data generation model is discussed in subsection 6.4.4.

For the seeding of the data generation model, we initially referred to the existing experiments in the service selection literature. However, the limited studies and experiments on service selection with bundling of web services and multiple requests for composite services led us to look for more realistic data on the Internet.

In this regard, we studied a number of web services’ communities on the Internet where service requesters need composite services. Based on the information we collected from these directories, we decided to perform the experiments of the simultaneous auction mechanism in specific market sections of web services rather than a general market. Without focusing on specific market sections, it is rather difficult to estimate the number of service providers, requesters and the type of the web service offers and requests in a generic market for web services.

The market sections are designed based on two factors dividing the web services marketplaces into the following categories:

(1) The market economy size, that is the magnitude of the number of participants in a market and categorizes the markets into small economy and large economy (Tang 2004),

(2) The composite service complexity, which divides the markets to market for simple composite services and market for complex composite services (Weinhardt et al. 2011a).

The resulting four market sections are presented in Table 4.1 below. The simplest section is the market section with small economy (limited number of service providers and requesters) and mainly trading simple composite services. An example of such a market

98

can be a newly-formed market for mobile applications. The most complex section is the market where many service providers and requesters participate to trade complex composite services. The example for this section can be the rather mature web service community around Life Science with the web service registry called BioCatalogue15. This directory currently has more than 790 active members who are mainly interested in complex scientific workflows. More details about these market sections and motivational scenarios are presented in subsection 6.4.3 (Scenarios to Investigate: Market Sections).

Table 4.1. The four market sections

Composite Service Complexity Economy Size

SIIMPLE COMPLEX

SMALL small-simple small-complex

LGRGE large-simple large-complex

These market sections and the number of web service providers and requesters in the existing web service communities helped us establish the seeding data for the simulation parameters in the data generation model.

In Table 4.2, we have presented a summary of the implementation detail of the evaluation process design element; which are the performance metrics, the baseline for comparison, the scenarios to be investigated (market sections), the simulation’s data generation model, and the seeding of the simulation parameters. More details are presented for the single auction approach in section 5.2 and for the simultaneous auction approach in section 6.4.

The proposed evaluation process in general, and the data generation model and the four market sections in particular provide a useful framework for the service selection community. The designed evaluation process improves the transparency of the experiments in the composite web service selection domain by identifying a number of elements that need to be supported and their implementation need to be clarified when conducting empirical evaluations.

99

Table 4.2. The implementation of the evaluation process for the proposed auction-based mechanisms Evaluation Element Single Auction (Chapter 5) Simultaneous Auction (Chapter 6) Performance Metrics

 Success rate of the mechanism

 Cost of procuring a composite service

 Solve time

 Success rate of the mechanism

 Cost of procuring a composite service

 Cost of procuring a task

 Cost homogeneity across different requests

 Solve time

Baseline A non-combinatorial auction

(single-item bids)

 The single auction mechanism when applied to a set of requests one at a time

 A fixed-price mechanism where the requesters determine the price to be paid for the composite service

Scenarios to Investigate

(Market sections)

Generic

Four market section based on two factors:

 The complexity of the requests (the number of tasks in a request)

 The economy size (the number of requesters and providers attending the market section)

Data Generation

Model

 CATS arbitrary

distribution: number of and choice of services in a bundle

 IPV and CM discounted pricing function: pricing of services and the bundles

 Decay distribution: number of and choice of services in a bundle

 IPV discounted pricing function: pricing of services and the bundles

 Decay distribution: number of and choice of services in a bundle

 Uniform: number of tasks in a request

Seeding Similar experiments in the

literature

 Similar experiments in the literature

 Data available from the web services’ communities on the Internet

The data generation model and the experimental scenarios establish the basis for more realistic data sets by considering the service selection requirements in the creation of the data set and relying on the data obtained from existing web service communities for seeding the simulation.

100

4.4 Conclusion

In this chapter, we first discussed the theoretical foundations of our research which are: web service technology, auction theory, mechanism design and mathematical optimization. Then, we introduced our research methodology which comprises three steps: design, modelling and evaluation. The design includes two parts: (1) identifying the design elements of an auction model, and (2) differentiating an auction model for composite service selection from other auction models by studying the web services characteristics in general and the requirements of the composite service selection problem to an auction-based solution in particular. Then, we discussed our modelling approach which is based on mathematical optimization, or more specifically integer linear programming (ILP). Finally, we presented the evaluation process which we designed to evaluate our proposed auction-based mechanisms for composite service selection. Due to lack of publicly available data sets of web services’ prices, bundling or composite service requests, we based our experiments on simulations.

101

Chapter 5

5 An Auction-based Mechanism

for Composite Service Selection

5.1 Introduction

In this chapter, we present an auction-based mechanism to solve the composite service selection problem for a single composite service. We refer to this mechanism as the “single auction” mechanism. The proposed mechanism is based on multi-attribute combinatorial procurement auctions and is formulated as an integer linear programming (ILP) problem.

In section 5.2, we discuss the requirement of considering the dependencies between the constituent services in a composition and the impact of these dependencies on the service providers and requesters’ preferences and constraints. We also introduce and develop two important constraints regarding: (1) the cohesion of the composite service, and (2) the configuration of the providers participating in the service composition.

The design of the proposed auction-based model is presented in section 5.3. The design includes the three elements of an auction model: the multi-attribute bid specification, winner determination problem which is mapped to an integer linear programing problem and a discussion of the payment rule.

The proposed model is evaluated through simulation-based experiments and the results are presented in section 5.4. The objective of the experiments is to study the performance of the proposed mechanism in terms of its success rate, the provisioning cost of the

102

composite service and the running time of the proposed mechanism to find the optimal solution.

Parts of this chapter have been previously published in (Moghaddam et al. 2013). The theory and design discussions have been revised and updated, and a new experiment has been added which will be discussed in subsection 5.3.2.3 (Stage 3).