Large Objective Function Results Set

Similar to the above section, larger result sets returned from the completion of a ob- jective function will have bigger message sizes and thus greater resource requirements. In addition, the data structure, format and encoding of these results sets will differ ac- cording to the objective function prototype. However, unlike state data in which the Optimisation Service understands which data is required, as the implementation of the optimisation algorithm is encapsulated, the client must supply the complete result set to the server. Furthermore, only a partial result set may be needed at a particular time by the optimisation algorithm thus wasting valuable resources. The issue is worsened with large data sets, complex or multiple file result sets, and binary formatted data which are costly to include in XML messages. These issues can be overcome by usage of a data/file server that both sides share accessibility and which supports partial file/data transfers. Upon completion of the objective function the client can arrange for the results to be made available on the shared file/data server to the optimisation server and then instead supply it with an URI reference to the data. The optimisation server upon receiving the message may then read as much or as little of the results set as it requires. It is important to note that results sets supplied to the Optimisation Service in this fashion must be in a non-relationsional and non-hierarchical format in order for the optimisation server to partially read yet fully understand the data. This unfortunately rules out using XML however this is an advantage because data formats such as flat files, arrays, and lists are significantly quicker and easier to generate and parse, and are generally much more concise.

5.8

Summary

The Optimisation Service demonstrates the successful usage of key technologies that we discussed in section2.5.4, including XML and HTTP, and has shown how it is possible to create stateless loosely-coupled numerical Optimisation Service. In addition, we have implemented the concepts described in chapter4, and can show, although they have only been utilised in a functionally simple example, that the analysis described therein is valid and has great potential in future projects. This chapter has given an example of how a key component of the problem solving process can be offered as services and highlight the significant advantages of a Service-Oriented Architecture. We now summarise important conclusions from work in this chapter:

1. The new technique of decoupling the objective function from the optimisation algorithm using a reverse communication model has allowed us to build an ex- tremely flexible, loosely-coupled service. Its adoption was a vital design decision which gives much greater flexibility in the choice of services which may be coupled together with it in to environments for problem solving; specifically computation, persistence and knowledge services of which it is independent of any specific im- plementation.

2. Reverse communications provides users with a programming style to build robust- ness, monitoring and control into their codes.

3. Adoption of a stateless service model was another key design decision that reduced the complexity of service’s implementation, allowed the service to better scale and in combination with reverse communication provides additional reliability through checkpointing. In addition, reverse communication made it possible to easily create a stateless service.

4. The numerical Optimisation Service is an example of how a purely message- driven service provides flexibility by allowing its integration with resources such as databases and compute cluster.

5. Encapsulation of optimisation algorithm within the service removed the implemen- tation details of the optimisation algorithm from the client which further promotes interoperability.

However, whilst we have shown that its is feasible to use some of the basic Grid tech- nologies in a single service it is apparent that in order to make sophisticated PSE it requires examination of the newer Grid concepts of Web Services. The following chapter will examine Web Service technology in detail.

Service-Oriented Computation

This chapter discusses the sharing and management of high throughput cluster resources using service-oriented methodologies discussed in chapter 4. We demonstrate the usage of Web Service technologies to provide meta-based computing system that virtualises standard cluster operations; including submission, management, monitoring of compute jobs and the discovery of compute resources and their matchmaking with job require- ments.

We will show how a legacy system, the Condor High Throughput Computing (HTC) system, may be modelled as a Grid Resource and consequently adapted and virtualised to provide the necessary interoperability for inclusion in Grid-enabled Problem Solving Environments (PSEs). We will demonstrate the success of our work by showing the deployment of the Service-Oriented HTC resource, the Computation Web Service, by discussing its deployment in the Geodise project for engineering design simulations. In addition, in the next chapter7we will demonstrate its employment in a micromagnetics PSE. The work presented in this chapter has been published in [66,65,125,126].

6.1

Virtualisation of Compute Clusters

Resource virtualisation of compute clusters provides a common operation and behaviour abstraction that allows easy migration between particular implementations and improves application level interoperability.

6.1.1 Issues with existing virtualised Compute Resource Systems