As has been discussed in section 5.1 and 3, challenges of optimisation for the user include: appropriate selection of optimisation methodology and constraints on the model and boundary conditions, and integration with other tools and resources. This chapter focuses primarily on the latter however, the first aim in creating the Optimisation Service was create a loosely coupled component of a PSE that eases its integration with a range of heterogenous tools, including knowledge services. Traditional optimisation practises, as discussed in3, do not have the openness nor flexibility to easily allow the integration of heterogenous tools and resources. There are four identifiable stages to the optimisation process:
Optimisation
Service
Client
1: Reques t nextiteration 2: Store n ew opt. co ntext and OF params on databas e 3: Calculate next O F iteration on cluster 4: return result of OF calculation 3: Calculate next OF iteration Compute Node Compute Node Compute NodeCompute
Cluster
Database2: Store new opt. context and OF params
4: result of OF calculation
Figure 5.1: Example integration of Optimisation service with a compute cluster and
persistence database using a document-oriented message architecture.
Optimisation
Service
Client
1: Request nc return result of f(n c-1) and c-1 2: Respond new c and nm 3: Calculate f(nc)4: Retrieved resu lt of f(nc) Compute Node Compute Node Compute Node
Compute
Cluster
Figure 5.2: Example integration of Optimisation service with a compute cluster with the client acting as the controller of the workflow.
2. Linking of objective function to optimisation algorithm,
3. Configuration of input parameters such as boundary conditions, then
4. Execution of optimisation algorithm
The Optimisation Service takes a fundamentally different approach to traditional opti- misation by loosely-coupling the optimisation algorithm and objective function. Service- based optimisation differs in that it reverses the roles of the optimisation and objective function such that the objective function calls the optimisation function. This reverse communication model simplifies the coupling between the functions and frees the com- putational aspect of optimisation from its functionality. This is essential if we are to
provide a loosely-coupled service that offers flexible integration with other resources, as shown in figures 5.1and 5.2.
The Optimisation Service does not perform any calculation of the objective function component. Instead, it offers functionality pertaining to the state and operation of the optimisation algorithm: providing bootstrapping (initialisation), stateful management of the process, issuing of boundary parameters for calculation and multi-client support for and session management. It operates a pull model architecture that demands that the client must send commands, containing instructions or results, to the server in order to control and cause the progression of the optimisation.
Separation of objective function from the optimisation algorithm not only allows pro- vision of common optimisation functionality, but, as a consequence, enables the user to concentrate on their objective function. In addition, the adoption of a service-based paradigm and reverse communication model simplifies the integration of the optimisation process with other associated PSE tools and computational resources.
5.4.1 Bootstrapping
Bootstrapping allows knowledge and information to be included at the initialisation stage of the optimisation process. We define this as the procedure that the optimisation server employs to guide the user through the initiation of an optimisation.
The service contains no knowledge only information about the available optimisation algorithms and their capabilities. Responsibility for selection of algorithms and bound- ary parameters is left to the client. This mirrors the traditional optimisation process whereby the engineer had to decided which optimisation algorithm and boundary pa- rameters to select. However, this gave the engineer greater flexibility. If the service provided a knowledge repository it would reduce its flexibility and go against the prin- ciple of simple services. We just want the service to provide optimisation and nothing more.
A scenario of this would be at the start of the optimisation process, the client (user) sends general information to an intermediary optimisation knowledge service, such as the target-problem and its dimensions. The knowledge service will then access its database so that it can suggest suitable algorithms. For instance, the service could offer extended search capabilities to enable refinement of parameter and algorithm selection or allow manually selection and specific configuration information.
5.4.2 Stateful Code and User Sessions
Stateful code allows optimisation algorithm code to fit into the framework of web ori- ented application architectures, such as Sun Microsystems’s Java Enterprise Server
(J2EE), where each optimisation algorithm is represented by an Enterprise Java Bean (EJB) [103]. In addition, each step of the optimisation becomes a transactional unit al- lowing consistency and durability. Both recoding approaches require no changes to the way in which the objective function is traditionally written. They are also independent of the operating system and programming language to which the objective function uses.
Upon receiving and accepting a new request from the user, the server initialises a new optimisation session by generating a unique session identification (ID) and creating a new associated session EJB. We have created a class structure with different implemen- tations of EJBs that may be employed by other optimisation algorithms. This allows the easy addition of more optimisation algorithms. In addition, each contains all state and initialisation data associated with the optimisations progress, such as parameter values, previous results and awaiting calculation points.
The service employs dissemination methods described in section 4.3.4 and implements the EJB using checkpoint strategy mentioned in section 5.5.1. After successfully com- pletion, the optimisation’s related state and current data could be stored in an external database, using the EJB created at the bootstrap stage providing robustness.
This strategy makes the service flexible enough to meet the requirements of different types of client and enables the server to deal with multiple service requests at the same time. Stateful code allows the possibility of parallel multiple clients calculating independent points for a single function optimisation.
5.4.3 Client Access and Service Communication
As a means to provide a cross-platform user interface to the Optimisation Service, we choose to employ the commonly supported Web-based technology, HTTP, as the basis for the services submission system. Currently, an interface to the HTTP methods are accessible throw Web page forms that exist as static files on the service-side however, in the context of our future design objectives, we would like to dynamically generate these at runtime using optimisation schemas as the template. With the initial aim of simplifying the process of adding new services but, also enabling the possibility, with the addition of a knowledge repository, of on-the-fly changes to the user’s interface that reflects their particular optimisation and environment requirements.
The emerging XForm [122] technology will, in the future, provide an effective dynamic solution with new features like platform-device independence, XML integration, and the separation between user interface and data. At the moment we support stand- alone client application through XML and Web Forms for browsers. We employ Servlet technology [123] to generate and process XML and Web forms. However, we shall drop support for Web forms in favour of just XML because providing the later does not fit into a our philosophy of simple services.
5.4.4 Checkpointing for Robustness, Monitoring and Control
We describe here a method, called checkpointing, that not only provides robust service execution but, in addition, provides users of traditional optimisation practices with a means to monitor and control (or steer) the progress of an optimisation. Checkpointing is an interesting side-effect that became apparent from the redevelopment of the Amoeba algorithm to support loose-coupling. Checkpointing is the ability of an optimisation algorithm to journal, or in other words, serialise its complete state to disk or database, allowing it to be resumed from any journalled point that occurred during its execution. This provides robustness such that in the case of a computer crash it may be restarted from its latest serialised instance. In addition, this also allows optimisations to be executed on compute resources in a distributed ownership environment, such as Condor (see section 6.5), where compute resources availability cannot be guaranteed requiring that executing jobs support the ability to be migrated from a resource that has become unavailable to an available resource. To support migration the job must have the ability to shut itself down and resume from where it left off on a new computer.
Another ability provided by checkpointing is too allow the possibility of steering an optimisation. Checkpointing in effect allows an optimisation to be paused or rolled back to a previous checkpoint, modified and then resumed however the optimisation must offer an interface to support modification. One last application of checkpointing would be to speed up repetitive workflows where for instance multiple optimisations are instantiated. The configuration of these optimisations may only differ slightly such that the initial parts are exactly the same and only diverge at known stages within the execution of the algorithm. Checkpointing at this stage would allow a short cut such that this stage need only be executed once and then started from that checkpoint.
A checkpoint may contain workable results however, some algorithms may only sporadi- cally produce useful data or only upon completion. In which case, time can be saved by employing resumable algorithms. Saved state within a checkpoint alleviates the service from recalculation of variables redeemable from stored data. In addition, it allows re- trieval of irrecoverable variables that would otherwise prevent the algorithm’s continued execution.
Viewing of checkpoints and resumable programs provide an uncomplicated way to re- spectively monitor and control an algorithms progression. For example, a user may view and compare checkpoints to determine if an algorithm was advancing correctly. If not then they may simple the restart the algorithm from scratch or, if recoverable, alter its configuration and resume execution from a checkpoint with correct data.
Optimization Server: Execute Optimization Function Client PC:
Execute Objective Function
1: Bootstrap
3: Return result of objective function 2: Send next value to be calculated
4: Return result of optimisation
Figure 5.3: Client driven optimisation.