Web Service Interfaces for Long-running Simulation Tasks

Services (and in particular web services) are understood as a standardized and interop- erable set of interfaces which expose functions of a software component (Josuttis, 2007). Because web services are typically stateless, the execution of (potentially long-running) simulations has two consequences:

5.1. Making 13CFLUX2 Service-oriented

1. The web service call itself is blocking, i.e., the web service call by the client will return after the simulation is finished.

2. State management has to be realized on top of the web service execution.

The Web Services Resource Framework (WSRF) addresses the realization of industry- grade standardized stateful web services and, thus, is ideally suited to work with other WS- * standards such as WS-Addressing or WS-Notification (Banks, 2006). Notably, WSRF is also used together with BPEL (Dörnemann, Smith, and Freisleben, 2008). § 5.2.3.2 presents an example where WSRF is used in a BPEL workflow.

However, well-tested and easy-to-use open source WSRF libraries outside the Java pro- gramming language are (to the author’s best knowledge) rather uncommon. To fill this gap in the SWF, FluxWS is developed as a lightweight library to realize stateless and stateful web services using plain SOAP. As an outstanding feature, FluxWS provides a custom-tailored parameterization concept for configuring 13CFLUX2 web service appli- cation. Exemplarily, the realization of the Fitfluxes web service within FluxWS is shown at the end of this paragraph.

5.1.2.1. FluxWS Web Service API

FluxWS provides two SOAP web service interfaces to invoke 13CFLUX2 programs:

1. Synchronous calls are blocking until the web service processing is finished (in which case the standard output result from the FluxCore call is returned), or an error occurred and a SOAPFault exception is returned.

2. Asynchronous calls immediately return with a string-based handle which refers to a unique ExecutionContext entity. In the context of FluxWS, this string is called JobTicket. A web service client may perform the following operations on the web service using the JobTicket object:

• delete: stop the job with a POSIX SIGKILL signal and remove the execution context data.

• stop: send a graceful job termination signal (realized as POSIX SIGINT signal). • isRunning: determine the job execution status.

• listFiles: return a list of file names in the execution context. • getFileData and getFileSize: return file data.

• waitFor: wait for a specified amount of time until the job is terminated. This method synchronizes the client-side execution of the service call.

The utilization of FluxWS using a Python SOAP web service client with the SUDS library is demonstrated in appendix B.5.

Chapter 5. Service-oriented 13C-MFA Solutions

5.1.2.2. Workflow Parameterization Concept of FluxWS

While 13CFLUX2 programs are parameterized independently from each other in a com- posite simulation, some program arguments have in practice the same value for all tasks within a workflow run. The selection of FluxML configurations, the provenance collec- tion behavior, and logging parameterization are identified as common 13CFLUX2 pro- gram arguments. In addition, several arguments are shared between simulation programs, e.g., optimizer parameters and solver configuration are uniformly defined in 13CFLUX2.

FluxCore provides a consistent mapping to ease the parameterization and composition of

complex workflow applications:

S : (P, C) ↦→ R

Here, the 13CFLUX2 service S maps a XML-based parameter string C and the service parameter P to the service result R. Typically, P is a FluxML model and R is XML or HDF5 data comprising simulation data. The workflow-specific configuration C is passed throughout each workflow step. Using this parameter XML document2, 13CFLUX2 pro- grams are configured individually. This parameterization concept leverages the composi- tion of 13C-MFA workflows with stateless web services.

5.1.2.3. The Fitfluxes Web Service

To demonstrate the lightweight character of FluxWS and the underlying FluxCore li- braries, the differences of the synchronous and asynchronous realization of the Fitfluxes web service are shown. The synchronous Fitfluxes service endpoint implementation consists of 11 lines of code (comments, logs, and error handling omitted):

1 public byte[] Fitfluxes(byte[] fmlcontent, Parameter P) throws Fault {

2 Fitfluxes srv = Fitfluxes.create(); 3 srv.setInputContent(fmlcontent); 4 srv.setParameter(P); 5 try { 6 return srv.run("stdout"); 7 } 8 finally { 9 srv.getExecutionContext().cleanup(); 10 } 11 }

Line 2 instantiates the Fitfluxes service object. After passing the input data (line 3) and workflow parameters (line 4), the program is run with stdout as program output file name in the execution context (line 6). This is a synchronous program execution, hence the result data is immediately returned and the execution context is cleaned up (Line 9). Besides the common asynchronous web service functions (e.g., waitFor or stop), the specific Fitfluxes realization for this API consists of a "start" method:

2

The FluxCore parameter XML format is defined in the following XSD schema: http://13cflux.net/ fluxparameter; last accessed: May 22, 2017

5.2. 13CFLUX2 in the Cloud with Apache Hadoop MapReduce

1 public JobTicket FitfluxesStart(byte[] fmlcontent, Parameter xmlparams) throws

Fault {

2 Fitfluxes srv = Fitfluxes.create(); 3 srv.setInputContent(fmlcontent); 4 srv.setParameter(xmlparams); 5 String ctx_id = srv.start();

6 JobTicket ticket = new JobTicket(ctx_id);

7 return ticket;

8 }

While the Fitfluxes call blocks until the simulation is finished, FitfluxesStart im- mediately returns with a JobTicket. The identifier of the execution context is retrieved after asynchronously starting the task execution (line 5). With this context identifier, a JobTicket instance is created and returned to the caller (lines 6-7).

5.2. 13CFLUX2 in the Cloud with Apache Hadoop MapReduce

This section presents the solution of this thesis for deploying computationally demand- ing 13C-MFA simulations on cloud computing resources using Hadoop MapReduce. On the example of 13CFLUX2, our primary simulation tool in the 13C-MFA environment, three realized variants of the MCB algorithm are presented: a straightforward approach which calls 13CFLUX2 programs from the command-line, an improved approach (called

FluxHadoop) using FluxCore and the Hadoop Java API, and a specialized version (called FluxHadoop2 ) that utilizes local CPU cores of a compute node. The general life-cycle of

a 13CFLUX2 simulation in a Hadoop MapReduce cloud setup is described first.