Developing Data Mining Scripts by Interactive and Rapid Prototyping

In Section 4.4, we described the high-level services layer of the architecture presented in Section 4.2 and its component GridR service. Now, we focus on the client layer. The client layer sits on top of the high-level services layer and includes client side components providing user interfaces for services in the lower layer. Such clients are, e.g., components of workflow environments or programming language components. Here, we will focus on a

4.5. Developing Data Mining Scripts by Interactive and Rapid Prototyping

Figure 4.7.: GridR client architecture.

certain client component - the GridR client - which is part of our approach on interactive development of data mining scripts in the grid.

The GridR client provides the functionality of submitting the execution of R scripts and R functions from an R session on a client side machine to the GridR service, thus allowing for an execution in the grid environment.

The GridR client is structured into the GridR R package, which represents an extension to the R environment, and a Java-based client to the GridR service, which is attached to the R package. Figure 4.7 depicts the GridR client in the architecture of our approach. The R package includes a set of functions which make the GridR client’s functionality available for the users. These functions are bound to a Java-based client for the GridR web service. The R to Java communication is performed via the RJava package [105].

The GridR client is used as an interactive tool and programming language interface for accessing grid resources and in particular for the remote execution of R code in the grid. More specifically, the task of executing R scripts and functions is submitted via the GridR service as a job to a remote machine in a grid environment by interactively calling the respective functions from an R session.

4.5.1. GridR Client Architecture

The R package part of the GridR client is internally structured around the components ”RemoteExecution” and ”Locking”. The RemoteExecution component is responsible for the execution of R code as a job in the grid environment by transforming the R code to execute into a set of files and submitting it for execution to the GridR service (see Section 4.4) via a Java-based web service client.

The GridR service is responsible for creating a job description file in the respective job description language and for submitting the job to the Resource Management service of the grid. During this process, the Locking component takes care of the consistency of files and variables in the R session. Figure 4.8 visualizes the detailed architecture of the GridR client.

Figure 4.8.: Details of the GridR client: RemoteExecution and Locking.

In practice grid access is performed through the call of predefined R functions collected in an R package. The client side components are based on usual R functions (see Table 4.2 and Table 4.3), so no changes in the core R implementation are necessary.

Name Action

grid.apply Performs a remote execution of R func- tions; waits (callback) or sets a lock (grid.lock).

grid.check Checks if the internal structure contains all variables and functions to execute a function f remotely. If not, the missing variables are returned.

grid.consistency Checks if the internal structure has errors or if there are local files without a running job.

grid.init Initialization of the grid environment. grid.isLocked Checks if a variable has a lock.

Table 4.2.: Public functions of the GridR client.

The functions are based on the following R functionalities:

• callbacks - functions that are executed automatically by R after the user has issued a command (this is used when checking for results).

• active bindings - a variable is replaced by a function call which is handling the locking system and allows working interactively with that variable. When the variable is read, the predefined function is called and returns the value associated to the variable (or an error code if the variable is locked). When a value is assigned to the variable, the function is called with the value as parameter for storage in an internal structure.

4.5. Developing Data Mining Scripts by Interactive and Rapid Prototyping

Name Action

grid.callback Performs a callback.

grid.waitForAll Performs grid.callback for all jobs. grid.readLock Read-lock a variable.

grid.writeLock Write-lock a variable.

grid.unLock Removes the lock from grid variable. grid.unlockAll Unlocking of all grid variables. grid.catchObjectNotFoundError Error handling by parsing of error

messages.

Table 4.3.: Internal functions of the GridR client.

• parsing of script and error code - checking for missing values, variables and functions in the code which is executed remotely as well as for errors in the result objects.

Thus, users can make use of the functionality of execution in the grid environment in a transparent way by passing the functions to be executed in the grid as input to one of those predefined functions (grid.apply) in their local code. Figure 4.9 shows the execution of a simple sum function with the GridR client. Details on the steps of execution are described in the following section.

Figure 4.9.: Simple GridR Client Code Example.

4.5.2. Process of Execution

As described in Section 2.1.3, the difference between a script and a function execution is as follows: an R script can be executed stand alone, which means that it does not depend on objects from an active R session. During the execution of a script, a new R session is created, which is closed after the execution finished. R scripts might only depend on input files that have to be accessible from the script’s code. In contrast, R functions depend on the workspace of an active R session. This means, as the computation of the R function is

supposed to be performed on a different machine than the client machine where the user’s R session is running, that the workspace has to be made available for reuse on the remote execution machine in the grid.

Figure 4.10.: Steps of executing an R function with the GridR client.

In the following the process of executing a single R function in a grid environment is described. Figure 4.10 visualizes the steps of executing an R function with the GridR client.

• Function loading. The GridR client functions are loaded from the GridR package into the workspace of the R client.

• Grid initialization. The grid environment is initialized by calling the function grid.init. This function sets the configuration for the GridR service (service location). To avoid the specification of settings on each submission, a configuration file can be used to pre-configure the settings.

• Code writing. The R code which is to be executed in the grid is written and wrapped as single R function in the local R environment.

• Grid submission. The grid.apply function is called, which launches the process of submission. At first, the function to be executed in the grid and the needed pa- rameters are written into a file (uniqueID-fx). Then, the R script which is executed on the machines of the fabric layer of the grid environment on job startup is gen- erated (uniqueID-RemoteScript.R). Next, an R file is created, which specifies the