This chapter looks at several data exchange languages and tools that are important in cloud com- puting, particularly XML and JSON. Most applications built on top of IaaS clouds need both XML and JSON. This chapter helps users gain a basic background in the topic without having to go to external sources to understand the content in the book.
Extensible Markup Language (XML)
XML has been around for a long time, but a few points about it are worth discussing in relation to clouds. The first problem a programmer encounters with XML is how to parse it. The Java API for XML Binding (JAXB) API makes parsing XML documents simple. It binds Java objects to XML elements in a preprocessing step, an approach that saves a lot of development effort com- pared to a hand-crafted parser using the Document Object Model (DOM) or Simple API for XML (SAX). Unfortunately, not all programming languages have a utility like this.
The IBM SmartCloud Enterprise uses XML files called parameters.xmlas a template to indicate which parameters the user should be prompted to enter when creating an image. The file is located in the RAM catalog entry for the image it is associated with, and the GET
offerings/image REST API gives a URL for it. The equivalent Java API is Image.getMani- fest(). The following example represents the template of an image for an install of IBM DB2.
ptg8286219
<?xml version=”1.0” encoding=”UTF-8”?>
<parameters xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
xsi:noNamespaceSchemaLocation=”platform:/resource/com.ibm.ccl.devcloud
.client/schema/parameters.xsd”>
<field name=”db2_user_password” type=”password” label=”Instance owner (db2inst1) password”
pattern=”^\w*(?=\w*\d)(?=\w*[a-z])(?=\w*[A-Z])\w*$” patternErrorMessage=”Invalid db2inst1 password. Must contain least ...”
description=”db2inst1 password. Must contain ...”> <values>
<value>db2cloud</value> </values>
</field>
...
</parameters>
You might need to parse the parameters.xmlfile in two scenarios: when provisioning an instance in a cloud-management application and when customizing the behavior of an instance from inside the instance itself. In the first case, the parameters.xmlfile tells you how to prompt the user for parameters that are passed in at the time the provisioning request is submitted (for example, for values of the different DB2 passwords listed earlier). This was discussed in the sec- tion on the IBM SmartCloud Enterprise REST API in Chapter 3, “Developing with IBM Smart- Cloud Enterprise APIs.” Your program must discover the name, description, type, and validation rules of the field to properly prompt the user to enter the data.
Now consider how DB2 will find the values of the passwords to set. The IBM SmartCloud Enterprise platform populates the parameters.xmlfile with the values submitted by the user and places them in the instance in the /etcdirectory. In this case, your program must discover the values of the fields.
The JAXB xjc tool binds XML elements to Java objects using the W3C XML schema lan- guage. The command takes a package name and a schema file as input, as shown here:
xjc -p <packageName> parameters.xsd
Alternatively, you can use the IBM Rational Software Architect (RSA) Wizard for this. Add the schema file (parameters.xsd) to your project and right-click. Select Generate, Java. A wizard displays, as in Figure 4.1.
ptg8286219
Data Exchange 159
Figure 4.1 Rational Software Architect XSD to Java Wizard
Enter the package name and click the Finish button. This simple program makes use of JAXB to parse the DB2 parameters.xmlfile:
package com.ibm.cloud.api.parser; import java.net.URL; import java.util.List; import javax.xml.bind.JAXBContext; import javax.xml.bind.Unmarshaller; import com.ibm.cloud.api.parser.parameters.Field; import com.ibm.cloud.api.parser.parameters.Parameters;
public class ParameterParser {
public static void main(String[] args) throws Exception { URL url =
ClassLoader.getSystemResource(“data/parameters.xml”); JAXBContext jc =
JAXBContext.newInstance(“com.ibm.cloud.api.parser.parameters”); Unmarshaller u = jc.createUnmarshaller();
ptg8286219
//Display the parameters
for (Field field : parameters.getField()) {
System.out.println(“Field: “ + field.getName()); System.out.println(“\tLabel: “ + field.getLabel()); System.out.println(“\tDescription: “ +
field.getDescription());
System.out.println(“\tType: “ + field.getType().name());
System.out.println(“\tNo. values: “ + field.getValues().getValue().size());
if (field.getValues().getValue().size() > 0) { List<String> values =
field.getValues().getValue();
for (String value: values) {
System.out.println(“\t\t” + value); }
} } } }
The program assumes that the parameters.xmlfile is stored in the location /datarelative to the classpath of the class ParameterParser. It creates a JAXBContext, which it uses to unmar- shal the file. Then the program iterates over the fields, printing the name, label, description, type, and values. The output of the program appears next:
Field: db2_user_password
Label: Instance owner (db2inst1) password
Description: db2inst1 password. Must contain at least 1 number, at least 1 lower case letter, and at least 1 upper case letter.
Type: PASSWORD No. values: 1
db2cloud Field: db2_fenc_password
Label: Fenced user (db2fenc1) password ...
See [Vernier, 2011, “Convert IBM Cloud Image Parameters into Java Using JAXB”] for more details on downloading and parsing parameters.xml.
JavaScript Object Notation (JSON)
The bodies of REST requests and responses might be in different formats, depending on the choices of the service implementers. In IBM SmartCloud Enterprise, the responses can be returned in either XML or JSON format. JSON might be new to some users, so we cover it briefly. JSON can be simpler and more lightweight than XML, but it lacks some of the tight defi- nition of types that XML has. Another advantage of JSON is that it can be instantiated immedi- ately by JavaScript in a browser. Although not a formal standard, JSON is becoming a de facto
ptg8286219 standard for REST services. JSON is described by IETF Request for Comments 4627, “The
application/json Media Type for JavaScript Object Notation (JSON).” However, this document is informational, not a formal standard.
Several packages exist for streaming objects to and parsing JSON, including the IBM JSON4J library packaged with WebSphere Application Server 7.0, the json.org library, and the Google API. Following is an example of JSON from the IBM SmartCloud Enterprise:
{“images”:[
{ “name”:”IBM DB2 Enterprise Developer Ed SLES 9.7.1 - BYOL”, “id”:”20009984” “state”:1, “visibility”:”PUBLIC” }] }
This example describes an image in the catalog. We have deleted some of the attributes to keep it short and simple. An example program that uses the json.org library to traverse this JSON data is shown next:
package com.ibm.cloud.examples.rest;
import org.json.JSONArray; import org.json.JSONException; import org.json.JSONObject;
/**
* Class to demonstrate parsing of JSON data. */
public class JSONTraverser {
private static String DATA = “{‘images’:[“ +
“{‘name’:’IBM DB2 Enterprise Developer Ed SLES 9.7.1 - BYOL’,” +
“‘id’:’20009984’,” + “‘state’:1,” +
“‘visibility’:’PUBLIC’” + “}]}”;
public void parse() throws JSONException { JSONObject object = new JSONObject(DATA);
JSONArray images = object.getJSONArray(“images”); for (int i=0; i<images.length(); i++) {
System.out.println(“JSON array”);
JSONObject image = (JSONObject)images.get(i);
ptg8286219
String name = image.getString(“name”); String id = image.getString(“id”); String state = image.getString(“state”);
String visibility = image.getString(“visibility”); System.out.println(“Name: “ + name); System.out.println(“ID: “ + id); System.out.println(“State: “ + state); System.out.println(“Visibility: “ + visibility); } } /** * Entry point
* @param args Not used */
public static void main(String[] args) throws Exception { System.out.println(“Parsing JSON string”);
JSONTraverser parser = new JSONTraverser(); parser.parse();
} }
The program can be invoked from the command line. It creates a JSONObjectfrom the data, from which it extracts a JSONArray. The program then iterates over the array, instantiating a
JSONObject representing an image for each element in the array. The output of the program appears here:
Parsing JSON string JSON array
Name: IBM DB2 Enterprise Developer Ed SLES 9.7.1 - BYOL ID: 20009984
State: 1
Visibility: PUBLIC
This code is fine if you have a very simple task to perform. However, if you need to do something more substantial, you might need a more generic library, such as Google gson.
REST
REST APIs are provided as an important part of cloud computing platforms, both IaaS clouds and other types of clouds. However, REST is not specific to cloud computing, and you can make use of cloud computing without it. This section is provided to give readers basic background material on the topic so that they do not have to consult external sources to understand the examples in the book. Most of the material in this section is not specific to IBM SmartCloud Enterprise. The text also highlights areas that are particularly relevant to cloud computing.
ptg8286219
Background
REpresentational State Transfer (REST) is a programming style modeled on the principles of the World Wide Web. Ron Fielding introduced it with his PhD thesis “Architectural Styles and the Design of Network Based Software Architectures,” with the goal of replicating the scalability and success of the web for application programming. REST architecture is defined as having the following characteristics:
• Addressable resources. The key entity is a resource, and it must be addressable with a uniform resource identifier (URI).
• A set of well-defined interfaces for managing the resources. • Representation-oriented nature (say, with XML, JSON, or HTML). • Stateless communication capability.
REST is a simpler paradigm than other kinds of distributed programming models that did not enjoy widespread adoption, such as SOAP and COBRA. The simplicity of the REST pro- gramming model is the main reason for its widespread popularity today.
HyperText Transfer Protocol
We mostly refer to REST as implemented using HTTP, although that is not necessarily the case. Other web services protocols, such as SOAP, use HTTP only as a transport, but REST uses the methods of HTTP, including GET, POST, PUT, and DELETE. We could look at flow of text to an HTTP server with a program like this one that connects with a TCP socket:
import java.io.*;
import java.net.Socket;
public class ReadHTTP {
public static void main(String[] args) throws IOException { Socket socket = new Socket(“localhost”, 80);
PrintWriter out = new PrintWriter(socket.getOutputStream(),
true);
BufferedReader in = new BufferedReader(new
InputStreamReader(socket.getInputStream())); out.print(“GET /index.html\r\n\r\n”); String line;
while ((line = in.readLine()) != null) { System.out.println(line);
} out.close(); in.close(); socket.close(); } } REST 163
ptg8286219 The program is simplified to demonstrate the principle, but it ignores some best practices of
catching exceptions and closing streams in finallyblocks. The program connects to the server
localhostat HTTP port 80 and writes the text GET /index.htmlover the socket. This is a GET
request. The request ends with \r\n\r\nso that the server gets the blank line that it needs to end the HTTP request. Then it reads the response back from the server and prints it to standard out- put. Executing the program returns something like this:
HTTP/1.0 200 OK
Date: Fri, 11 Feb 2011 00:18:23 GMT Expires: -1
Cache-Control: max-age=0
Content-Type: text/html; charset=ISO-8859-1
<html><head>...
The HTTP response code is 200, which indicates that the response succeeded. A series of HTTP headers exist: Date, Expires, Cache-control, and Content-type. After the HTTP head- ers are printed comes a blank line, and then the HTML body is written out. The HTTP protocol is described in RFC 2616 [Fielding and Gettys, 1999], which includes the different request types, the HTTP headers, and the response codes.
REST Architecture
The idea that every resource must be addressable via a URI is a big help in simplifying program- ming models. Other programming models have often forced client programs to get objects for objects for objects before finally getting to the object that the client programmer is really inter- ested in.
The REST principle of a constrained interface is difficult for object-oriented programmers to get used to. Only a handful of methods are available for all the different operations you need to provide. The methods include the following:
• GET—A read-only operation that requests specific information.
• PUT—Stores the body of the message to the server. This is usually an insert or update. The operation is idempotent, which means that, no matter how many times you PUTan object, the result is the same. Just as when clicking the Save button of a desktop applica- tion, the file will be saved the same no matter how many times you click the button. • DELETE—Removes a resource.
• POST—Modifies the resources in some way. This is the only HTTP operation that is not idempotent. POSTcan also be used to create resources. It is the most open and extensible of the HTTP methods.
• OPTIONS—Enables querying of the server for capabilities.
ptg8286219 HTTP uses the Content-Typeheader to tell the server or client what type of data is encoded
in the body. This is a MIME type. Here is the format:
Content-Type: type/subtype; name=value; name=value ...
Consider these examples:
text/plain
application/xml;charset=utf-8
Using the Acceptheader, clients can tell the server what type of format they need: XML, JSON, or other. These headers can also be used for versioning.
The principle of statelessness helps applications scale and simplifies server implementa- tions as well. It is also natural. For example, a client makes a call, uses the returned data to make another call, and so on to compose an application.
Hypermedia as the engine of application state (HATEOAS) is an additional REST principle promoted by some proponents. The interpretation is that we should use full URLs when referring to resources within responses, not just an identifier. The HATEOAS engine of application state principle means that it is much easier to navigate from resource to resource using links than it is to know exactly how to construct the addresses for each resource in advance. For example, sup- pose that a search result returned too many results to manage in one request. The server could return the first batch with a link to the next batch. This works just like searching with a browser at popular search portals.
Implementing and Consuming REST Services
Many tools help developers implement REST in their applications. At least one of these is a stan- dard: JAX-RS is a set of Java APIs for development of REST services. It is defined by Java Spec- ification Request 311, “Java API for RESTful Web Services.” Earlier examples used PHP with the cURL library to invoke REST web services. Let’s look at some other ways of producing and consuming REST services.
Business Scenario: File Uploads
One of the most fundamental capabilities of IoT Data’s cloud service is enabling file uploads from devices. This section looks at how to do that. The Internet Engineering Task Force RFC 1867 Form-based File Upload in HTML defines a filevalue for the typeattribute of the HTML input element. It also defines a multipart/form-datavalue for the MIME media type. Example HTML code is shown here:
<form enctype=’multipart/form-data’ action=’DeviceServlet’ method=’post’>
<fieldset>
<label for=’devicefile’>File</label>
<input id=’devicefile’ ‘name=’devicefile’ type=’file’/>
ptg8286219
<input type=’submit’ value=’Upload File’/> </fieldset>
</form>
This HTML fragment generates a file upload widget, which looks like Figure 4.2.
166 Chapter 4 Standards
Figure 4.2 Business scenario: file upload
The Apache FileUpload project provides a utility for processing file uploads. The project depends on the Apache Commons IO library. Make sure that you have both JAR files in your classpath (WebContent/WEB-INF/lib). You can use this within the doPostmethod of the same servlet, as shown here:
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException { PrintWriter writer = response.getWriter(); Device device = getDevice(request); Customer customer = device.getCustomer();
writeTitle(writer, customer, “File upload confirmation”); DiskFileItemFactory factory = new DiskFileItemFactory(); ServletFileUpload upload = new ServletFileUpload(factory); factory.setSizeThreshold(2097152); // 2 MB
try {
List items = upload.parseRequest(request);
for (Object item : items) {
FileItem fileItem = (FileItem)item;
writer.println(“<p>File item: “ + fileItem.getName() +
“</p>”);
writer.println(“<p>Content Type: “ + fileItem.getContentType() + “</p>”);
writer.println(“<p>Size: “ + fileItem.getSize() +
ptg8286219
File uploadedFile = new
File(“/opt/IBM/WebSphere/files/” + fileItem.getName());
fileItem.write(uploadedFile); }
} catch (Exception e) {
writer.println(“Could not upload file: “ + e.getMessage());
e.printStackTrace(); }
writer.println(“<p><a href=’DeviceServlet?deviceid=” + device.getId() + “‘>Upload another file</a></p>”); writeFooter(writer);
}
The method first creates a DiskFileItemFactory object, which is used to create a ServletFileUpload object. A limit of 2,097,152 bytes is set for the uploaded files. The
ServletFileUploadobject parses the file uploads and puts references (handles) to them in the
FileItem objects. These are written out to the local file system under /opt/IBM/WebSphere/ files/. Figure 4.3 shows the confirmation.
REST 167
Figure 4.3 File upload confirmation
Writing an equivalent REST service is similar. One change to make, however, is that the URL should use a REST-style pattern line:
http://host_name/devices/234
To use a style like this, you can add a mapping in the web.xmlfile as shown:
<servlet-mapping>
<servlet-name>RESTServlet</servlet-name> <url-pattern>/devices</url-pattern> <url-pattern>/devices/*</url-pattern> </servlet-mapping>
ptg8286219 The HttpServletRequest.getPathInfo() method returns something like /234, from
which you can find the device record. Besides this difference, the server implementation is exactly the same as for a browser client, although you can ignore the HTML returned. You can invoke the REST API with a client, as shown in the uploadFilemethod:
package com.ibm.cloud.examples.iotdata.rest.client;
import java.io.File;
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.methods.multipart.*;
public class FileUploadClient {
private static final String BASE_URL =
“http://host:9080/CloudAPIClientWeb/devices/”;
public void uploadFile(int deviceId, String fileName) { System.out.println(“Uploading file: “ + fileName); HttpClient httpclient = new HttpClient();
HttpState state = new HttpState(); httpclient.setState(state);
PostMethod post = new PostMethod(BASE_URL + deviceId); File file = new File(fileName);
try {
Part[] parts = new Part[1];
parts[0] = new FilePart(“devicefile”, file); MultipartRequestEntity requestEntity = new
MultipartRequestEntity(parts, post.getParams()); post.setRequestEntity(requestEntity);
int statusCode = httpclient.executeMethod(post); System.out.println(“Response code: “ + statusCode); } catch (Exception e) {
System.err.println(“Error executing HTTP request: “
+ e.getMessage()); e.printStackTrace(); } finally { post.releaseConnection(); } } 168 Chapter 4 Standards
ptg8286219
public static void main(String[] args) {
FileUploadClient client = new FileUploadClient(); client.uploadFile(1,
“d:/temp/air_quality_sanfernando.gif”);
} }
This program uses the Apache HTTPClient 3.1 library. The critical part of the program is the use of the MultipartRequestEntity class to add a FilePartto the request body. If every- thing goes well, you should see output similar to this:
Uploading file: d:/temp/air_quality_sanfernando.gif Response code: 200
Response code 200indicates that the request was successful.
Example: Uploading Files When Creating Instances with REST
In the IBM SmartCloud Enterprise “Java API” section in Chapter 3, you saw how to upload files using the Java API. Now that you understand more about the HTTP protocol and Apache Http-