Autonomic Cloud Computing!
Dominick Lam Noah Lee
[email protected] [email protected]
Dennis Park Juilee Malavade
[email protected] [email protected]
Abstract
The study of fluid dynamics requires the generation of multiple simulations for attributing certain properties of fluids. Producing these images on a single computer usually takes an inordinate amount of time. To reduce the amount of time, a different system of processing data must be used. This system is known as cloud computing. CometCloud, a cloud computing framework developed by the Rutgers University National Science Foundation’s Center for Autonomic Computing, has been developed to run computationally intensive applications. In this case, the Fluid Flow Application simulates the flow of a fluid in a microchannel using CometCloud. To access the application, a webpage was created. Users enter parameters of the fluid on the web interface front end, calculations are performed in the CometCloud back end, and results are displayed on the front end. Creating a comprehensive cloud-based application involves integrating Rutgers' CometCloud with a user-friendly front-end for scientists to easily input parameters and receive output reasonably quickly.
1. Introduction
The field of fluid mechanics relies on a broad range of computationally intensive equations. These equations are used to model how a fluid behaves as it flows through a channel with variable conditions. The complexity and volume of these equations demand powerful computing resources to generate these simulations in a useful amount of time. In particular, computer-generated flow simulations require the input of multiple parameters into an application to help scientists determine specific properties of specific fluids.
To calculate such properties, thousands of simulations1 must be performed and compared to find an optimal match for certain flow characteristics. For example, a simulation can be used to reproduce a cancer patient's bloodstream and find an optimal environment to separate the cancer cells from
Figure A. An example of a fluid flow simulation.
the bloodstream, inhibiting the cancer cells from spreading. In this case, the speed of the calculations is vital to potentially life-saving research. These calculations and simulations such as that in Figure A are performed by the Fluid Flow Application, developed by Rutgers researchers.
Because of the immense amount of processing power the application would require, the task cannot be performed on a single computer. Alternatively, a cloud structure could be used to distribute tasks from one computer to several others to calculate fluid properties. With this process, scientists can generate and compare simulated images across servers worldwide to receive precise data in one application. Cloud computing effectively reduces time by simultaneously performing calculations across many computers, cutting storage space by consolidating images on the cloud instead on a single computer, and enabling a sustainable exchange between third party resources and the main server.
2. Background
In the 1950s, cloud computing was only a concept known as time-sharing. This was when mainframe computers were first being made available in academia and corporations. Tools known as terminals were used to access these computers. Terminals lacked any computational capabilities and were used solely as a means to communicate with the mainframe computers. By allowing multiple people to utilize a single terminal at one time, these users were able to share the time that they spent on the computer, thus the name time-sharing. Since then, cloud computing has grown into a revolutionary concept that anybody can use to call up
third-party resources in order to create timely and cost-efficient results.
A single computer does not have the computational capacity to take the users’ given parameters and produce the results that are expected in a reasonable amount of time.2 Such a task could take days, or even weeks. Before the inception of cloud computing, supercomputers were accessed by terminals. The modern cloud engine, however, is accessible by a simple web interface. The cloud engine calls up third-party resources such as supercomputers so that they can work in tandem to create the visual simulations of the micro-fluid.
2.1 Cloud Computing
The importance of the cloud has become much more relevant in today’s increasingly complicated world of technology. Known for both its storage space and computationally-intensive properties, the cloud system plays an important role in the creation and implementation of applications. One benefit of cloud computing is the system’s ability to prevent any waste of resources. The cloud supports a usage-based payment-model that allows its users to rent virtual resources and pay for what they use. In the past, people had to purchase software and hardware to run their application. The problem with this method was the possibility of over- or under-use. Sometimes, people would overestimate what they actually needed to run their computer applications and they would end up having spent more money on resources than they needed to. Today, using the cloud, people can buy virtual resources as the need arises. Therefore, the chance that there may be unused resources is minimized and costs are reduced.
The flexibility of the cloud with its availability of virtual resources makes it useful for dynamic applications, which do not have a known resource requirement. The amount of virtual resources, multiple supercomputers or servers for example, which are necessary to run the application is ambiguous throughout the execution of the application. Through the third-party resources that the cloud makes immediately available, applications with fluctuating resource requirements are a perfect fit for the cloud system.
2.2 Sockets
Like any other cloud system, the cornerstones of the Fluid Flow Application are connections. The cloud is a multitude of machines working together to accomplish a single goal in a timely fashion by dividing it into many smaller goals. Therefore, each machine must somehow communicate with the others to divide and perform the computations. The medium of communication is known as a socket connection, which is made between two entities known as a server and a client. The data relayed between the server and client is possible due to socket connections.
Two sockets, each consisting of an IP address and a port number, can form a communication link with each other, enabling transmission of data. The server constantly listens on the assigned port, waiting for a connection request from a client. The server typically only runs on a single computer, handling all client requests to that socket. The client, when run, sends a communication request to the server socket. If the server is running and listening on the correct port, a connection is established and
the server and client are able to exchange information.
2.3 CometCloud
CometCloud is the specific cloud system that was used in the Fluid Flow Application. Created at Rutgers University, CometCloud has the capacity to support multiple programming languages to operate and manage dynamic applications. In addition, CometCloud supports cloud infrastructures to make very computationally intensive programs such as Fluid Flow function smoothly and efficiently. The Rutgers CometCloud system also has the ability to communicate with other clouds to coordinate how to handle dynamic workloads, changes in demands, and other extensive problems.
In the Fluid Flow Application, the cloud system incorporates a master-worker relationship in which the Java server sends input to the master. The master divides the entire problem into small tasks. It distributes tasks to the supercomputers, the workers, case to carry out the task of computing thousands of simulations that the Navier-Stokes equations will generate. It should be noted that the supercomputers are within in the cloud system and that they are not separate entities from CometCloud.
2.4 Navier-Stokes Equations
The Navier-Stokes equations are a series of mathematical equations which describe the flow of fluids. Developed by Claude-Louis Navier and George Gabriel Stokes, they are vital to various scientific and medicinal studies. Calculating specific properties of fluids allows scientists to
determine how they will behave in controlled environments.
The Navier-Stokes equations determine values such as velocity, pressure, temperature, and density of a moving liquid3, which can be used to create images such as those in Figure B. These images display the change in certain fluids due to the variables entered into the equations. When run multiple times, the equations can generate a database of hundreds of images.
However, in order to create such a large database, vast numbers of calculations must be performed. Due to its complexity, these processes would take weeks to execute on an ordinary machine. Because of their time-consuming calculations, the Navier-Stokes equations have been implemented into CometCloud in order to cut down run time by multiple factors.
The Navier-Stokes equations rely on multiple variables in order to compute results. For the Fluid Flow Application, microchannel height, pillar location, pillar distance, and the Reynolds number are used. The variables are properties of the
microchannel which the fluid flows through and the pillars placed within the microchannel. Each of these variables will contribute to changes in the flow of the fluid through the channel. Changing these in the program will allow the user to simulate different flows of fluids.
3. Creating a Cloud Application
To develop a complete application, several phases of development took place. Initially, socket connections were developed and tested to ensure data could be exchanged between the front end and back end of the application. The user interface was then developed and combined with the socket connections to ensure that data entered by the user could be sent to the back end. Additionally, computing resources were made available to ensure that computations could actually be performed. The entire system is depicted in Figure C.Figure B. Examples of fluid flow simulations. The left figure shows two different placements of pillars in a fluid. The right figure shows multiple fluids mixing.
3.1 Application Overview
Using HTML, a web application was made so users could input their data and retrieve their results with ease and celerity. The web application was a constant work in progress and its appearance changed over time as shown in Figure D.
After integrating the master server, worker servers, and user interface, a useable cloud system is formed. With this system, users simply need to enter parameters onto a single web page to run computationally intensive methods in a matter of minutes. The cloud allows for efficiency in calculations which otherwise would require weeks to do by one supercomputer. The only requirement for a user to access the Fluid Flow Application is an internet connection, which today is nearly ubiquitous. The Fluid Flow Application reduces processing time by using the cloud, but remains easy to use.
3.2 Socket Connections
For this project, the two connections that were experimented with each involved a Java server. However, two different clients were used, one coded in Java and the other in PHP. Ultimately, the PHP client was used for the Fluid Flow Application because it was more compatible with HTML, the basic code used to create web pages and applications.
In this case, the PHP client transmits the parameters of the Navier-Stokes equation as input to the Java server via a socket connection. The server passes this data onto the Fluid Flow Application on the CometCloud framework via another socket connection. After performing the calculations, the Fluid Flow Application sends the results back to the Java server, which then sends them back to the PHP front end. The sockets allow the different
1. User inputs sent to Java server.
2. Java Master divides calculations
into small l tasks
3. Worker pulls task from pool
4. Worker completes task
5. Worker returns task to master server.
6. Master compiles tasks into complete results.
7. Master returns results to front end. 8. Front end displays results.
Figure C: The diagram above models the flow of data from the first step of inputting the parameters to the process of the exchange from the master server and external servers back to the web interface.
platforms to communicate via the Internet to exchange information.
3.3 User Interface
Running applications on the CometCloud framework requires a web-based interface. HyperText Markup Language (HTML) is used to create and display virtually all web pages, so it was used as the foundation of the web-based interface for the Fluid Flow Application. This allows for the creation of features such as headers, columns, and tables which make up the basic layout of the webpage. Additionally, Cascading Style Sheets (CSS) was used to further style and format the webpage, such as adding the background and changing the font size and color.
Figure D. Phases of the design of the web interface.
Apart from the format and style of the webpage, some functionality was provided by PHP Hypertext Preprocessor, simply referred to as PHP. This language provides the ability to include the parameter input boxes, the resource checkboxes, and the submit button. The PHP code is contained within specific HTML tags which allow it to collect user input. It stores this input in variables and transmits it to the Java server for processing. The development of the web application is depicted in Figure D.
3.4 Resources
High Performance Computer (HPC) resources were used to execute certain files and calculations that could not be carried out on normal machines such as household computers. For example, the clients used in the application are created in PHP and such a file type requires a server to function. Rutgers University provided a resource known as the Amazon EC2 Machine which acted as the server that allowed the PHP file to run and form a connection to a running Java server. HPC Resources that were used to run the Fluid Flow Application include the Excalibur and Snake machines hosted by Rutgers University and the machines India, Sierra, Hotel, located in Indiana, San Diego, and Chicago respectively. Unlike PHP files, other files do not require the use of external servers and can be run on simple machines. For example, the experimentation with the connection between a Java server and Java client was conducted on a basic computer.
Two operating systems were used for the experimentation of socket connections: Linux Ubuntu 7.3 and Windows 7. Ubuntu, a form of Linux, was run on a virtual machine program known as VMWare Player within Windows because the Linux terminal yields easy access to the Amazon EC2 Machine,
allowing files to be transferred to the machine with simple commands. Linux terminal is a type Command Line Interface where a user interacts with a computer. Windows 7 was used to execute the socket connection between the Java server and Java client. Meanwhile, the coding and testing of the Java server and the Java client were both performed within the integrated development environment Eclipse.
3.5 Autonomic Cloud Computing
The foundation of Cloud Computing is the ability to manage pending tasks and available resources. This ability is the power to assign the tasks to the available resources, optimizing the system in various aspects. Speed and cost are possible optimization factors.4
Speed can be determined by recording the time at which the first task of a job begins and the time at which the last task of the job completes. The master server can optimize for speed by using faster computers and having many tasks running across many computers simultaneously, reducing the total
amount of runtime as seen in Figure E. The figure shows that. Using the cloud, six tasks can be computed in five minutes. If these six tasks were computed on a single server, it would take fifteen minutes, three times the amount of that of the cloud. However, this may require the purchase of more expensive resources over a shorter period of time. Users may wish to optimize for speed if the data is time sensitive.
Users may instead choose to optimize for cost. To achieve this, the master uses the cheapest computers available, regardless of the speed or availability. It may choose to allow tasks to sit in a queue unperformed while waiting for less costly resources to become available.
4. Results and Discussion
Using various languages, protocols, and frameworks, an application was created to easily allow users to access HPC resources in the cloud. This allows scientists and researchers to run models and calculate data in relatively short periods of time.
4.1 The Fluid Flow Application
The HTML/PHP front end, Java server, and CometCloud back end function as a whole to facilitate fluid dynamics research. Scientists enter parameters for microchannel height, pillar location, pillar diameter, and Reynolds number into a graphical user interface coded in HTML and PHP, known as the front end of the application. Scientists can also select which resources to use for computations. The data is transmitted to a Java server which relays it to the Fluid Flow Application running on the CometCloud framework. Within the framework, a master
Figure E. Time optimization chart. In the same amount of time it takes for the longest task to run on Server 1, the master can append shorter tasks to other servers to minimize total time consumption.
server divides up the computations to be done into smaller tasks and distributes the tasks to worker servers. The results of individual tasks are returned to the master server, which compiles them into a comprehensive dataset and returns it to the front end for viewing or download by the user. Using many external processors concurrently, scientists can quickly and efficiently compute a wide range of different problems, each representing a combination of different speeds, thicknesses, heights or offsets. The simulations can then ultimately be used for a variety of conveniences and applications.
4.2 Discussion
Use of the cloud-integrated web application has a substantial advantage over individual use of several computing resources. The web interface of the Fluid Flow Application enables users to enter parameters and select resources to use. Without the cloud, users would have to input the parameters for each computation into each supercomputer individually, taking up a copious amount of time and valuable resources. Having the user interface integrated with CometCloud allows a potential client to input parameters and distribute the calculations on a global scale, at a high speed. Thus, thousands of simulations of fluid flow dynamics can be computed and analyzed for research.
The use of cloud computing for scientific applications is most appealing to uses in which the calculations involved would take an inordinate or virtually infinite amount of time to do by hand. It is most vital for urgent and potentially life-saving medical research and development. One such example is the study of cancer cells in blood flow, manipulating the fluid to separate
healthy white blood cells from diseased cells. The Fluid Flow Application has the ability to quickly allocate resources, transfer large amounts of data, and run multiple instances of the same application.
5. Conclusion
A single computer producing thousands of fluid flow simulations is incredibly inefficient and time-consuming. To reduce the time required to display the data that was inputted into the Navier-Stokes equations, a cloud system was employed. By using multiple supercomputers and the ability to distribute tasks among a network of workers, CometCloud has optimized the analysis of fluid flow in microchannels in terms of workload per computer and time to process.
Another goal was to create a user-friendly interface. Using the programming languages HTML, CSS, PHP, and Java, a web page was constructed to allow its users easy access to supercomputers. A task that would have taken one computer weeks to accomplish took the cloud system minutes to achieve. Consequently, users would receive results soon after their request.
Although efficient, the Fluid Flow Application can be further improved upon. Unlike the study on time, there has been no cost analysis, and therefore it is not known whether or not the cloud system reduces cost. This element will be inspected in the future to see if the cloud truly does maximize profit by minimizing cost.
Another direction to consider taking in the future is polishing the user-interface. It was created with only a basic knowledge of HTML and CSS. It can be augmented using more sophisticated code techniques to create
a more aesthetically pleasing and interactive user interface.
The Fluid Flow Application shows that the cloud has potential to be extremely practical and beneficial. Modern society has already implemented the cloud in many technological services; some examples include Google Drive, iCloud, and Dropbox. Consumer use of the cloud has experienced rapid growth in recent years. This shows the growing importance of cloud computing and how it will continue to develop over time and expand its use into other disciplines.
Acknowledgements
This creation of the Fluid Flow Application required the work of many others besides that of our own. Numerous people such as those from Rutgers University gave an immense amount of help when obstacles were encountered. Without their help, the application would have never been finished. We would like to specifically thank Dr. Javier Diaz and Dr. Ivan Rodero for their guidance and sharing their boundless knowledge of computer science with us to accomplish this feat. We would also like to thank Dr. Manish Parashar for directing the course of this project. Without him, this project would not have even been possible. In addition, we would like to thank Neha Desai for being such a contributive project RTA, helping us organize our workload and edit our papers. Next, we thank the NJ’s Governor’s School of Engineering and Technology and its director, Jean Patrick Antoine. Lastly, we would like to thank all the sponsors of the Governor’s School program.
References
1
J. Diaz-Montes, B. Ganapathysubramanian, I. Rodero, M. Parashar, Y. Xie, J. Zola, Rutgers Discovery Informatics Institute, Federated Computing for the Masses: Understanding Fluid Flow, http://nsfcac.rutgers.edu/CometCloud/uff/
2J. Diaz-Montes, M. Parashar, I. Rodero, J.
Zola, B. Ganapathysubramanian, Y. Xie, HPC in the Cloud, CometCloud: Using a Federated HPC-Cloud to Understand Fluid
Flow in Microchannels, http://www.hpcinthecloud.com/hpccloud/20 13-05- 22/cometcloud:_using_a_federated_hpc-cloud_to_understand_fluid_flow_in_microc hannels.html
3Glenn Research Center, National
Aeronautics and Space Administration,
https://www.grc.nasa.gov/WWW/k-12/airplane/nseqs.html
4News Staff, Science 2.0, Sculpting
Tailor-made Fluid Flows with Microfluidic Channels,
http://www.science20.com/news_articles/sc ulpting_tailormade_fluid_flows_microfluidi c_channels-116279, 12 July 2013
5H. Kim, Y. el-Khamra, I. Rodero, S. Jha, and
M Parashar, The Applied Software Systems Laboratory, Rutgers, The State Univeristy of New Jersey, Autonomic Management of Application Workflows on Hybrid Computing Infrastructure, June 1, 2011, http://www.cac.rutgers.edu/TASSL/Papers/s pj2011kim.pdf
