• No results found

Processing Large Data Sets using a Cluster Computing Framework

N/A
N/A
Protected

Academic year: 2021

Share "Processing Large Data Sets using a Cluster Computing Framework"

Copied!
5
0
0

Loading.... (view fulltext now)

Full text

(1)

ISSN 1991-8178

Corresponding Author:Imran Sarwar Bajwa, Department Of Computer Science & IT, The Islamia University of Bahawalpur, Pakistan.

Processing Large Data Sets using a Cluster Computing Framework

1

Imran Sarwar Bajwa,

2

Ahsan Ali Chaudhri,

3

M. Asif Naeem

1

Department Of Computer Science & IT, The Islamia University of Bahawalpur, Pakistan

2

Director of Academic Programs, Queens Academic Group, Auckland, New Zealand

3

Department of Computer Science,University of Auckland, Auckland, New Zealand

Abstract: Increase in the scientific disciplines has caused large data collections as important community resources. The volume of interesting data is already measured in terabytes and will soon total in peta-bytes. This research proposal presents the issue of processing massive amount of satellite data. A single LEO satellite sends around 2 GB of data in 24 hours of a day. To process this huge amount of data, normal digital computers face constraints like processing time, recourses and cost. A solution is needed that can provide quick way of processing at low cost. Cluster computing is network based distributed environment that can be a solution for fast processing support for huge sized jobs. A middle-ware is typically required in cluster computing. In this proposal a middle-ware is proposed for handling the existing processing problems in distributed environments. In a typical heterogeneous computation, a middleware can be employed to provide incorporation and interoperability in the underlying applications and services.

Key words: Distributed Computing, Middleware Design, Computing cluster, Satellite data processing INTRODUCTION

In recent times, the need of perforamne and availablity based computation for gigantic sized processing jobs has emerged cluster computation into a future technology. In clusters based computing, high performance computation (Bayardo et al.,). A computer cluster is a set of phisically connected computers working together like a single machine. The major purpose of using cluster is to provide high computation power for processing intensive applications such as weather foreasting, information retrieval, virtual reality, distributed randering, etc. However, a typical concern in computing clusters is the architetural setting such that coupling level (loosely coupled or tightly coupled) of the individual nodes. Possible architectural styles for cluster computing can be client-server, n-tier, tightly coupled, peer-to-peer, and space-based (Nodine et al., 1997). Grids are other type of distributed computing but more focused on throughput like a computing utility rather than running fewer, tightly-coupled jobs. In this research paper, we are focused on cluster computing.

A distributed computation infrastructure such as cluster computing can be useful in fast and reliable proessing of gigantic sized data with the help of integrated and collaborative employment (http://www.anderson.ucla.edu/faculty/jason.frand/teacher/technologies/palace/index.htm) of the geographically separated, autonomous resources. Multiple computing resources are linked up together in a computer cluster (including processors and storage devices) to constitute a larger, more powerful single virtual computer. A computer cluster typically provides with a variety of benefits (Nodine et al., 1997; Duschka and Genesereth, 1997) as following:

a. Cluster computation can provide a high speed computation facility b. Clusters can making individual applications run much faster.

c. Clusters are cost effective solution as they allow multiple applications to share the available computing resources.

d. The clusters also make better use of all available computing resources.

e. A layer of abstraction is provided by clusters by removing knowledge of IT infrastructure from applications.

f. The clusters provide flexibility to add more resources to a cluster as needed.

(2)

wireless devices, grid and cluster based computing, etc. (Rajkumar Buyya, 1999) The main purpose of using a middleware is to achieve a set of key functionalities such as recognition, authentication, security, authorization, soft-switching, and certification. (Jeffrey and Layton, 2009) In this paper, we have introduced a middleware for better resource utilization in typical computer cluster for processing satellite date. Our presented architecture for a middleware in a cluster computer has two main layers. One layer consists of computational cluster which is responsible for performing not only all analytical computation but also capable to manage knowledgebase with it. (Zdzislaw Meglicki, 2004) This approach is used so as to shorten the response time for user. The other layer is data cluster, which offers huge storage capacity. The data can be stored here in any format.

1.2. Satellites and Meteorological Data:

In this paper, our major focus is the fast processing of the meteorological data. The metrological satellites are common source of such data. Typically, the global system of meteorological satellites consists of set of five satellites. All five satellites are evenly spaced around the equator in geostationary orbit, and at least two satellites in near-polar orbits. Typically, following three orbits:

a. Low Earth Orbit (LEO):

This is one of the three categories and this orbit is nearest to earth (100 to 300 miles), called low earth orbit (LEO). Satellites in LEO travel very fast as in slow speed they may be pulled out of orbit due to gravity and can be crashed into the earth. The satellites in LEO typically have a fast speed of 17,500 miles per hour (see figure 1) and can complete the earth circle in an hour and a half. It means they normally complete 8 revolutions of earth in a day but at different angles.

Fig. 1: Satellite Orbits

b) Medium Earth Orbit (MEO):

This is the second orbit and little bit higher (6,000 to 12,000 miles) as compared to LEO, called MEO satellites. The orbit shape of MEO satellites is oval.

c) Geostationary Earth Orbit (GEO):

Satellites in these orbits are at the highest (22,282 miles above the earth) position as compared to other two orbits, LEO and MEO. In a GEO orbit, there are typically five satellites in a set those cover the whole earth and the GEO satellites complete their one circle of earth in 24 hours.

Problem Statement:

Increase in the scientific disciplines has caused large data collections as important community resources. The volume of interesting data is already measured in terabytes and will soon total in peta-bytes. Processing massive amount of satellite data is an open question. A single LEO satellite sends around 2 GB of data in 24 hours of a day. To process this huge amount of data, normal digital computers face constraints like processing time, recourses and cost. Supercomputing can be a solution but it is quite expensive. Cluster computing with the help of middleware can be low-cost and appropriate solution.

(3)

The rest of the paper is structured as section 2 describes the design consideration of a computer cluster; section 3 explains framework of computer cluster for satellite data processing; section 4 presents the workflow of the proposed system; section 5 discusses the related work to the presented research and the paper ends with a conclusion section.

Cluster Design Considerations:

A typical cluster is a composed of a set of individual computing machines but these machines do work like single computing machine. However, with respect to the computing architecture of client/server a computation clusters, there can be of two types of computing machines such as cluster nodes and cluster

server. (Cousins et al., 2006) Here, the cluster server is the real manager of the whole communication going on among all cluster nodes. Moreover, the cluster server not only manages the way processing tasks are divided among the cluster nodes but also systematic distribution of the processing workload to each cluster node. (Rajkumar Buyya, 1999) In typical client/server architecture, the main responsibility of a cluster node is to processes an assigned job and sends the results either back to the cluster server or directly to the particular client who requested the result.

On the basis of the computation type, there can be two types of the clusters such as a data cluster and a computational cluster. Major responsibility of a computational cluster is to manage the sharing of distributed computational power while data cluster’s key task is to manage the sharing of distributed storage. Following are the major design considerations for a typical computing cluster (Nwana et al., 1999):

C Typical architecture of cluster computing contains vendor supported blade/ rack system of pile of PCs. C Cluster software is a middle ware for cluster distribution software. This middleware is capable of

self-configuration.

C RAID Technology is used in cluster computing. RAID technology helps reducing CPU time up to 30%. C Particular file systems are required in a computing cluster i.e. Andrew File System (AFS) and Network

File System (NFS).

C Efficient interconnect mechanism is required for distributed computing i.e. Gigabit EthernetMessage Passing is also significant phase and TCP/IP can be helpful in this phase. Message Passing Interface (MPI) is another tool.

C Testing is the last phase.

Framework of Satellite Data Processing:

In this section, a typical framework of middleware has been proposed for cluster computing to process satellite date. Satellite data is received from satellite is handed over to the application layer. In typical architecture of a computing cluster, the application layer is the top most layer and this layer receives the user’s request to be processed. However, the user request can be a computational or storage related job. The second layer is middle layer, which is further divided into its sub components (Cousins, et al., 1996; Rajkumar Buyya, 1999). The framework is shown in figure 2:

Job Management Systems:

Job Management Systems is responsible to take up the job and distribute it over network. Cluster Management Tools (CMT) provides us tools that are needed to perform maintenance and other operations on cluster. Parallel execution libraries are available in which user can program their applications.

Cluster Management Tools:

Cluster Monitor (CM) is tool that performs monitoring of cluster node’s status and their work load. Global Process Area provides space to accumulate results from nodes in server system. Moreover some other tools may be presented at this middle layer. The third (lowest layer), consist of nodes that performs the execution of jobs assigned by the server.

Parallel Execution Libraries:

In a typical cluster, among job management systems and cluster management tools, a set of parallel execution libraries are provided to the application developers. The major responsibility of these libraries is to provide a set of operations and methods to acquire parallelization and synchronization among different processing tasks (jobs). The processing tasks/processes can be under execution on the local machine or a remote machine. A few examples of such libraries provided for parallel computation are MPI and PVM.

(4)

Besides such parallel libraries, different programming languages can also be used to provide the facility of parallel computation in developed applications.

Cluster Monitoring:

During the distributed computing in a cluster, a cluster management system is deployed to provide complete control over the underlying cluster nodes. The cluster management system enables the administrator of the cluster to recover from error conditions or to configure the cluster system. Various cluster monitoring systems are available such as automatic installer is an important tool typically used for the said purpose. One of the major implications of using such tools is installation of various software and applications stack on the remote computers (cluster nodes).

Fig. 2: A framework of computational and data cluster for satellite data processing

Global Process Area:

This layer is the base of whole computation processing going on in a typical cluster. Major emphasis of this layer is to provide a global process space and to control all the running processes on any particular node in a cluster. This global process space not only enables brawny synchronization in running processes but also enables the cluster to identify processes requiring a certain support for communication and provides a particular messaging mechanism to share data among different processes.

Work Flow of System:

User query for certain information, this query is taken up by computational cluster’s first layer i.e. top most layers that are ontology services. These services are then treated by Job Management System that takes the refined job then make its independent subtasks and then these sub tasks are distributed among its multiple nodes. Here the process of recursion may be involved to break the major problems into sub tasks. These jobs are distributed over all the nodes of computational cluster. Multi and distributed agents verify the required knowledge from knowledge based Replica and knowledge based cache. If the required result is found here at computational cluster then there is no need to inquire the data cluster. If the required information does not exist here, then query is refined through agents. This refined query is forwarded to data cluster. Data is extracted from data cluster and the result set is returned to KRE in Global Process Area. KRE includes important data mining steps like cleaning, transformation, data mining query, pattern matching and extraction of knowledge like modules resulting as wisdom (knowledgebase). In this way the knowledgebase is also updated and synchronized through KRE.

(5)

Literature Review:

Proposal of such a gigantic enterprise influenced me to investigate “Cloud Computation”. As I started with the record of the underlying concept that was actually tossed in 1960 (Bayardo et al.,) by John McCarthy when he predicted the term “Public utility". First mutual practice of the term “Cloud” as a symbol for the internet in a document was published by MIT in 1996 (Nodine et al., 1997). At the beginning of new century IBM Started to deploy new computing concepts (e.g. Grid computing, pervasive computing) in applications and services. Other organizations declared related projects around the same era. In 2005, Amazon utility computing based services gave computing new direction by modernizing their data centers after dot-com babble. (Duschka and Genesereth, 1997)

In 2007, Google, IBM and other big giant computing companies started to invest on large scale cloud computing research project, (Nwana et al., 1999) In mean while the term was urged by mid-2008 it became a hot topic, gained fame in press and several organized events occurred. (Cousins et al., 1996) A common type of clusters is industry cluster. Typically, the industry clusters can assist in clarifying the conflicting influence concerning the underlying methodology used in various clusters. A similar approach to identify a cluster is based on quantitative techniques, including and input-output analysis and location quotients. Such approaches and tools are typically useful in recognizing qualified applications of particular industries.

Conclusion:

In-time information retrieval is the most important issue when the amount of data is increasing so fast. Data mining is wonderful tool for accessing accurate and quick data sets. Still this is insufficient for huge data and when time is most important. So Implementation of Cluster is the ultimate solution. We have proposed a framework with combination of data cluster, computational cluster and data mining algorithms, which will definitely produce very effective and improved efficiency.

ACKNOWLEDGMENTS

This is the extended version of our own paper presented and published as Conference proceedings in “International Conference on Computers & Emerging Technologies” (ICCET 2011) held on 22-23 April 2011 at Shah Abdul Latif University, Khairpur, Sindh, Pakistan.

REFERENCES

Bayardo, R.J., Jr., W. Bohrer, R. Brice, A. Cichocki, J. Fowler, A. Helal, V. Kashyap, T. Ksiezyk, G. Martin,

Cousins, S., H. Garcia-Molina, S. Hassan, S. Ketchpel, M. Roscheisen and T. Winograd, 1996. Towards interoperability in digital libraries. IEEE Computer, 29(5).

Duschka, O.M. and M.R. Genesereth, 1997. Query planning in Infomaster. In Proceedings of the 12th A n n u a l A C M S y m p o s i u m o n A p p l i e d C o m p u t i n g ( S A C ’ 9 7 ) , . http://logic.stanford.edu/people/duschka/papers/Infomaster.ps.

http://www.anderson.ucla.edu/faculty/jason.frand/teacher/technologies/palace/index.htm

Jeffrey, B. Layton, 2009. CAOS NSA and Perceus: All-in-one Cluster Software Stack, Linux Magazine, February 5th 2009, online article at: http://www.linux-mag.com/id/7239, last accessed in May.

Nodine, M., M. Rashid, M. Rusinkiewicz, R. Shea, C. Unnikrishnan, A. Unruh and D. Woelk, 1997. Infosleuth: agentbased. semantic integration of information in open and dynamic environments. In Joan Peckham, editor, Proceedings of the ACM SIGMOD international conference on management of data, SIGMOD Record, 26(2): 195-206.

Nwana, H., D. Ndumu, L. Lee and J. Collis, 1999. ZEUS: A toolkit for building distributed multi-agent systems. Applied Artifical Intelligence, 13(1): 129-186.

Rajkumar Buyya 1999. (editor). High Performance Cluster Computing: Architectures and Systems, Vol. 1. ISBN 0-13-013784-7, Prentice Hall PTR, NJ, USA,.

Rosenfeld, Stuart A., 1997. “Bringing Business Clusters into the Mainstream of Economic Development.” European Planning Studies., 5(1): 3-23.

Zdzislaw Meglicki, 2004. High Performance Data Management and Processing, Document version Id: I590.tex, v. 1.197. Section 7462, Course notes, University of Indiana, USA, http://beige.ucs.indiana.edu/I590/

References

Related documents

If you have already submitted a proposal as an applicant for a research grant and have received a letter informing you of the funding decision, or if you have led an independ-

those with shorter HP tend to fall the right and b) phase of the flood cycle. Diatom taxon richness plotted against hydroperiod class. Invertebrate taxon richness and abundance

In this study, we compared seven frequently used CNV detection methods: circular binary segmentation (CBS) (8), CNVFinder (9), cnvPartition, gain and loss of DNA (GLAD) (7),

Madill

Our discussion with this group focused on issues that are important to them concerning online education, what their drivers and motivations for pursuing an

The different, oxalic acid based control resulted in significantly higher mite mortality.. Other technological factors may have important role in the control

The paper brings several contributions: i) We model a multi-data center virtual machine placement problem with quality of service guarantees for the application; ii) The

In this study, we have followed the saddle-point dynamics based distributed ESC (DESC) scheme proposed by Ye and Hu (2016), and performed simulation study with the Modelica