Online Workflow Scheduling System Using
Grid Computing
1
G. Athira Krishna, 2 N.Sumathi
1
Reseach Scholar, SNR SONS College, Coimbatore, Tamil Nadu
2
Head, Department of Information Technology, SNR SONS College, Coimbatore, Tamil Nadu
Abstract - One computer, usually a server, which handles all the administrative duties for the system is considered to be the control node in a grid . In grid computing, every server is mentioned in terms of nodes. The most important work of grid computing is to give resources to each node as per the requirement and divide the work of grid into single jobs, then schedule jobs for each node. This work aims at building a grid system which does the function of monitoring of the status of the resources and creates prediction models based on the negotiations made by the clients. This article presents the design and evaluation of system architecture for grid resource monitoring and prediction with a new model of service provisioning. This evaluation is done inheterogeneous distributed systems on users negotiating with service providers on their required Quality of Service (QoS) and on the corresponding price to reach a Service Level Agreement (SLA).The model is initiated by proposing a hybridization algorithm, Parallel Hybrid Particle Swarm Optimization(PH-PSO).
Keywords - Grid Computing, Particle Swarm Optimization, Workflow , Scheduling, Machine Learning Strategies.
1. Introduction
A Grid Computing system is a collection of distributed computing resource available over a local or wide area network that appears to an end user or application as one large virtual computing system. It is an approach to distributed computing that spans not only locations but also organizations, machine architectures, and software boundaries.
Scheduling is the method by which threads, process or data flows are given access to system resource. This is usually done to achieve load balance and share system resources effectively or achieve a target quality of service.
The need for a scheduling algorithm arises from the requirement for most modern systems to perform multitasking and multiplexing. This project uses PH-PSO algorithm to give a better satisfaction to grid users. In the grid system, an end user submits the job that has to be executed with some constraints like job execution deadline, cost for the execution and the time required for the execution. Grid resource manager estimates the resource requirements and provides the functionality for discovery and publishing of resources as well as scheduling, submission and monitoring of jobs. Thus different performance goals also play great impacts on the design of scheduling systems. Desirable performance goals of grid scheduling includes: maximizing system throughput, maximizing resource utilization, minimizing execution time, minimizing cost on the user side and fulfilling economic constraints.
The resources are requested by a Grid application, which use computing, data and network resources etc. However, scheduling an application of a Grid system is absolutely more complex than scheduling an application of a single computer. Because to get the resources information of single computer and scheduling is easy, such as CPU frequency, number of CPU’s in a machine, memory size, memory configuration and network bandwidth and other resources connected in the system. But Grid environment is a dynamic resources sharing and distributing environment. So it is is hard to get resources information for an application, such as CPU load, available memory, available network capacity etc. And in Grid environment, it is also hard to classify jobs characteristic, that run in Grid.
functions of grid computing. The function of grid is based on the information about the resources available currently. To ensure that the grid is working perfectly, it is necessary to know the status of resources allocated to nodes and resources that are free yet to be allocated. This work aims at building a distributed system for grid resource monitoring and prediction models based on the user negotiation strategies. This paper presents a platform where the user can negotiate on the required time and cost to meet the deadline based on the SLA and within the user prescribed limits of time, cost and QoS, the work will be completed. This is done through the use of hybridization algorithm, PH-PSO.
.
2. Related Work
The author in [12], proposed a multi-agent based peer-to-peer grid computing architecture. This architecture provides a reasonable compatibility and interoperability with the conventional grid systems and clients. The main characteristics of this architecture are highlighted by its promising performance and scalability, and its adaptive resource management and scheduling mechanisms. With this architecture, it is promising to build large scale high performance commodity computing grids at low cost. In this paper, the author firstly analyzed the service-oriented architecture and the server-based computing model of the conventional grid, and designed a new computing architecture. By applying a new job/service model and a peer-to-peer computing model, the new architecture is more efficient and flexible when dealing with open systems. Secondly, the author introduced an adaptive resource management and scheduling framework to achieve load balance and high ratio of resource utilization. Hiroshi Sunaga [11] described advanced grid architecture efficient for homology searching in bioinformatics in the year 2007. An integrated architecture of the computing-grid and data grid through peer-to-peer (P2P) communication attains efficient job processing. Grid computing has been studied, standardized, and developed in various areas in both scientific and business areas, such as high energy physics, astronomy, etc. In this paper, an advanced grid architecture that can reduce total processing time was discussed, in particular the P2P-based integration of the computing-grid and data-grid. Field trial results (homology searching) showed effectiveness to the original objective.
Lei Zhang et al [15] proposed a heuristic approach based on particle swarm optimization algorithm for solving task scheduling problem in grid environment. In this article, scheduling algorithm based on PSO is proposed for task scheduling problem on computational grids. The approach is to generate an optimal schedule so as to complete the tasks in a minimum time as well as utilizing the resources
in an efficient way. Simulation results demonstrate that PSO algorithm can get better effect for a large scale optimization problem.
The author in [3] discussed that before the enterprise resource planning (ERP) concepts, business functions within enterprises were supported by small and isolated applications, most of them developed internally. Yet today ERP platforms are not by themselves the answer to all organizations needs especially in times of differentiated and diversified demands among end customers. ERP platforms were integrated with specialized systems for the management of clients, customer relationship management and vendors, supplier relationship management. They were integrated with manufacturing execution systems for better planning and control of production lines. In order to offer real time, efficient answers to the management level, ERP systems were integrated with business intelligence systems. This article analyses the advantages of grid computing at this level of integration, communication and interoperability between complex specialized informatics systems with a focus on the system architecture and data base systems.
better results. In this work, a new heuristic algorithm is proposed for scheduling workflows in utility Grids, and its performance is evaluated on some well-known scientific workflows in the Grid context. Experimental results show that PH-PSO algorithm gives accurate and efficient outcome with greater performance on workflow scheduling, meeting the required service level for users.
3. Proposed Methodology
The proposed system tries to avoid such issues and provide better workflow scheduling in grid. An SLA is a contract between the provider and the consumer of services describing the qualities and the guarantees of the service provisioning of resource with minimum execution time. Proposed system is reliable in security or resources and available to proper workloads. Grid resource state prediction has many fast constraints to track of resources in a grid system by means of modeling and analyzing historical monitoring data. Proposed system handles the Diagnosing fault, and maintaining dynamic load balancing, thus, to help grid users obtain desired computing results by efficiently utilizing system resources in terms of minimized cost, maximized performance, or trade-offs between cost and performance
3.1 Construction of Grid Computing System Architecture
Grid architecture is a complex distributed system composed of resources and Tasks. Resource information extraction process has to be built to monitor and predict grid resource state information. Design principle for the system architecture is Responsiveness and Robustness. Since grid resource states vary dynamically, the information monitored or predicted has to be timely updated through extraction functionality, to guarantee online reflection of resource conditions. In this system, resource sensors and prediction models are periodically placed and executed to generate up-to-date information for users. Resource sensors are deployed on computing nodes since it’s inevitable, they also run and sleep dynamically to reduce overhead, while other components are deployed out of computing nodes to avoid extra overhead. Grid users do not need traversal of all the nodes or grid expertise to get information. A uniform and friendly interface component is designed for accessing the information monitored or predicted.
3.2 Prediction of Task and Resources Using Machine learning based strategies
Fix a machine learning algorithm for the prediction model, and set its default hyper parameters. Separate the sample
set into three parts: training set, validation set, and test set. Feed the learning algorithm with a sample of training set, and repeat it one by one until all samples are used. For some algorithms, the training procedure runs only once; for others, iterations are needed. Feed the trained model with all samples of validation set, and record the errors between true data and predicted ones. Grid user logs on information node and customizes three terms before sending a monitoring/prediction request: which node, which resource type, and how long the prediction will last. A prediction request is then created accordingly and sent to information service. Information service launches the monitoring work flow by sending a monitoring request to the monitoring service.
3.3 Construction of Hybrid Resources Scheduling Algorithm
The algorithm is initialized with a population of random particles and searches a multidimensional solution space for optima by updating particle generations. Each particle moves based on the direction of local best solution discovered by itself and global best solution discovered by the swarm. Initialize system: set parameters for the PSO system, including population, iteration number, and dimensional search intervals; set parameters for particles such as inertia weight, personal learning factor, and social learning factor; randomly generate position and velocity for each particle which is coded using hybrid vector PR. The vector consists of real numbers and binary numbers, which are used to set parameters.
3.4 Estimation of Partial Critical Paths of Workflows
The Partial critical path method has two main phases to estimate the workloads: Deadline Distribution and Planning. In the first phase, the overall deadline of the workflow is distributed over individual tasks, such that if each task finishes before its sub deadline then the whole workflow finishes before the user defined deadline. In the second phase, the scheduler has to select the cheapest service for each task while meeting its sub deadline.
3.5 Construction of Resources Management Systems based on Workloads
3.6 Swarm Intelligence And PH-PS0
Swarm Intelligence (SI) is based on the collective behavior of decentralized, self organized systems. The expression was introduced by Gerardo Beni and Jing Wang in 1989, in the context of cellular robotic systems. SI systems are typically made up of a population of simple agents interacting locally with one another and with their environment. Natural samples of SI include ant colonies, bird flocking, animal flocking etc. PH-PSO is an Artificial Intelligence (AI) technique that can be used to find approximate solutions to extremely difficult or impossible numeric maximization and minimization problems. Hypothesis are plotted in this space and seeded with an initial velocity, as well as a communication channel between the particles.
3.6.1PH-PSO Algorithm
A basic variant of the PH-PSO (Parallel Hybrid Particle Swarm Optimization) algorithm works by having a population (called a swarm) of candidate solutions (called particles). These particles are moved around in the search-space according to a few simple formulae. The movements of the particles are guided by their own best known position in the search-space as well as the entire swarm's best known position. When improved positions are being discovered, these will then come to guide the movements of the swarm. The process is repeated and by doing so it is hoped, but not guaranteed, that a satisfactory solution will eventually be discovered.
3.6.2 Working of the Algorithm
PH-PSO is initialized with a group of random particles (solutions) and searches for optimal by updating generations. Particles move through the solution space and are evaluated according to some fitness criterion after each time step. In every iteration, each particle is updated by following two “best” values.
The first one is the best solution (fitness) it has achieved so far (the fitness value is also stored). This value is called personal best (pbest). Another best value that is tracked by particle swarm optimizer is the best value obtained so far by any particle in the population. This second best value is called global best (gbest).
When a particle takes part of the population as its topological neighbors, the second best value is a local best and is called lbest. Neighborhood tests allow parallel exploration of the search space and reduce the susceptibility of PH-PSO to falling into local minima, but slow down convergence speed. Each particle tries to modify its current position and velocity according to the
distance between its current position and pbest, and the distance between its current position and gbest.
Pseudo code for PH-PSO:
For each particle
Initialize particle with feasible random no End
Do
For each particle
Calculate the fitness value
If the fitness value is better than the best fitness value (pbest) in history
Set the current value as new pbest End
Choose the particle with the best fitness value of all particles as the gbest
For each particle
Calculate particle velocity according to velocity update equation
Update particle position according to position update equation.
End
While maximum iterations (or) minimum error criteria is not attained.
In PH-PSO, the randomly generated particles are initialized with random positions and velocities. Each of the particles flies through multidimensional solution-search space and, at each iteration, its velocity is dynamically adjusted according to its previous best position and best positions of others. Unlike Genetic Algorithm, each particle remembers its best position so far. In other words, each particle’s best position is stored in its memory, which is regarded as experience. The experience is replaced when new experience is discovered in its search process. PSO was initially designed to simulate flock of birds seeking for food. A scenario of group of birds randomly searching for food in an area demonstrates the PH-PSO process. In this analogy, there is only one piece of food in the area being searched. The birds do not know where thefood is. But they know how far the food is and their peer’s positions. An effective strategy to find the food is to follow the bird which is nearest to the food. The goal of each particle is to attain the global best position before convergence. The PH-PSO optimization process is driven by the velocity vector, which is modeled as follows:
The modification of the particle’s position can be
mathematically modeled according the followingequation:
Vik+1 = wVik +c1 rand1(…) x (pbesti-sik) + c2 rand2(…) x
(gbest-sik) ….. (1)
Where, vi : velocity of agent i at iteration k,
w : weighting function, cj : weighting factor,
rand : random number between 0 and 1,
sik : current position of agent i(iteration k) pbesti : pbest of agent i, gbest : gbest of the group
The following weighting function (w) is usually utilized in (1)
w = wMax-[(wMax-wMin) x iter]/maxIter (2)
Where wMax = initial weight,
wMin = final weight,
maxIter = maximum iteration number, iter = current iteration number. sik+1 = sik+ Vik+1
(3)
Velocity (v) according to iteration : (4)
c1 : acceleration factor related to gbest
c2 : acceleration factor related to lbest
gbest : gbest position of swarm
pbest : pbest position of particle
In the PSO algorithm each individual is called a "particle", and is subject to a movement in a multidimensional space that represents the belief space. Particles have memory, thus retaining part of their previous state. There is no restriction for particles to share the same point in belief space, but in any case their individuality is preserved. Each particle's movement is the composition of an initial random velocity and two randomly weighted influences: individuality, the tendency to return to the particle's best previous position, and sociality, the tendency to move towards the neighborhood's best previous position. These formulas are used to implement the PH-PSO algorithm.
4. Results and Discussions
4.1 Simulation Environment
The resource management and scheduling systems for Grid computing need to manage resources and application execution depending on either resource consumers’ or owners’ requirements, and continuously adapt to changes in resource availability. A Java-based discrete-event Grid simulation toolkit called GridSim is used to implement this proposed algorithm. The toolkit supports modeling and simulation of heterogeneous Grid resources (both time- and space-shared), users and application models.
4.2 Experimental Results
The management of resources and scheduling of applications in large-scale distributed systems is a complex undertaking. In order to prove the effectiveness of resource brokers and associated scheduling algorithms, their performance needs to be evaluated under different scenarios such as varying number of resources and users with different requirements. In a Grid environment, it is hard and even impossible to perform scheduler performance evaluation in a repeatable and controllable manner as resources and users are distributed across multiple organizations with their own policies. To overcome this limitation, a Java-based discrete-event Grid simulation toolkit called GridSim was developed. The toolkit supports modeling and simulation of heterogeneous Grid resources (both time- and space-shared), users and application models.
The GridSim toolkit is used which allows modeling and simulation of entities in Parallel and Distributed Computing (PDC) systems-users, applications, resources, and resource brokers (schedulers) for design and evaluation of scheduling algorithms. It provides a comprehensive facility for creating different classes of heterogeneous resources that can be aggregated using resource brokers. for solving compute and data intensive applications. A resource can be a single processor or multi-processor with shared or distributed memory and managed by time or space shared schedulers. The processing nodes within a resource can be heterogeneous in terms of processing capability, configuration, and availability. The resource brokers use scheduling algorithms or policies for mapping jobs to resources to optimize system or user objectives. The Proposed system produces desired QoS (such as deadline) by paying an appropriate price.
A Gridlet is a package that contains all the information related to the job and its execution management details such as job length expressed in MI (Millions Instruction),
2
1( ) *( CurrentPosition )n 2( )*( CurrentPositionn
1 1 best,n rand best,n )
vn =vn+c rand p − +c g −
the size of input and output files, and the job owner id. Individual users model their application by creating Gridlets for processing them on Grid resources. These basic parameters help in determining execution time, the time required to transport input and output files between users and remote resources, and returning the processed Gridlets back to the originator along with the results. For a Gridlet that requires many Processing Elements (PEs) or CPUs, the gridlet length is calculated only for 1 PE for simplicity. For example, this Gridlet has a length of 500 MI and requires 2 PEs. This means each PE will execute 500 MI of this Gridlet. In GridSim 3.1, there is a classType attribute which can be used to provide differentiated service to scheduling gridlets on a resource. The higher the classType means the higher the priority of scheduling this Gridlet on the resource. However, it is up to the resource's allocation policy to schedule gridlets based on their priority or not.By default, this object records a history of every activity, such as when this Gridlet is being put into a queue, and when it is being executed by a resource. You can find the history by using getGridletHistory(). However, this approach requires a significant amount of memory if you have many Gridlet objects. To disable this functionality, use Gridlet(int,double,long,long,Boolean)constructor instead.
Fig. 1 various jobs and their execution time using PSO.
Fig.2 various jobs and their execution time using PH-PSO.
Various files are chosen by the user and those files are assigned to concerned grids by the system, which is called
scheduling. This assigning, if done in proper manner , reduces the execution time and cost and it is done using the PH-PSO algorithm. The results show that the estimated time and cost has decreased with the usage of PH-PSO(Fig. 2) when compared to usage of PSO algorithm(Fig. 1).
5. Conclusion and Future Work
This work is implemented in a distributed resource monitoring and prediction architecture that seamlessly combines grid technologies, resource monitoring, and machine learning-based resource state prediction. This system consists of a set of distributed services to accomplish all required resource monitoring, data gathering, and resource state prediction functions. A Parallel Hybrid Particle Swarm Optimization algorithm was proposed which combines discrete PSO and continuous PSO, for the purpose of combinational optimization of prediction model. Evaluation results indicate that the combinational model using PH-PSO meets the accuracy and efficiency demand of an online system. For the expectation of achieving higher performance, hybrid algorithm PH-PSO was used. Comparative simulations indicate that the PH-PSO achieves lower error and costs less optimizing time thereby fulfilling the user requirements. The results of this work will contribute to building and advancing of computing grid infrastructure. Further research will be done in the following aspects: monitoring and prediction of grid tasks, classification and evaluation of grid resources, tasks, etc. It is believed that machine learning strategies are applicable tools for modeling and optimizing, and they will play a more important role by the virtue of their potential in distributed computing environment.
References
[1] S.Baghavathi Priya, “An Optimal Scheduling Algorithm for Real Time Applications in Grid System”, IJCSI International Journal of Computer Science Issues, Vol. 10, Issue 1, January 2013. [2] Sunita Bansal1, et al, “Dynamic Task-Scheduling in
Grid Computing using Prioritized Round Robin Algorithm”, IJCSI International Journal of Computer Science Issues, Vol.8, Issue 2, 2011.
[3] LaurentiuCiovica, et al, “A Grid Architecture for Manufacturing Database System”, Database Systems Journal vol. 2, Issue 2, February 2011.
[4] Munehiro Fukuda, et al, “Resource Management and Monitoring in Agent Team work Grid Computing Middleware”, IEEE Explore, Vol. 10, Issue 1, August 2007.
[6] Adan Hirales, et al, “Workload Generation for Trace Based Grid Simulations”,Cimarron.ens.uabc.mx, Vol.1, Issue 3, 2010.
[7] Vishnu Kant Soni, et al, “Constraint-Based Job and Resource scheduling in Grid Computing”, IEEE Explore, Vol.4, Issue 5, 2010.
[8] Dr. N. Malarvizhi, “Resource Scheduling in Hybrid Grid Environment”, IJCSE International Journal of Computer Science and Engineering, Vol.4, Issue 8, August 2012.
[9] D. Spooner, “Performance-aware Workflow Management for Grid Computing”, Oxford Journals, Vol.48, Issue 3, June 2004.
[10] Weifeng Sun, et al, ” A Priority-based Task Scheduling Algorithm in Grid”, Browse Conference Publications, Vol.7, Issue 1, December 2010.
[11] Hiroshi Sunaga, et al, “P2P-based Grid Architecture for Homology Searching”, IEEE Explore, Vol.4, Issue 1,May 2005.
[12] Jia Tang, et al, “An agent-based peer-to-peer grid computing architecture”, ACSW Frontiers, Vol.54, Issue 1 , 2006.
[13] S.ThamaraiSelvi, ”Scheduling in virtualized grid environment using hybrid approach”, IJGCA International Journal of Grid Computing & Applications , Vol.1, No.1, September 2010
[14] ChenkhongTham, “Sensor grid computing and sensor grid architecture for event detection, classification and decision-making”, CSI Communications, Vol.2, Issue 1, July 2005.
[15] Lei Zhang, et al, Center, “A Task scheduling algorithm based on PSO for grid computing”, International Journal of Computational Intelligence Research, Vol.4, Issue 1, 2008.
[16] HP Grid Computing
http://www.hp.com/techservers/grid/index.html [17] IBMGridComputingBenefits
http://www-1.ibm.com/grid/about_grid/benefits.shtml
[18] Oracle Grid Index Report
