A Survey On BPA: Bigdata Parallelism Algorithm For Distributed Machine Learning
Prajakta Ugile , Dr.S.B.Chaudhari
Savitribai Phule University of Pune,
Department of Computer Engineering JSPM, Hadapsar, Pune
---
************************
---Abstract:
A variety of machine learning (ML) algorithms are used to explore and exploit big data originating from diverse sources such as activities on the internet, profiling and transactions in finance, advertisement and healthcare, or experiments in the physical sciences, to name a few. It is important to outline a sensible methodology that gives dynamic correspondence streamlining to enhance the usage of the bunch, quicken the preparation times and reinforce the precision of the preparation display. Due to the increase in data volume and the complexity of the ML model, especially the increase in the number of parameters, single machines have been unable to fully and quickly train the ML model. Using distributed systems that partition the big data across machines and allow each machine to read and update all ML model parameters.
Keywords— Machine Learning, Big data Parallelism Algorithm.
---
************************
---I. INTRODUCTION
Machine learning (ML) calculations are ordinarily connected to enormous information. In recent years, distributed systems have mainly been used to train machine learning (ML) models. In the age of big data, distributed machine learning has become a hot research field due to its ability to adapt to the complexity of big data, obtain higher prediction accuracy and support more intelligent tasks [1]. Distributed optimization of ML is becoming a prerequisite for solving large-scale ML problems [2]–[4].
Due to the increase in data volume and the complexity of the ML model, especially the increase in the number of parameters, single machines have been unable to fully and quickly
train the ML model. However, it is not easy to realize an efficient distributed algorithm [5], [6]. The system design needs to consider the large amounts of computation and data traffic.
The actual size of the training data will be between 1TB and 1PB, which can be used to create a powerful and complex model. To solve this problem, this work proposed big data Parallelism Algorithm (BPA) for Distributed Machine Learning. This model is usually shared globally by all computational nodes. A computational node computes local updates on its data subset in each iteration and then submits the local update to the parameter server, which updates the global model parameters; the parameter server distributes the new global model parameters to each computational node.
•
This system proposed Big-data Parallelism Algorithm (BPA) for Distributed Machine Learning.•
User wants to apply machine learning algorithm to big data.•
Due to the size, it takes more time for training model.•
To deal with this problem, the proposed algorithm split big data and distributes it into many computational nodes.•
Followed by, this algorithm applies ML in each computational node and provides training results to parameter server.Furthermore, parameter server applies one more ML for collected ML Results from computational nodes.
II. RELATEDWORK
Big data analytics [1] is the process of examining large and varied data sets. Gaining from greatly expansive and unstructured information brings critical open doors for various divisions. Still, most of these routines are not much computationally efficient, practical or scalable enough. The authors discuss the need for the research that aims at proposing new techniques that can be used for analysis of big data. However, most of the traditional AI involved methods are not scalable to manage data with the properties of its huge volume, diverse types, inconsistency, and uncertainty along with incompleteness. Accordingly, there is a requirement for machine figuring out how to revive itself for huge information preparing. They started with various types of learning methods. Advance it talks about a portion of the critical and down to earth issues of machine learning for huge information examination.
The authors analyze [2] the convergence of gradient-based optimization algorithms that base their updates on delayed stochastic gradient information. The main application of their results is to the development of gradient-based distributed
optimization algorithms where a master node performs parameter updates while worker nodes compute stochastic slopes dependent on neighborhood data in parallel, which may offer ascent to delays because of asynchrony. They take motivation from statistical problems where the size of the data is so large that it cannot fit on one computer; with the coming of enormous datasets in science, stargazing, and the web, such issues are presently normal. Their main contribution is to show that for smooth stochastic problems, the delays are asymptotically negligible and they can achieve order-optimal convergence results. In application to distributed optimization, they develop procedures that overcome communication bottlenecks and synchronization requirements.
While high-level data parallel frameworks [3], simplify the design and implementation of large-scale data processing systems, they do not naturally or efficiently support many important data mining and machine learning algorithms and can lead to inefficient learning systems. To help fill this basic void, the creators presented the GraphLab reflection which normally communicates nonconcurrent, dynamic, chart parallel calculation while guaranteeing information consistency and accomplishing a high level of parallel execution in the mutual memory setting. In this paper, they stretch out the GraphLab system to the generously all the more difficult conveyed setting while at the same time protecting solid information consistency ensures. They develop graph based extensions to pipelined locking and data versioning to reduce network congestion and mitigate the effect of network latency. They also introduce fault tolerance to the GraphLab abstraction using the classic Chandy-Lamport snapshot algorithm and demonstrate how it can be easily implemented by exploiting the GraphLab abstraction itself.
on partitioning a graph so as to minimize the number of neighbouring vertices rather than edges across partitions. Our decomposition is based on a streaming algorithm. It is organize mindful as it adjusts to the system topology of the fundamental computational equipment. They used local copies of the variables and an efficient asynchronous communication protocol to synchronize the replicated values in order to perform most of the computation without having to incur the cost of network communication.
The authors present [5] a scalable parallel framework for efficient inference in latent variable models over streaming web-scale data. Our structure tends to three key difficulties: 1) synchronizing the worldwide state which incorporates worldwide inert factors (e.g., group focuses and word references); 2) productively putting away and recovering the vast neighborhood state which incorporates the information focuses and their relating inactive factors (e.g., bunch enrolment); and 3) successively fusing gushing information (e.g., the news). They address these challenges by introducing: 1) a novel delta-based aggregation system with a bandwidth-efficient communication protocol; 2) schedule-aware out-of-core storage; and 3) approximate forward sampling to rapidly incorporate new data.
The authors [6] developed STRADS to improve the convergence speed of model parallel ML at scale, achieving both high progress per iteration (via dependency checking and prioritization through SchMP programming), and high iteration throughput (via STRADS system optimizations such as pipelining and the ring topology). Consequently, SchMP programs running on STRADS achieve a marked performance improvement over recent, well-established baselines: to give two examples, SchMP-LDA converges 10x faster than YahooLDA, while SchMP-Lasso converges 5x faster than randomly scheduled Shotgun-Lasso.
As Machine Learning (ML) applications [7] embrace greater data size and model complexity, practitioners turn to distributed clusters to satisfy
the increased computational and memory demands. Effective use of clusters for ML programs requires considerable expertise in writing distributed code, but existing highly abstracted frameworks that pose low barriers to distributed-programming have not, in practice, matched the performance seen in highly specialized and advanced ML implementations. The ongoing Parameter Server (PS) worldview is a centre ground between these limits, permitting simple change of single-machine parallel ML programs into appropriated ones, while keeping up high throughput through loose "consistency models" that permit offbeat (and, consequently, conflicting) parameter peruses. Be that as it may, because of lacking hypothetical investigation, it isn't clear which of these consistency models can truly guarantee amend ML calculation yield; in the meantime, there stay numerous hypothetically roused yet unfamiliar chances to augment computational throughput. Inspired by this challenge, the authors study both the theoretical guarantees and empirical behaviour of iterative-convergent ML algorithms in existing PS consistency models. They used the gleaned insights to improve a consistency model using an “eager” PS communication mechanism, and implement it as a new PS system that enables ML programs to reach their solution more quickly.
The authors proposed [9] a parameter server framework for distributed machine learning problemsThe two information and remaining tasks at hand are circulated over labourer hubs, while the server hubs keep up all around shared parameters, spoke to as thick or meager vectors and grids. The system oversees nonconcurrent information correspondence among hubs, and backings adaptable consistency models, flexible versatility, and persistent adaptation to internal failure.
The authors described [10] a third-generation parameter server framework for distributed machine learning. This structure offers two relaxations to adjust framework execution and calculation productivity. They proposed a new algorithm that takes advantage of this framework to solve non-convex non-smooth problems with convergence guarantees. They presented an in-depth analysis of two large scale machine learning problems ranging from -regularized logistic regression on CPUs to reconstruction ICA on GPUs, using 636TB of real data with hundreds of billions of samples and dimensions. They demonstrated using these examples that the parameter server framework is an effective and straightforward way to scale machine learning to larger problems and systems than have been previously achieved.
III. DISTRIBUTEDMACHINELEARNING
SYSTEM
During this process [11], the optimizer tries to find a plan that quickly returns a first quality answer to the user, allowing MLbase to improve the result iteratively in the background. In this paper, we reported first results showing the potential of the optimizer as well as the performance advantages of Algorithm-specific execution strategies. We describe the MLbase run-time and show how it differs from traditional database pipelines. A novel streamlining agent to choose and progressively adjust the decision of learning calculation.
In this paper [12], they presented a programmable framework for dynamic Big Model-parallelism that
provides the following benefits: (1) scalability and efficient memory utilization, allowing larger models to be run with additional machines; (2) the capacity to summon dynamic timetables that decrease demonstrate parameter conditions crosswise over specialists, prompting lower parallelization mistake and in this manner quicker, remedy intermingling. Likewise need to investigate the utilization of STRADS for other prominent ML applications, for example, bolster vector machines and calculated relapse.They demonstrate the efficacy of model-parallel algorithms implemented on STRADS versus popular implementations for topic modeling, matrix factorization, and Lasso.
This paper [13] examined the application of a third generation parameter server framework to modern distributed machine learning algorithms. They show that it is possible to design algorithms well suited to this framework; in this case, an asynchronous block proximal gradient method to solve general non-convex and non-smooth problems, with provable convergence. Parameter server structure is a successful and direct approach to scale machine figuring out how to bigger issues and frameworks. Only coordinates that would change the associated model weights are communicated to reduce network traffic.
IV. BULK SYNCHRONOUS PARALLEL MODEL
In this paper [15], they report the first attempt of combining the theory of constructive algorithmic and proved correct BSML parallel programs for systematic development of certified BSP parallel programs, and demonstrate how it can be useful to develop certified BSP parallel programs. In this paper they focus on the semantics of the programming model of Bulk Synchronous Parallel ML, which was first express as an extension of the λ-calculus. It serves as a bridge in mapping from high level specification to low level BSP parallel programs. Bulk Synchronous Parallelism (BSP) is a model of calculation which offers a high level of reflection like PRAM models.
Through the study [16] of Ocelot using several microbenchmarks and full applications, on-chip memory pressure, context- switch overhead, and variable CTA execution time were identified as fundamental issues that impact performance when compiling highly parallel programs to systems with few hardware resources. A model for the translation of explicitly parallel bulk synchronous applications to a multi-threaded execution model. Once the function has been created, the PTX control flow graph is walked and each basic block is examined.
In this paper [17], they map the high-level modeling concepts of algebraic graph transformations to the bridging model Bulk Synchronous Parallel (BSP) which provides an abstraction layer for implementing parallel algorithms on distributed data. They have implemented a code generator that takes Henshin models as input and generates code for the BSP-based framework. To apply them to large-scale graphs, we introduce a method to distribute and parallelize graph transformations by mapping them to the Bulk Synchronous Parallel model.
V.STALE SYNCHRONOUS PARALLEL
MODEL
They propose [18] a parameter server system for distributed ML, which follows a Stale Synchronous Parallel (SSP) model of computation that maximizes the time computational workers spend doing useful work on ML algorithms. The parameter server gives a simple to-utilize shared interface for read/compose access to a ML model's qualities. Timekeepers are like emphases, and speak to some unit of advancement by a ML calculation. They provide a proof of correctness under SSP, as well as empirical results demonstrating that the SSP model achieves faster algorithm convergence on several different ML problems. A perfect SSP usage would completely abuse the slack conceded by the SSP's limited staleness property, with the end goal to adjust the time specialists go through looking out for peruses with the requirement for freshness in the mutual information.
This paper studied [19] two popular asynchronous parallel implementations for SG on computer cluster and shared memory system respectively. Two calculations (CON and ASYSG-INCON) are utilized to portray two usage. Offbeat parallel executions of stochastic angle (SG) have been extensively utilized in understanding profound neural system. Their results generalize and improve existing analysis for convex minimization. All specialists trade data with the ace autonomously and all the while. While the master is aggregating stochastic gradients from workers, it does not care about the sources of the collected stochastic gradients.
Synchronous Parallel (SSP) consistency demonstrates.
VI. CONCLUSION
On this paper, we addressed the problem of using ASP as a communication strategy to train a large-scale distributed ML model will yield low model accuracy. Training the distributed ML model with BPA can reduce the communication and synchronization costs and improve the utilization rate of the computational nodes and the efficiency of calculation. Furthermore, it balances the training time and the model accuracy, and improves the performance of the distributed ML algorithm. It reduces execution time and increase training speed. The experimental results show that BPA can guarantee better accuracy and achieve higher convergence speed, and it has good expansibility.
ACKNOWLEDGMENT
I profoundly grateful to prof.for his/her expert guidance and continuous encouragement throughout to see that this project rights its target since its commencement to its completion. I would like to express my deepest appreciation towards Principal, prof., HOD department of computer engineering and PGcoordinator. I must express my sincere heartfelt gratitude to all staff members of computer engineering department who helped me directly or indirectly during this course of work. Finally, I would like to thank my family and friends, for their precious support.
REFERENCES
1) E. P. Xing, Q. Ho, P. Xie, and D. Wei, “Strategies and principles of distributed machine learning on big data,” Engineering, vol. 2, no. 2, pp. 179–195, 2016.
2) A. Agarwal and J. C. Duchi, “Distributed delayed
stochastic optimization,” in Advances in Neural
Information Processing Systems, ser. Advances in Neural Information Processing Systems. Granada: NIPS, 2011, pp. 873– 881.
3) Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M. Hellerstein, “Distributed graphlab: a framework for machine learning and data mining in
the cloud,” Proceedings of the VLDB Endowment,
vol. 5, no. 8, pp. 716–727, 2012.
4) A. Ahmed, N. Shervashidze, S. Narayanamurthy, V. Josifovski, and A. J. Smola, “Distributed large-scale
natural graph factorization,” in International
Conference on World Wide Web, ser. International Conference on World Wide Web. Rio de Janeiro: ACM, 2013, pp. 37–48.
5) A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. Smola, “Scalable inference in latent variable
models,” in ACM International Conference on Web
Search and Data Mining, ser. ACM International Conference on Web Search and Data Mining. Seattle: ACM, 2012, pp. 123–132.
6) J. K. Kim, Q. Ho, S. Lee, X. Zheng, W. Dai, G. A. Gibson, and E. P. Xing, “Strads: a distributed framework for scheduled model parallel machine
learning,” in Proceedings of the Eleventh European
Conference on Computer Systems, ser. Proceedings of the Eleventh European Conference on Computer Systems. London ACM, 2016, p5.
7) DAI, W., KUMAR, A., WEI, J., HO, Q., GIBSON, G. A., AND XING, E. P. ”High-performance distributed ML at scale through parameter server consistency
models”, In AAAI (2015).
8) Zaharia M, Chowdhury M, Das T, Dave A, Ma J, Mccauley M, Franklin M J, Shenker S, And Stoica I,
“Resilient distributed datasets: a fault-tolerant
abstraction for in-memory cluster computing “ In
NSDI (2012).
9) M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B. Y. Su, “Scaling distributed machine learning with the
parameter server,” in International Conference on Big
Data Science and Computing, ser. International Conference on Big Data Science and Computing. New York: ACM, 2014, pp. 583–598.
10) M. Li, D. G. Andersen, A. Smola, and K. Yu,
“Communication efficient distributed machine
Conference on Neural Information Processing Systems, ser. International Conference on Neural Information Processing Systems. Montreal: NIPS, 2014, pp. 19–27.
11) Tim Kraska, Ameet Talwalkar John Duchi, “MLbase:
A Distributed Machine-learning System,” publication
in the proceedings of CIDR, January 6-9, 2013.
12) Seunghak Lee, Jin Kyu Kim, Xun Zheng, Qirong Ho, Garth A. Gibson and Eric P. Xing, “On Model
Parallelization and Scheduling Strategiesfor
Distributed Machine Learning,” Advances in Neural
Information Processing Systems NIPS 2014.
13) Mu Li, David G. Andersen, Alexander Smola and Kai Yu, “Communication Efficient Distributed Machine
Learning with the Parameter Server,” Conference on
Neural Information Processing Systems - Volume 1, December 08 - 13, 2014.
14) Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang and Minjie Wang, “MXNet: A Flexible and
Efficient Machine Learning Library for
Heterogeneous Distributed
Systems,”volume1512.01274, Dec 2015.
15) Louis Gesbert, Zhenjiang Hu, Frederic Loulergue, KiminoriMatsuzakiand Julien Tessonm, “Systematic Development of Correct Bulk Synchronous Parallel
Programs,” IEEE P.P. 2379-5352, Dec. 2010.
16) Gregory Diamos, Andrew Kerr
andSudhakarYalamanchili, “Ocelot: A Dynamic
Optimization Framework for Bulk-Synchronous
Applications in Heterogeneous Systems,” IEEE
PACT’10,Sept11–15, 2010.
17) Christian Krause, Matthias Tichy and Holger Giese,
“Implementing Graph Transformations in the Bulk
Synchronous Parallel Model,” IEEE Conference
paper, volume 8411, 2014.
18) Qirong Ho, James Cipar, Henggang Cui, Jin Kyu Kim and Seunghak Lee, “More Effective Distributed ML
via a StaleSynchronous Parallel Parameter Server,”
Neural Information Processing Systems - Volume 1, December 2013.
19) XiangruLian, Yijun Huang, Yuncheng Li, and Ji Liu,
“Asynchronous Parallel Stochastic Gradient
forNonconvex Optimization,” International
Conference on Neural Information Processing Systems - Volume 2, Jun 2015.
