• No results found

An Optimized BP Algorithm Based on PLS


Academic year: 2020

Share "An Optimized BP Algorithm Based on PLS"


Loading.... (view fulltext now)

Full text


2016 International Conference on Computer, Mechatronics and Electronic Engineering (CMEE 2016) ISBN: 978-1-60595-406-6

An Optimized BP Algorithm Based on PLS



* and Xiao-feng JI



Nantong Polytechnic College, China


Nantong Shipping College, China

*Corresponding author

Keywords: Partial least squares, BP neural network algorithm, PLS-BP algorithm.

Abstract. Back-propagation (BP) neural network has been widely used for dealing with large-scale

nonlinear problems. However, the BP algorithm is powerless, when in the face of singular sample data, which with high characteristic dimensional and small sample size. Too many input makes the network structure is difficult to determine, cause slow convergence rate; fewer number of samples makes the network training is not complete, thereby affecting the recognition precision of the neural network. Aiming at these deficiencies, in this paper proposed the optimized BP neural network algorithm based on the partial least squares (PLS) algorithm (PLS-BP algorithm), firstly the new algorithm reduce feature dimension for singular sample data used PLS method, which can fully take into account the level of correlation characteristic variables and the dependent variables, then get the low-dimensional data is used for network training and simulation. New algorithm simplifies the network structure and improves the network training speed and recognition precision.


Hundreds of network algorithm models have been proposed, in which Back-Propagation (BP) neural network [2] is one of most mature and most widespread algorithms. A three-layer BP neural network can approximate any nonlinear function, has been widely applied in various fields[3-6].

The BP neural network with own superiority for dealing with problems, also plays a huge role in the practical application, but it also has some of its own limitations, such as slowly convergence, easy to fall into local minimum, difficult to determine the number of hidden layer neurons, and so on. These deficiencies are more apparent, when in the face of singular sample data, which with high characteristic dimensional and small sample size. Such as too many characteristic variables will be detrimental to network design, while taking up a lot of storage space and computing time, influence the network convergence; fewer number of samples may result in network training is not sufficient, these deficiencies will ultimately affect recognition precision of the network. Therefore, it is necessary to reduce the feature dimensional for original singular sample, to analyze and extract useful characteristics of variables from large quantities of data and eliminate the influence of relevant and repeated factors, so as to reduce as many feature dimensions of the samples as possible without affecting solving the problems[7-9].

Although the above instance has good application effect, but dose not form a complete and systematic algorithm. In this paper, PLS method is used for singular sample feature dimension reduction, get the low dimensional data to train the BP neural network, in order to establish the BP neural network based on PLS optimized algorithm. To better illustrate the superiority of the new algorithm, compared with the traditional BP neural network algorithm, based on principal component analysis (PCA) optimize BP neural network algorithm (PCA-BP algorithm).

PLS-BP Algorithm

BP Neural Network


signal. The forward-propagating direction is input layer→ hidden layer→ output layer, the state of neurons of each layer only affect the neurons of next layer, if it can not obtain the expected output, then turns to the process of reverse spread of the error signal. Figure I show the topology structure of the BP network.

Figure 1. The topology of three-layer BP neural network.

The network training is a continual readjustment process to the weight and the threshold value, to make the network error reduce to a pre-set minimum or stop at a pre-set training step. Then input the forecasting samples to the trained network, and obtain the forecasting result.

PLS Dimension Reduction

Partial Least Squares is the characteristic development of Ordinary Least Squares, its basic idea is that

the characteristic variable matrix X is compressed, at the same time, give consideration to the

correlation of the dependent variable matrix Y.

PLS aim to each dimension do iterative calculation use each other's information, each iteration

continuously according to residual information of X, Y to adjust ti, uj for the second round extracted,

until the residual matrix element of absolute value approximate to zero, the precision satisfied the

requirements, the algorithm stops. In the iteration process, ti, uj can maximize the expression of

variance of X and Y at the same time.

PLS regression does not need to use all the components to establish the regression equation, only

need select the front l components (0ln), and then can get better regression equation. Generally

used K-fold cross-validation method to calculate prediction residual sum of squares to determine the number of components extracted, reaching the purpose of dimension reduction.

PLS-BP Neural Network

It is inherently difficult to determine the number of hidden layer neurons of BP neural network, is more of a challenge when hit the singular samples. Too many features input and few samples training challenge the performance of BP neural network training, also affect recognition precision of the network. When singular samples for dimension reduction, should take into account the correlation between independent variables and dependent variables, in order to get more accurate low-dimensional data. PLS dimension reduction method on singular sample has unique advantage, establish an optimized BP neural network algorithm based on PLS (PLS-BP algorithm), in order to get a more simplified network structure, and improve the training speed and recognition precision. The PLS-BP algorithm flow shows Figure 2.


Pre-set the basic parameters of optimization algorithms, PLS feature dimension reduction using K-fold Cross Validation (K-CV) method, calculate prediction residual sum of squares. This method can effectively avoid over-learning or owe-learning, get the result with more persuasiveness.

In the formula, s, n, m represent the number of neurons of hidden layer, input layer and output

layer, respectively. Network training using trainlm (LM) algorithm, which is the combination of the gradient descent method and quasi-newton method, give full play to advantage of gradient descent method with high convergence speed at the beginning and quasi-Newton method quickly produce a ideal search direction at close to the extreme; connection weights and threshold learning using the learngdm algorithm.

The basic step of PLS-BP NN algorithm is descried as follow:

Step 1 Normalize the original data, get the characteristic variables matrix X and dependent variable matrix Y;

Step 2 Respectively, extracts the first pair component t1, u1 from X, Y and make up to the maximum correlation;

Step 3 Establish the regression equation of X on t1, and Y on u1, respectively.

Step 4 Determine the absolute value of the residual matrix elements whether close to zero, if yes turn to step 6, otherwise turn to Step 5;

Step 5 Using residual error matrix E and F instead of X and Y, turned to Step 2;

Step 6 With K-CV method, by calculate residual sum of squares and determine the number of components extracted;

Step 7 From the perspective of feature compression, get the compression matrix X and Y, as new samples;

Step 8 Divide the new samples into two as training samples and simulation samples according to the need of the problem;

Step 9 Design the number of neurons of hidden layer, according to the number of factors extracted and outputs of practical problems;

Step 10 By the training sample using LM and learngdm algorithm for network training, to determine the value of all the connection weights and threshold of BP neural network;

Step 11 Predict the simulation samples with trained BP neural network model.

Case Analysis

We have known data [10], to forecast the occurrence degree of the wheat blossom midge. Here we choose the data of 15 samples from 1986 to 2000 as the study object, use x1-x14 to represent the 14 feature variables (weather factors) of the original data; Y represents the occurrence degree of the wheat blossom midge during that year. Use the standardized methods to process the original data (the processed data is still called X). The forecast of the pest occurrence system, in essence can be seen as an input-output system, the transformation relations include the data fitting, the fuzzy transformation and the logic reasoning, these can all be represented by the artificial neural network.


In order to better illustrate the problem, respectively, using BP neural network, PCA - BP neural network algorithm, PLS - BP neural network algorithm to test the case, contrast test results. In this case, choosing 10 samples from 1986 to 1995 as training sample, with 5 samples from 1996 to 2000 as the simulation and prediction sample

Table 1. The comparison of the simulation results of BP algorithm, PCA-BP algorithm and PLS-BP algorithm.

Model Steps 1996 1997 1998 1999 2000 SSE

Actual value — 1 2 2 2 1 —

BP algorithm 1036 1.8109 2.3207 1.3358 2.2819 0.6651 1.3932

PCA-BP 925 0.5031 1.8796 2.2593 1.6927 1.1268 0.4392


Table 1 shows that, from the convergence rate perspective, two improved algorithm is superior to the traditional BP algorithm. From the prediction results, the improved algorithm although loss some of information, but the recognition precision does not reduce; the error sum of squares of PLS-BP algorithm lower than the other two algorithms, recognition precision of PLS-BP algorithm better than PCA-BP algorithm, and also illustrates the effect of dimension reduction for singular sample, PLS better than PCA.


Although case analysis results show that PLS-BP algorithm processing singular sample data, its ability to deal with the problem is far superior to PCA-BP combination algorithm and the traditional BP algorithm. The PLS-BP optimization algorithm performance has a large degree of increase than the traditional neural network, enhanced self-learning ability, accelerate the convergence rate, save the running time, and improve the efficiency of the neural network; recognition precision has certain improved.

Using neural network processing singular sample, which with many characteristic variables and few samples, in the characteristic variables set may be exist a high degree of correlation between the variables, may cause data redundancy, take up storage space, and computing time; while the relatively few samples may lead to network training is not complete. Aim at these deficiencies of singular sample, in this paper, based on PLS algorithm, combined with BP neural network algorithm, and proposed and optimized BP neural network algorithm based on PLS algorithm (PLS-BP algorithm).

Through PLS for singular sample data dimension reduction, can make full use of the data information in the system, the characteristic variables are decomposing and filtering, extracted comprehensive variables have strong explanatory. In other word, dimensions reduce characteristic variables dimensional, meanwhile taking into account the degree of correlation between the dependent variables. PLS extracts the features with more practical significance than PCA extracts the principal components. The BP neural network has many advantages, such as better self-learning, adaptive, robustness and generalization ability, a three-layer BP neural network can approximate any nonlinear function with arbitrary precision, and so on. The new algorithm is integrated two algorithm advantages, better fitting nonlinear singular sample problem.


This work is supported by Education Study Projects of Nantong Polytechnic College (2015010).


[1] S. McCulloch, W. Pitts. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 10(1943), 115-133.

[2] D.E. Rumelhart, G. E. Hinton, R. J. Williams. Learning representation by Back-Propagating errors. Nature, 3(1986), 533-536.

[3] K. Benyamin, R. Shahin, O. Mahmoud. Prognostication of environmental indices in potato production using artificial neural networks. Journal of Cleaner Production, 52 (2013), 402-409.

[4] L.Q. Xu, S.Y. Liu. Study of short-term water quality prediction model based on wavelet neural network. Mathematical and Computer Modelling, 58(2013), 801-807.

[5] A. Hamed, M. Mohammad-Reza, S. Saeid. Classification of GPS satellites using improved back propagation training algorithms. Wireless Personal Communications, 71(2013), 789-803.


[7] S.F. Ding, W.K. Jia, C.Y. Su, et al. A Survey on Statistical Pattern Feature Extraction. Lecture Notes in Artificial Intelligence, 5227 (2008), 701-708.

[8] S.F. Ding, Z. Zhu, W.K. Jia, et al. A survey on feature extraction for pattern recognition. Artificial Intelligence Review, 37 (2012), 169-180.

[9] F.X Song, X.M. Gao, J.Y. Yang, et al. Dimensionality reduction in statistical pattern recognition and low loss dimensionality reduction. Chinese Journal of Computers, 28(2005), 1915-1922.


Figure 1. The topology of three-layer BP neural network.
Table 1. The comparison of the simulation results of BP algorithm, PCA-BP algorithm and PLS-BP algorithm


Related documents

Figure 2 Left picture: comparison of the hourly pedestrian flows on the Redentore bridge estimated from the GPS dataset (continuous curves) and the empirical measures by a direct

Based on Fibonacci lattice-quantization, we develop a novel color global scale-invariant feature transform (CGSIFT) for better description of color contents in video frames for

Finally, besides the control of ticks with fire in the communal fire paradigm, it is be - lieved that the frequency of burning is gener - ally too high and that

T h e children's responses to questioning about picture-free stories as part of whole grade participation, and the responses of a small group of ten children to retelling of the

fire burning logs fire burning by the blowing wind field of wheat stalks swinging fragrant in the breeze yellow light drops reflecting myriad liquidly alive..

nanograin sublayers of ZnO that are created at the initial stage of electrodeposition for the morphology, structure, optical and electrical properties of nanostructured arrays

Therefore, farmers’ markets and agricultural fairs were ideal locations to conduct the Buy Local Survey and meet the study objectives to (a) better understand beliefs and

(E) Strong MIF mRNA (purple) expression and macrophage accumulation (brown-purple) in a damaged tubules of UA groups, where some evidence for deposition of uric acid crystals