Using the Neural Network
Using the Neural Network Time Series ToTime Series Toolol 1.
1. If If neeneededed, d, opopen en ththe Ne Neueural ral NeNetwtwork ork StStart art GUGUI wI witith th thihis cs comommanmand:d: 2.nnstart
2.nnstart
3
3.. CClliicckk Time Series ToolTime Series Tool to ope to open the n the Neural Network Time Series ToNeural Network Time Series Tool.ol. !o
Notice that this openin# pane is different than the openin# panes for the Notice that this openin# pane is different than the openin# panes for the other GUIs. This is $ecause
other GUIs. This is $ecause ntstoolntstoolcan $e used to sol%e threecan $e used to sol%e three different kinds of time series pro$lems.
different kinds of time series pro$lems. •
• In the first t&pe of time series pro$lem, &ou would like to predictIn the first t&pe of time series pro$lem, &ou would like to predict future %alues of a time series
future %alues of a time series y y t t " from past %alues of that time series" from past %alues of that time series and past %alues of a second time series
and past %alues of a second time series x x t t ". This form of prediction is". This form of prediction is called nonlinear autore#ressi%e with e'o#enous e'ternal" input, or called nonlinear autore#ressi%e with e'o#enous e'ternal" input, or N()* see
N()* see +N()* Network++N()* Network+ nar'net, closeloop"", and can $e written as nar'net, closeloop"", and can $e written as follows:
follows: y
y t t " " f f y y t t - 1", ..., - 1", ..., y y t t - - d d ",", x x tt - 1", ..., - 1", ..., t t - - d d """"
This model could $e used to predict future %alues of a stock or $ond, This model could $e used to predict future %alues of a stock or $ond, $ased on such
$ased on such economic %aria$les as uneeconomic %aria$les as unemplo&ment rates, G/, mplo&ment rates, G/, etc. Itetc. It could also $e used for s&stem identification, in which models are
could also $e used for s&stem identification, in which models are
de%eloped to represent d&namic s&stems, such as chemical processes, de%eloped to represent d&namic s&stems, such as chemical processes, manufacturin# s&stems, ro$otics, aerospace %ehicles, etc.
manufacturin# s&stems, ro$otics, aerospace %ehicles, etc. •
• In the second t&pe of time series pro$lem, there is onl& oneIn the second t&pe of time series pro$lem, there is onl& one series in%ol%ed. The future %alues of a time series
series in%ol%ed. The future %alues of a time series y y t t " are predicted onl&" are predicted onl& from past %alues of that series. This form of prediction is called nonlinear from past %alues of that series. This form of prediction is called nonlinear autore#ressi%e, or N(), and can $e written as follows:
autore#ressi%e, or N(), and can $e written as follows: y
This model could also $e used to predict financial instruments, $ut without the use of a companion series.
• The third time series pro$lem is similar to the first t&pe, in that two series are in%ol%ed, an input series x t " and an output0tar#et
series y t ". ere &ou want to predict %alues of y t " from pre%ious %alues of x t ", $ut without knowled#e of pre%ious %alues of y t ". This input0output model can $e written as follows:
y t " f x t - 1", ..., x t - d ""
The N()* model will pro%ide $etter predictions than this input2output model, $ecause it uses the additional information contained in the pre%ious %alues of y t ". owe%er, there ma& $e some applications in which the pre%ious %alues of y t " would not $e a%aila$le. Those are the onl& cases where &ou would want to use the input2output model instead of the N()* model.
. 4or this e'ample, select the N()* model and click Next to proceed.
5. Click Load Example Data Set in the Select ata window. The Time Series ata Set Chooser window opens.
Note Use the Inputs and Targets options in the Select Data window when you need to load data from the MATLAB® workspace.
6.
7. Select pH Neutralization Process, and click Import. This returns &ou to the Select ata window.
8. Click Next to open the 9alidation and Test ata window, shown in the followin# fi#ure.
The %alidation and test data sets are each set to 15 of the ori#inal data.
;ith these settin#s, the input %ectors and tar#et %ectors will $e randoml& di%ided into three sets as follows:
• 7< will $e used for trainin#.
• 15 will $e used to %alidate that the network is #enerali=in# and to stop trainin# $efore o%erfittin#.
• The last 15 will $e used as a completel& independent test of network #enerali=ation.
See +i%idin# the ata+ for more discussion of the data di%ision process."
The standard N()* network is a two2la&er feedforward network, with a si#moid transfer function in the hidden la&er and a linear transfer
function in the output la&er. This network also uses tapped dela& lines to store pre%ious %alues of the x t " and y t " se?uences. Note that the output of the N()* network, y t ", is fed $ack to the input of the network
throu#h dela&s", since y t " is a function of y t - 1", y t - @", ..., y t - d". owe%er, for efficient trainin# this feed$ack loop can $e opened.
Aecause the true output is a%aila$le durin# the trainin# of the network, &ou can use the open2loop architecture shown a$o%e, in which the true output is used instead of feedin# $ack the estimated output. This has two ad%anta#es. The first is that the input to the feedforward network is more accurate. The second is that the resultin# network has a purel& feedforward architecture, and therefore a more efficient al#orithm can $e used for trainin#. This network is discussed in more detail in +N()*
Network+ nar'net, closeloop".
The default num$er of hidden neurons is set to 1<. The default num$er of dela&s is @. Chan#e this %alue to . !ou mi#ht want to adBust these num$ers if the network trainin# performance is poor.
11. Select a trainin# al#orithm, then click Train. e%en$er#2Dar?uardt trainlm" is recommended for most pro$lems, $ut for some nois& and small pro$lems Aa&esian )e#ulari=ation trainbr" can take lon#er $ut o$tain a $etter solution. 4or lar#e pro$lems, howe%er, Scaled ConBu#ate Gradient trainscg" is recommended as it uses #radient calculations which are more memor& efficient than the Eaco$ian calculations the other two al#orithms use. This e'ample uses the default e%en$er#2 Dar?uardt.
The trainin# continued until the %alidation error failed to decrease for si' iterations %alidation stop".
1@. Under Plots, click Error !utocorrelation. This is used to %alidate the network performance.
The followin# plot displa&s the error autocorrelation function. It descri$es how the prediction errors are related in time. 4or a perfect prediction
model, there should onl& $e one non=ero %alue of the autocorrelation function, and it should occur at =ero la#. This is the mean s?uare error." This would mean that the prediction errors were completel& uncorrelated with each other white noise". If there was si#nificant correlation in the prediction errors, then it should $e possi$le to impro%e the prediction 2 perhaps $& increasin# the num$er of dela&s in the tapped dela& lines. In this case, the correlations, e'cept for the one at =ero la#, fall
appro'imatel& within the >5 confidence limits around =ero, so the model seems to $e ade?uate. If e%en more accurate results were
re?uired, &ou could retrain the network $& clickin# "etrain in ntstool. This will chan#e the initial wei#hts and $iases of the network, and ma& produce an impro%ed network after retrainin#.
13. 9iew the input2error cross2correlation function to o$tain additional %erification of network performance. Under the Plots pane, click Input# Error $ross#correlation.
This input2error cross2correlation function illustrates how the errors are correlated with the input se?uence x t ". 4or a perfect prediction model, all of the correlations should $e =ero. If the input is correlated with the error, then it should $e possi$le to impro%e the prediction, perhaps $& increasin# the num$er of dela&s in the tapped dela& lines. In this case, all of the correlations fall within the confidence $ounds around =ero. 1. Under Plots, click Time Series "esponse. This displa&s the inputs,
tar#ets and errors %ersus time. It also indicates which time points were selected for trainin#, testin# and %alidation.
15. Click Next in the Neural Network Time Series Tool to e%aluate the network.
(t this point, &ou can test the network a#ainst new data.
If &ou are dissatisfied with the networkFs performance on the ori#inal or new data, &ou can do an& of the followin#:
• Train it a#ain.
• Increase the num$er of neurons and0or the num$er of dela&s. • Get a lar#er trainin# data set.
If the performance on the trainin# set is #ood, $ut the test set
performance is si#nificantl& worse, which could indicate o%erfittin#, then reducin# the num$er of neurons can impro%e &our results.
16. If &ou are satisfied with the network performance, click Next.
17. Use this panel to #enerate a D(T(A function or Simulink dia#ram
for simulatin# &our neural network. !ou can use the #enerated code or dia#ram to $etter understand how &our neural network computes outputs from inputs, or deplo& the network with D(T(A CompilerH tools and other D(T(A and Simulink code #eneration tools.
18. Use the $uttons on this screen to #enerate scripts or to sa%e &our results.
• !ou can click Simple Script or !d%anced Script to create D(T(A code that can $e used to reproduce all of the pre%ious steps from the command line. Creatin# D(T(A code can $e helpful if &ou want to learn how to use the command2line functionalit& of the tool$o' to customi=e the trainin# process. In Usin# Command2ine 4unctions, &ou will in%esti#ate the #enerated scripts in more detail.
• !ou can also ha%e the network sa%ed as net in the workspace. !ou can perform additional tests on it or put it to work on new inputs. 1>. (fter creatin# D(T(A code and sa%in# &our results, click &inish. Using $ommand#Line &unctions
The easiest wa& to learn how to use the command2line functionalit& of the tool$o' is to #enerate scripts from the GUIs, and then modif& them to
customi=e the network trainin#. (s an e'ample, look at the simple script that was created at step 15 of the pre%ious section.
% Solve an Autoregression Problem with External % Input with a NARX Neural Networ
% Script generate! b" N#S#$$ %
% #his script assumes the variables on the right o& % these e'ualities are !e&ine!(
%
% phInputs ) input time series.
% ph#argets ) &ee!bac time series. inputSeries * phInputs+
targetSeries * ph#argets+
% ,reate a Nonlinear Autoregressive Networ with External Input input-ela"s * (/+ &ee!bac-ela"s * (/+ hi!!ena"erSi0e * 1+ net * narxnetinput-ela"s3&ee!bac-ela"s3hi!!ena"erSi0e4+ % Prepare the -ata &or #raining an! Simulation
% #he &unction PREPARE#S prepares time series !ata % &or a particular networ3 shi&ting time b" the minimum
% amount to &ill input states an! la"er states. % 5sing PREPARE#S allows "ou to eep "our original
% time series !ata unchange!3 while easil" customi0ing it
% &or networs with !i&&ering numbers o& !ela"s3 with % open loop or close! loop &ee!bac mo!es.
6inputs3inputStates3la"erStates3targets7 * ... preparetsnet3inputSeries3893targetSeries4+
% Set up -ivision o& -ata &or #raining3 :ali!ation3 #esting
net.!ivi!eParam.trainRatio * ;1<11+ net.!ivi!eParam.valRatio * =<11+ net.!ivi!eParam.testRatio * =<11+ % #rain the Networ
6net3tr7 *
trainnet3inputs3targets3inputStates3la"erStates4+ % #est the Networ
outputs * netinputs3inputStates3la"erStates4+ errors * gsubtracttargets3outputs4+
per&ormance * per&ormnet3targets3outputs4 % :iew the Networ
viewnet4 % Plots
% 5ncomment these lines to enable various plots. % &igure3 plotper&ormtr4 % &igure3 plottrainstatetr4 % &igure3 plotregressiontargets3outputs4 % &igure3 plotresponsetargets3outputs4 % &igure3 ploterrcorrerrors4 % &igure3 plotinerrcorrinputs3errors4 % ,lose! oop Networ
% 5se this networ to !o multi)step pre!iction.
% #he &unction ,$SE$$P replaces the &ee!bac input with a !irect
% connection &rom the outout la"er. netc * closeloopnet4+
netc.name * 6net.name > ) ,lose! oop>7+ viewnetc4
6xc3xic3aic3tc7 * preparetsnetc3inputSeries3 893targetSeries4+
"c * netcxc3xic3aic4+
close!oopPer&ormance * per&ormnetc3tc3"c4 % Earl" Pre!iction Networ
% ?or some applications it helps to get the pre!iction a
% timestep earl".
% #he original networ returns pre!icte! "t@4 at the same
% ?or some applications such as !ecision maing3 it woul!
% help to have pre!icte! "t@4 once "t4 is available3 but
% be&ore the actual "t@4 occurs.
% #he networ can be ma!e to return its output a timestep earl"
% b" removing one !ela" so that its minimal tap !ela" is now
% 1 instea! o& . #he new networ returns the same outputs as
% the original networ3 but outputs are shi&te! le&t one timestep.
nets * remove!ela"net4+
nets.name * 6net.name > ) Pre!ict $ne Step Ahea!>7+ viewnets4
6xs3xis3ais3ts7 * preparetsnets3inputSeries3 893targetSeries4+
"s * netsxs3xis3ais4+
earl"Pre!ictPer&ormance * per&ormnets3ts3"s4 !ou can sa%e the script, and then run it from the command line to
reproduce the results of the pre%ious GUI session. !ou can also edit the script to customi=e the trainin# process. In this case, follow each of the steps in the script.
1. The script assumes that the input %ectors and tar#et %ectors are alread& loaded into the workspace. If the data are not loaded, &ou can load them as follows:
2. loa! ph!ataset
B. inputSeries * phInputs+ /. targetSeries * ph#argets+
5. Create a network. The N()* network, narxnet, is a feedforward network with the default tan2si#moid transfer function in the hidden la&er and linear transfer function in the output la&er. This network has two inputs. ne is an e'ternal input, and the other is a feed$ack connection from the network output. (fter the network has $een trained, this
feed$ack connection can $e closed, as &ou will see at a later step." 4or each of these inputs, there is a tapped dela& line to store pre%ious
must select the dela&s associated with each tapped dela& line, and also the num$er of hidden la&er neurons. In the followin# steps, &ou assi#n the input dela&s and the feed$ack dela&s to ran#e from 1 to and the num$er of hidden neurons to $e 1<.
C. input-ela"s * (/+ ;. &ee!bac-ela"s * (/+ D. hi!!ena"erSi0e * 1+ . net *
narxnetinput-ela"s3&ee!bac-ela"s3hi!!ena"erSi0e4+
Note Increasin the num!er of neurons and the num!er of delays re"uires more computation# and this has a tendency to o$erfit the data when the
num!ers are set too hih# !ut it allows the network to sol$e more complicated pro!lems. More layers re"uire more computation# !ut their use miht result in
the network sol$in comple% pro!lems more efficiently. To use more than one hidden layer# enter the hidden layer si&es as elements of an array in
the &itnet command.
1<. /repare the data for trainin#. ;hen trainin# a network containin# tapped dela& lines, it is necessar& to fill the dela&s with initial %alues of the inputs and outputs of the network. There is a tool$o' command that facilitates this process 2 preparets. This function has three input
ar#uments: the network, the input se?uence and the tar#et se?uence. The function returns the initial conditions that are needed to fill the tapped dela& lines in the network, and modified input and tar#et
se?uences, where the initial conditions ha%e $een remo%ed. !ou can call the function as follows:
. 6inputs3inputStates3la"erStates3targets7 * ... 2. preparetsnet3inputSeries3893targetSeries4+ 13. Set up the di%ision of data.
/. net.!ivi!eParam.trainRatio * ;1<11+ =. net.!ivi!eParam.valRatio * =<11+ C. net.!ivi!eParam.testRatio * =<11+
;ith these settin#s, the input %ectors and tar#et %ectors will $e randoml& di%ided, with 7< used for trainin#, 15 for %alidation and 15 for
testin#.
17. Train the network. The network uses the default e%en$er#2 Dar?uardt al#orithm trainlm" for trainin#. 4or pro$lems in which
e%en$er#2Dar?uardt does not produce as accurate results as desired, or for lar#e data pro$lems, consider settin# the network trainin# function to Aa&esian )e#ulari=ation trainbr" or Scaled ConBu#ate Gradient trainscg", respecti%el&, with either
D. net.train?cn * >trainbr>+ net.train?cn * >trainscg>+
To train the network, enter: 6net3tr7 *
trainnet3inputs3targets3inputStates3la"erStates4+ urin# trainin#, the followin# trainin# window opens. This window
displa&s trainin# pro#ress and allows &ou to interrupt trainin# at an& point $& clickin# Stop Training.
This trainin# stopped when the %alidation error increased for si' iterations, which occurred at iteration 7<.
1>. Test the network. (fter the network has $een trained, &ou can use it to compute the network outputs. The followin# code calculates the
network outputs, errors and o%erall performance. Note that to simulate a network with tapped dela& lines, &ou need to assi#n the initial %alues for these dela&ed si#nals. This is done
withinputStates and la"erStates pro%ided $& preparets at an earlier sta#e. 21. outputs * netinputs3inputStates3la"erStates4+ 2. errors * gsubtracttargets3outputs4+ 22. per&ormance * per&ormnet3targets3outputs4 2B. per&ormance * 2/. 2=. 1.11/2 2C.
@7. 9iew the network dia#ram. 2D. viewnet4
@>. /lot the performance trainin# record to check for potential o%erfittin#. B1. &igure3 plotper&ormtr4
This fi#ure shows that trainin#, %alidation and testin# errors all
decreased until iteration 6. It does not appear that an& o%erfittin# has occurred, $ecause neither testin# nor %alidation error increased $efore iteration 6.
(ll of the trainin# is done in open loop also called series2parallel architecture", includin# the %alidation and testin# steps. The t&pical
workflow is to full& create the network in open loop, and onl& when it has $een trained which includes %alidation and testin# steps" is it
transformed to closed loop for multistep2ahead prediction. ikewise, the R %alues in the GUI are computed $ased on the open2loop trainin# results.
31. Close the loop on the N()* network. ;hen the feed$ack loop is open on the N()* network, it is performin# a one2step2ahead
prediction. It is predictin# the ne't %alue of y t " from pre%ious %alues of y t " and x t ". ;ith the feed$ack loop closed, it can $e used to perform multi2step2ahead predictions. This is $ecause predictions of y t " will $e used in place of actual future %alues of y t ". The followin# commands can $e used to close the loop and calculate closed2loop performance
B2. netc * closeloopnet4+
BB. netc.name * 6net.name > ) ,lose! oop>7+ B/. viewnetc4 B=. 6xc3xic3aic3tc7 * preparetsnetc3inputSeries3 893targetSeries4+ BC. "c * netcxc3xic3aic4+ B;. per&c * per&ormnetc3tc3"c4 BD. per&c * B. /1. 2.D;// /.
@. )emo%e a dela& from the network, to #et the prediction one time step earl&.
/B. nets * remove!ela"net4+
//. nets.name * 6net.name > ) Pre!ict $ne Step Ahea!>7+ /=. viewnets4 /C. 6xs3xis3ais3ts7 * preparetsnets3inputSeries3 893targetSeries4+ /;. "s * netsxs3xis3ais4+ /D. earl"Pre!ictPer&ormance * per&ormnets3ts3"s4 /. earl"Pre!ictPer&ormance * =1. =. 1.11/2
=2.
4rom this fi#ure, &ou can see that the network is identical to the pre%ious open2loop network, e'cept that one dela& has $een remo%ed from each of the tapped dela& lines. The output of the network is then y t J 1"
instead of y t ". This ma& sometimes $e helpful when a network is deplo&ed for certain applications.
If the network performance is not satisfactor&, &ou could tr& an& of these approaches:
• )eset the initial network wei#hts and $iases to new %alues with init and train a#ain see +Initiali=in# ;ei#hts+init"".
• Increase the num$er of hidden neurons or the num$er of dela&s. • Increase the num$er of trainin# %ectors.
• Increase the num$er of input %alues, if more rele%ant information is a%aila$le.
• Tr& a different trainin# al#orithm see +Trainin# (l#orithms+".
To #et more e'perience in command2line operations, tr& some of these tasks:
• urin# trainin#, open a plot window such as the error correlation plot", and watch it animate.
• /lot from the command line with functions such
as plotresponse, ploterrcorr and plotper&orm. 4or more information on usin# these functions, see their reference pa#es."
(lso, see the ad%anced script for more options, when trainin# from the command line.
Kach time a neural network is trained, can result in a different solution due to different initial wei#ht and $ias %alues and different di%isions of data into
trainin#, %alidation, and test sets. (s a result, different neural networks trained on the same pro$lem can #i%e different outputs for the same input. To ensure that a neural network of #ood accurac& has $een found, retrain se%eral times.
There are se%eral other techni?ues for impro%in# upon initial solutions if hi#her accurac& is desired. 4or more information, see Impro%e Neural Network Generali=ation and (%oid %erfittin#.
