This section introduces the commands you need to work with dynamic neural networks. Examples can be found in Section 8.2, Examples.
8.1.1 Initializing and Training Dynamic Neural Networks
There are two commands to obtain dynamic model structures. NeuralARXFit is used to model dynamic systems with input signals, and NeuralARFit is used to model time series where there is no external input signal. As the names indicate, they produce neural ARX and neural AR models as described in Section 2.6, Dynamic Neural Networks. The term neural AR(X) will be used when either a neural ARX or a neural AR model can be considered. There is no restriction on the dimensionality of the input and output signals; that is, the package supports multi-input, multi-output (MIMO) models. In contrast to other neural network types in the Neural Networks package, there are no specific commands for initialization. Instead, if you only want to initialize the dynamic neural network, you use the training commands with 0 training iterations.
Both commands call NeuralFit internally (see Section 7.1, NeuralFit). Algorithmic details are described in Chapter 7, Training Feedforward and Radial Basis Function Networks. There are quite a few input argu-ments, and depending on how they and the options are chosen, a large variety of different models and algorithms may be obtained.
NeuralARXFit@u, y, 8na, nb, nk<,type,nhD
initializes and estimates a neural ARX model using input data u, output data y, regressor8na, nb, nk<, and neural network of type with hidden neurons nh NeuralARXFit@u, y, 8na, nb, nk<, type, nh, uv, yv, iterationsD
initializes and estimates a neural ARX model using input data u, output data y, regressor8na, nb, nk<, and neural network of type with hidden neurons nh, also submits validation data uv and yv,
and specifies the number of training iterations NeuralARXFit@u, y, NeuralARX, uv, yv, iterationsD
continues the training of an already-existing neural ARX model; the arguments uv, yv, and iterations are optional
Initializing and training neural ARX models.
NeuralARXFit returns a list of two variables. The first variable is the trained neural ARX model, with head NeuralARX, and the second is a training record.
An existing neural ARX model can be submitted for more training by setting the argument NeuralARX equal to the neural ARX model or its training record.
The format of the training data u and y is described in Section 3.2, Package Conventions. The regressor is specified as indicated in Section 2.6, Dynamic Neural Networks. The number of lagged outputs to be included in the regressor is given by na, and the number of lagged inputs is given by nb. The delay is given by nk. For SISO systems, these indices are given as three (positive) integers. For MIMO systems, each one of them should be a list with one component for each input or output.
The neural network type indicated by the argument type should be either FeedForwardNet or RBFNet.
The number of neurons (and layers for FF networks) is specified by nh as described in Section 5.1.1, Initialize-FeedForwardNet, and Section 6.1.1, InitializeRBFNet. A linear model is obtained by choosing type = FeedFor wardNet and nh = 8 <.
Time-series models are obtained with NeuralARFit in similar fashion to NeuralARXFit. The only differ-ence is that only the number of lagged outputs have to be indicated in the call.
NeuralARFit@y, na,type,nhD initializes and estimates a
neural AR model using output data y,
regressor indicated by na, and the neural network type with hidden neurons indicated by nh
NeuralARFit@y,
na,type, nh, yv,iterationsD initializes and estimates a neural AR model using output data y, regressor indicated by na, and the neural network type with hidden neurons indicated by nh, also submits validation data yv and specifies the number of training iterations NeuralARFit@y,
NeuralAR,yv, iterationsD
trains an existing neural AR model further, where the arguments yv and iterations are optional
Initializing and estimating neural AR models.
NeuralARFit returns two arguments just as NeuralARXFit does; the first one is a model of type Neu ralAR, and the second argument is the training record.
An existing neural ARX model can be submitted for more training by setting the argument Neural ARX equal to the neural ARX model or its training record.
In addition to the compulsory arguments of NeuralARXFit and NeuralAR, you can also specify a number of additional arguments:
è The number of training iterations to be applied in the call to NeuralFit.
è Validation data uv and yv, so that the RMSE can be computed on validation data after each training iteration. The returned trained model is the model that gives the smallest RMSE on the validation data. This can be the model at an iteration before the last one. In that case the model is obtained by stopped search, which can be useful to avoid overfitting. See Section 7.5, Regularization and Stopped Search, for more details.
NeuralARXFit and NeuralARFit have no options of their own, but because they rely on initialization and training of FF or RBF networks, you can submit any option applicable to InitializeRBFNet or Initial-izeFeedForwardNet and NeuralFit.
During the training, intermediate results are displayed in a separate notebook, which is created automati-cally. After each training iteration you can see the value of the RMSE. The step length control parameter of
the minimization algorithm is also shown; see Chapter 7, Training Feedforward and Radial Basis Function Networks. If you submit validation data to NeuralFit, then you also get the RMSE on this data set.
At the end of the training, a plot is displayed showing the RMSE reduction as a function of the iteration number.
Using the various options of NeuralFit, as described in Section 7.7, Options Controlling Training Results Presentation, you can change the way the training results are presented.
There are often different warning messages given at the end of the training, providing some feedback as to the success of the training. By inspecting the error reduction plot, you can usually tell whether more itera-tions are necessary. Specifically, if the curve exhibits a reasonable downward slope by the end of the train-ing, you should consider continuing the training.
The first output argument, the neural AR(X) model, consists of two components. The first component is the FF or the RBF network, which the model is based on. The second component is a replacement rule, Regres sor->{ na, nb, nk}, specifying the regressor by the three indices na, nb, and nk.
The second output argument of NeuralAR(X)Fit is the training record. Its head is NeuralFitRecord, which contains two components. The first component is a copy of the trained neural AR(X) model and the second component is a list of rules specifying intermediate values of parameters and the RMSE during the training. In Section 7.8, The Training Record, you learn how to extract the information from the training record. You can also use NetPlot to plot some information.
8.1.2 NetInformation
The command NetInformation can be used to obtain some written information about the neural dynamic model.
NetInformation@NeuralARHXLD
presents some information about the network model
NetInformation.
This information specifies the regressor and the FF or RBF network used in the model.
8.1.3 Predicting and Simulating
Unlike the other neural network types, a dynamic model cannot be applied directly on a data input vector.
The reason for this is that the models are dynamic and you have to consider sequences of data to yield simulations and predictions. The commands for this are described in the following.
The one-step ahead prediction y` Ht » t - 1L is obtained as described in Section 2.6, Dynamic Neural Networks.
It is based on measured outputs up to time t - 1.
It is often interesting to check if the model is capable of performing predictions several time-steps ahead;
that is, to increase the prediction horizon. Consider, for example, the two-step ahead prediction, y` Ht » t - 2L, where past values only up to yHt - 2L are given. This is accomplished by using the estimate y` Ht - 1 » t - 2L in place of the missing yHt - 1L value to do the normal one-step prediction. This procedure may be similarly extended to obtain larger prediction horizons.
If the prediction horizon is infinite, it is said that the model is being used in simulation. Then, measured outputs are not used at all, and all outputs in the regressor are replaced by model outputs.
Predictions can be obtained with the command NetPredict.
NetPredict@u,y,NeuralARXD predicts the output signal y using the model NeuralARX and input data u;
default prediction horizon: 1
NetPredict@y,NeuralARD predicts the time series y using the model NeuralAR;
default prediction horizon: 1
Predicting future outputs.
The prediction error, which is the difference between predicted and true output, can be obtained using NetPredictionError in the following way.
NetPredictionError@
u,y,NeuralARXD
computes the prediction error of the model
NeuralARX applied to data uHinputL and y HoutputL;
default prediction horizon: 1
NetPredictionError@y,NeuralARD computes the prediction error of the model NeuralAR applied to time series data y; default prediction horizon: 1
Obtain the prediction error.
NetPredict and NetPredictionError have the following options.
option default value
PredictHorizon 1 indicates the prediction horizon;
if set to Infinity, a simulation is obtained MultiplePrediction False obtains a series of predictions with horizons 1,
2, …, PredictHorizon if set to True InitialOutput Automatic initial values of the lagged output values InitialInput Automatic initial values of the lagged input values
Options of NetPredict and NetPredictionError.
The two options InitialOutput and InitialInput can be used to set the values of the regressor at the beginning of the prediction. The default is to use the first available value in the submitted data; that is, yH-1L, yH-2L, … are set identically to yH1L and similarly for uH-1L, uH-2L, ….
You can set InitialOutput and InitialInput to other values according to the following rules.
For SISO models you can set the options to:
è Real numbers; in which case all lagged inputs and outputs are set to these real numbers.
è Sequences of lagged values that are lists of real-valued numbers, such as 8yH-1L, yH-2L, ..., yH-naL< or 8uH-1L, uH-2L, ..., uH-nk- nb+ 1L<.
For MIMO models, uHtL and yHtL are vectors with the number of components equal to the number of inputs and outputs, respectively. Also the structural indices na, nb, and nk are vectors. In this case the options InitialOutput and InitialInput can be set to: