Accuracy with the Bumps function

Problem Description

As the last study was found to be very informative, a similar study as been deemed worthy of attention.

Although the results of the sine wave study were enlightening and beneficial, it can be argued that the time-series involved showed very little similarities to the financial time-series used in this project. Therefore, showing that the ANN and the estab-lished parameters work very well with a sine wave, does not sufficiently infer that they are correct for our studies. However, the sine wave study can be extended to establish the inference we require. The Bumps function⁷ is well known and com-monly employed when testing pattern matching technologies, because it is a difficult function for systems to learn and predict. In [Booker, 2005] it is utilised along with the Blocks, Doppler and HeaviSine functions. The Bumps function has been chosen because when plotted it demonstrates the same random and volatile characteristics evident in financial time-series. By using this function it is possible to establish a similar study to the sine wave study, but more in-line with financial time-series.

Justification of Choice

This study is worthwhile for two reasons. Firstly, as stated in the Problem Descrip-tion, this study can be used as a control study to establish whether the ANN is set up correctly. By using a well defined function which mimics financial time-series, we can determine to a higher level of conclusiveness whether the ANN parameters are correct for this investigation and that the ANN is functioning correctly. Secondly, the Bumps function is more complex and harder to learn than the simple sine function, therefore undermining the argument that the sine wave example was trivial, meaning its results are insignificant.

Results regarding exposure to the function, shown in figure 6.7, were significantly negative and because they were conclusive on a trivial scenario it is reasonable to assert that a more complex study would provide similar results. Therefore, this trial will not be extended here. However, trials two and three regarding the size of the data-sets (or resolution) will be re-applied, because one could argue that the sine wave study was too trivial for resolution to be a factor. The Bumps function is well known and used because it has proved to be a difficult function to teach to pattern matching

7See Appendix B for a plot of the function together with its definition.

systems. If the resolution of training sets is shown to be a redundant consideration in this study, then within this project we need not deliberate on it further. Therefore, we know that the ‘Data Multiplication’⁸pre-processing methods described in [Zemke, 2002] will not be necessary.

The target output will be the Bumps function with index (which is t) in the range of [0, 1].

The input functions will be:

• 20 - Bumps - this is the bumps function inverted and offset.

• Bumps/index + mod(index, 0.5) - this divides the bumps at a varying rate and adds a varying offset.

• Bumps/4 such that the time-series is sub-divided into tenths, with every odd tenth set to zero⁹.

• Bumps/4 such that every even tenth block is set to zero - between inputs 3 and 4 the bumps function can be made, but to a diluted magnitude.

These inputs were chosen for the same reason as in the sine wave study.

Hypothesis

The ANN will accurately learn and predict the Bumps function using the provided in-puts. The performance will not match that shown by the ANN in the sine wave study, however when visably examining the test predictions it is clear that they are accurate.

Comparing trials, the study will show that resolution, that is size of training and testing data-sets, will not affect the accuracy of predictions.

Study Framework

The ANN settings will, once again, be as follows:

• Each time the connections’ weights are to be re-initialised to random values, within the range [-1, 1]

• Shuffling: yes

• learning rate: 0.2

• learning algorithm: back-propagation with online weight updates.

• epoch: 10000

• topology 4-4-1

• activation function: logistic

8Data Multiplication is used to increase the resolution of time-series data. Remember however, this does not mean the ANN sees a larger section of the functions, but rather sees the same section in more detail.

9For example, if the time-series contained one hundred points, set the points with the following blocks to zero, 1-10, 21-30, 41-50, 61-70, 81-90.

As can be seen in Figure 6.8, predictions in the sine study were extremely consistent.

Therefore, to avoid unnecessary additional trials only two iterations of training and learning will initially be executed. If results fail to show sufficient consistency then the study will revert to five iterations per trial.

There will be 3 trials. The data sets will be created by taking index values in the following manner:

• Increment the index in 0.005 steps, making a set of 200.

• Increment the index in 0.002 steps, making a set of 500

• Increment the index in 0.001 steps, making a set of 1000.

Once again, the initial 80% will be used for training and the remaining 20% will be used for testing.

Results

Tables 6.10, 6.11 and 6.12 show the accuracy measures for the different data-set sizes.

Figure 6.9 shows the corresponding test predictions charts.

Table 6.10: The accuracy measures computed on the test predictions made during the Bumps function experiment, using 200 input-target pairs.

Bumps Experiment, 200 Data-Points

Trial Absolute Direction Mean Information 1 0.99852 0.53846 0.12804 0.09649 2 0.99853 0.53846 0.11405 0.08523

Table 6.11: The accuracy measures computed on the test predictions made during the Bumps function experiment, using 500 input-target pairs.

Bumps Experiment, 500 Data-Points

Trial Absolute Direction Mean Information

1 0.99678 0.52 0.13756 0.13769

2 0.99657 0.56 0.15069 0.15111

Table 6.12: The accuracy measures computed on the test predictions made during the Bumps function experiment, using 1000 input-target pairs.

Bumps Experiment, 1000 Data-Points Trial Absolute Direction Mean Information

1 0.99766 0.56716 0.12523 0.19773 2 0.99785 0.58209 0.12000 0.18936

Figure 6.9: These charts show the results of training and testing with the different sizes of sampling data.

Figure 6.9 shows that the system has learnt and predicted the Bumps function well.

The results show that the prediction quality is unaffected by the resolution of the data-sets.

Conclusions

In conclusion, the system has accurately learnt and predicted the Bumps function. As the Bumps function shows random and volatile characteristics, which can be observed in financial time-series, we can assert that the ANN is functioning correctly and the parameters are adequate for our dataset.

Results suggest that the the resolution of the dataset had an insubstantial effect on the quality of learning and testing and as a result resolution will not be further considered in this project.