Changes in Methodology

As in the previous chapter, an investigation is conducted into type-2 fuzzy logic and specifically how the change in the FOU size affects performance. However, as param- eters of the wind cannot be controlled as in simulation they will simply be recorded. This means that every run has a unique level of variation and wind levels that cannot be reproduced.

This work has three objectives, namely:

1. to observe the effect of FOU size upon performance;

2. to demonstrate how real-world environments present a difficult task for con- trollers under test in comparison to the simulation environment previously used. 3. to show which configurations perform best at differing degrees of variability

within the environment.

6.4 Type-2 Main Study

of grouping the experiments per FOU sizes. It is anticipated that the weather condi- tions will not change significantly over the time it takes to run one experiment for each considered controller configuration, estimated at 3 hours.

Based on shortcomings identified when evaluating the results of the pilot study, three changes have been made:

1. A required turn in order to complete the course is introduced. The new course is shown in Figure 6.5. This turn has been added in order to increase the amount of work that is required for each controller to complete the experiment and therefore allow ‘better’ controllers to differentiate themselves from those which perform poorly.

2. The filming component is eliminated. In these experiments, the operator does not record the course. Recording adds complexity to the experiment, as the operator must control the boat from a laptop, while simultaneously recording it and being in the correct position to stop the boat as it reaches the end point. The videos made as part of the pilot study did not really assist the task and therefore they will be eliminated.

3. A time limit of fifteen minutes to complete the run is added. After this time, the run is considered a failure and the boat is removed from the water as soon as it reaches any side of the pond and the data is discarded.

6.4.2

Experiments

The way point data is collected every day and uploaded to the boat as discussed in Sec- tion 6.3.1. Each experiment is then performed following the modified run procedure:

1. The boat hardware system is reset, realigning all motors. The controller is ini- tialised and begins to sense values and change actuator positions in response. 2. The robot is lowered into the water and aligned by the operator to face directly

towards the opposite side (e.g. towards point B). The boat is released and a timer is started.

6.4 Type-2 Main Study

Figure 6.5: Modified boat course including newly added turn indicated by θ. Start point is shown by ‘A’ and end point shown by ‘B’

3. When the boat reaches the end point, the end time is noted and the run is consid- ered finished.

4. If the time exceeds the limit, the boat is removed from the water as soon as it is possible to do so (i.e. it reaches an edge).

5. If the boat reaches the end point, it is removed from the water and walked back to the starting point, where the controller is halted and the data stored.

These experiments were run during the week of 3rd December of 2012, when the overall weather was cold but relatively calm. There were no adverse weather condi- tions such as storms. However, sufficient wind to sail the boat was present with overall

6.4 Type-2 Main Study

6.4.3

Results

Figure 6.6 depicts how the calculated RMSE changes as the FOU size is increased. It can be observed that there is a small increase as the FOU size moves from size 0 to 10, and after this point it starts to decrease. Additionally from this graph it can be observed that the RMSE for FOU size 40 decreases by a noticeable amount. This further supports the hypothesis that a real-world environment will allow better or worse performing controllers to become apparent, more so than in the experiments performed in previous works. 0 10 20 30 40 140 150 160 170 180 ● ● ● ● FOU Size RMSE

Figure 6.6: RMSE of the different sizes of FOU under test

Table 6.2 provides a more quantitative outlook on the data obtained from the exper- iments. The runs column indicates how many runs were performed for each FOU size, with the goal being 10 runs. The p-value shown here is the value obtained by perform- ing T-test between the rows FOU size and FOU size 0, with most p-values resulting a

6.5 Discussion

FOU Runs Mean RMSE Std Dev P-Value

0 12 162.95 13.81 N/A

10 11 162.93 10.18 0.27

20 14 157.57 15.86 0.15

40 10 157.04 12.36 0.07

Table 6.2: Mean RMSE with its variance and the p-value result of a t-test between type-1 and the indicated FOU size. A smaller p-value indicates a less significant difference.

statistically significant difference. In order to keep the input data sets the same size for the t-tests, the first 10 runs of each configuration were used.

The RMSE values were calculated using the same calculation as those used through- out this thesis. Overall, the results show the RMSE decreasing, meaning improved performance as the FOU grows. It can also be noted that the RMSE values are much greater than those obtained in previous works — showing that the environment does have more variation present, which makes it a more difficult task to complete. The standard deviation is also much larger in these experiments than those in the previous experiments. It is believed that this occurs for the same reasons as for the RMSE value — the environment introduces more variation, within a smaller data set.

6.5

Discussion

A significant number of issues arose while working in the real world that were absent during the simulation experiments. Some of these issues were anticipated, some not. Overall, the hardware used in the boat was reliable and worked as expected. How- ever, some relatively minor problems occurred, such as the battery life of the platform, which did not provide enough power for a complete day of experiments on a single charge. It has been suggested by Doel and Pai [95] that battery usage is another metric of performance that should to be considered. The goal would be minimising the bat- tery usage by reducing the amount of motor usage used during an experiment. This

6.5 Discussion

obviously involve a trade-off, especially when significant variation is present, as each course correction would require motor movement which in turn would use battery power.

The behaviour of the boat in the water was much more variable than anticipated, with many small movements occurring in all directions. For example, the tilt of the boat seems to have a significant impact on how much control the rudder has upon the boat in a given direction, as well as the amount of wind that is converted into forward motion. As the boat has a tilt sensor as part of the digital compass, this could be used in future work to further increase performance, using the tilt as an additional input to the fuzzy system.

Performance and localisation of the GPS sensor on-board the robot was satisfac- tory, and in general it matched the measurements of the external A-GPS device. How- ever, the time taken to obtain a good satellite fix was considerably longer than antici- pated. A-GPS chips are becoming more widespread and hence the replacement of the current GPS receiver with a more accurate device would benefit the accuracy of the position data generated in future work.

One potential issue, to be subject of future investigation and study, is the effect of the update rate of the sensors/actuators. It has already been explained in Section 4.4.1 why the selected rate was used. However, observations show that at some points in real-world experiments, the controller was not able to handle certain conditions, such as when a large gust would blow, with its direction significantly different from the current direction. This would cause the boat to over or under-shoot a turn and therefore dramatically alter the speed.

The FOU sizes used in the main study of this chapter show some changes in results as they increase. However, these changes are smaller than would be expected in a real world with large levels of variation. This supports the idea that the levels of variation present in this real-world environment are not as large as anticipated, either due to the prevailing conditions or the location selected. The relatively small differences obtained between the different FOU sizes seem to imply that FOU choice is a small factor in performance of a fuzzy-based system.

The sample size used in this experiment was established by the methodology and limitations upon potential duration of the study. As the statistical test results (Table 6.2) generally indicate no significant changes, it may be possible that the sample size

6.5 Discussion

needs to be significantly larger, in order for the changes to become apparent. This was attempted to be partially corrected by increasing the difficulty of the task between the pilot and main studies — that of adding in a turn into the course. However, this did not alter the results.

The complexity and difficulty of the course (i.e the number and size of each turn), is one potential reason for the similar performance levels of each different configuration. Additionally, the tendency of the wind to blow as shown by the white arrow in Figure 6.5 (page 152) should also be considered. The fact that the wind most often blows perpendicular to the route required means that overall the course is fairly easy — a human sailor, for example, would have little difficulty in completing a similar course. It may be that orienting the course so that the boat must move into the wind or at a more difficult angle may further differentiate between different controller configurations and ease comparisons.

In the pilot study, the results presented very little variation regarding the RMSE values, and this caused changes to be incorporated into the main study, where an ex- tra turn was added to the course. This should have made the route more difficult to complete, allowing better controllers to show correspondingly better results. This was indicated by a larger spread of RMSE values across different configurations. While the small amount of data collected in the pilot study makes direct comparisons difficult, it was hypothesized that the larger study, with the added turn would show a larger gap in performance between better and worse controllers. This was somewhat supported by the results in the main study.

As has previously been stated, the goal of the work in this thesis is not to develop the best controller possible but to understand the relative performance of each. Due to this, the tuning of the fuzzy controllers under test was not considered important, as each was derived in the same manner. Due to the similarity of the RMSE values obtained, the effect of additional tuning of each of the fuzzy systems may have been a useful area of study. However, this would have been a considerable investment in time and will be left as an avenue for future work.

Overall, this chapter shows that more difficult and variable real-world experiments cause more variable results to be observed. The hypothesis made in Section 6.2.4 is supported by the results in that larger FOU sizes seem to give better performance than

6.6 Summary

this hypothesis, as the differences observed, while larger than those previously found, still do not achieve the magnitude of changes that were anticipated. This has lead to considering other factors for the reasons for this lack of differences. The main considerations are (i) the original type-1 fuzzy sets may have been far from optimal, making it hard to improve upon performance; and (ii) the method by which the type-2 fuzzy logic controllers were derived from the type-1 may have been too simple to ever give good results. Both of these considerations are out of the scope of this thesis but present good opportunities for future work.

6.6

Summary

In this chapter, the same fuzzy logic controllers in the previous chapter were applied to a real-world autonomous sailing boat context, as opposed to simulation. The results obtained show some additional differences than those observed in the previous chapter. The reasons for this are discussed along with reasoning as to why these differences have been found, possible solutions and avenues of future work are then identified.

In the next chapter an in-depth discussion about the work and findings of this thesis is presented. In addition, the ideas for future work and improvements to address some shortcomings are presented.

7

Discussion

7.1

Introduction

In this thesis, the topic of fuzzy logic and specifically, how its behaviour changes across different scenarios was investigated using three case studies: the tipping problem and autonomous sailing robots in simulation and real-world environments. The effects of variation in the environment, how it can affect performance and how it can be intro- duced into the environment was studied. Multiple varieties of fuzzy logic control were investigated, with comparisons between them being the major focus of study. Each variety was evaluated using several different internal configurations, generally deter- mined by the FOU size.

The main motivation behind this work was to be able to identify which factors are likely to cause type-2, dual surface and non-stationary fuzzy logic types to outper- form type-1 and the relative import of such factors, with most focus applied to interval type-2 control. This was intended to act as a starting point for being able to develop techniques for the selection and justification of the type of fuzzy logic control for a given application. Task difficulty in the context of the sailing robot application was defined by (1) the sailing boats defined course, including how many turns and the total cumulative angle; and (2) the conditions under which the sailing occurs, including the wind, water and other sources of environmental variation.

7.2 Evaluation of Aims

7.2

Evaluation of Aims

The aims stated in Chapter 1 (shown in boldface) have been addressed as follows: • To show that variations on standard type-1 fuzzy logic control such as type-

2, Dual Surface, and Non-stationary fuzzy logic control can provide signif- icantly improved performance over standard type-1 fuzzy logic based con- trol systems. This was addressed by using the different varieties of fuzzy logic across experiments of increasing difficulty and complexity utilising the simula- tion environment in Chapter 5. Within each experimental set-up, many variables were kept constant, which enabled us to perform meaningful comparisons. The differences were most obviously shown in the experiments within sections 5.4 and 5.5 (pages 107, 115) where differences in performance are found throughout several different set-ups.

• To study how performance changes as the environment is made more or less complex, by changing the degrees of environmental variation and the task difficulty defined. While moving through the case studies, the experimental environment generally increased in complexity from the very simple Tipper ex- periments in Section 3.4 (Page 54) to the real-world experiments in Section 6.4 (Page 150). In addition, within the simulated sailing study the environment was studied with several combinations of task difficulty and environmental set-up. • To investigate how the internal configuration of a given controller (referred

to as the FOU size) changes the level of performance of type-2, DS and NS based fuzzy systems in comparison with the more standard type-1 based configuration. This was achieved by gradually increasing the range of the FOU sizes used in each case study. The Tipper experiments in Section 3.3 (Page 50) used a narrow range of values (sizes 1 to 4) of FOU, which is increased to a range of 10 to 40 in the real-world experiments. It was anticipated due to the greater variation present in the real world.

• To determine the combination of factors (FOU Size, environmental varia- tion and task difficulty) with which type-2 fuzzy logic would consistently outperform type-1 based control. Each of the case studies in the thesis utilised