Sensitivity Analysis: The Morris Method

All input variables listed in Table 4-8 were tested for their importance in the estimation of yield of the hypothetical urban water supply system using the Morris method. The Morris method algorithm provides three indices:

μ - The overall sensitivity effect due to all first- and higher-order effects.

μ* - The ‘true’ importance measure, free of any non-monotonic input to output behaviour that could be present in μ.

σ - The possible non-linearity of an input variable or interactions of an input variable with other variables. The Morris method does not distinguish between these two effects.

The Morris algorithm requires the selection of the number of levels p, Δ as a multiple of 1/(p - 1), the number of trajectories to perform and a random seed.

The experiments in this study were performed using a variety of algorithm settings and random seeds as noted in Table 4-10. The level p determines the number of equally spaced sampling points in the variables’ range (i.e. the sampling resolution) from which two are sampled with a Δ change between. The higher p is, the higher the number of possible points that can be sampled from the variable space. Different number of levels, p, were used so that

the sampling can be over a sparse and a fine resolution. Different Δ were also used so that small and broad perturbations were produced. The number of trajectories required, r, and the number of input variables, k, determine the number of required model simulations: r(k + 1). The random seed ensures that different sets of trajectories are constructed for each p and Δ setting.

The results from all experiments are combined for simplicity and shown in Table 4-11. The reason for the various settings and the combined results is so that any bias that may occur from a particular setting is avoided and that all effects from small and large Δ changes are captured.

Table 4-10. Algorithm Settings for the Morris Method Sensitivity Analysis Experiment. Experiment Number of

Trajectories Level Δ Seed

1 10 4 1 18936437 2 10 4 1 874366872 3 10 6 2 18936437 4 10 6 2 874366872 5 10 6 3 18936437 6 10 6 3 874366872 7 10 6 4 18936437 8 10 6 4 874366872 9 10 8 3 18936437 10 10 8 3 874366872 11 10 8 4 18936437 12 10 8 4 874366872 13 10 8 5 18936437 14 10 8 5 874366872 15 20 4 2 18936437 16 20 4 2 874366872

Individual results of the each experiment listed in Table 4-10 are given in Appendix B. The results in Appendix B show that the μ, μ* and σ measures for most input variables are relatively stable for the various setting used. Of particular significance is the target storage curve point 3 variable which results in zero μ, μ* and σ measures for all experiments except experiment 16. Similarly the relative position point 1, stage 2 percentage restrictable and lower RRC position variables show that in some experiments they return zero μ, μ* and σ measures, while in other experiments they have a low importance measure. The combined results given in Table 4-11 reflect a more reliable estimation of the importance of the input variables as the effects of the various Morris algorithm settings are capture.

Included in Table 4-11 is the μ, μ* and σmeasures and the μ* rankings. μ* is used for ranking as it is a better measure of the total sensitivity of an input variable because it considers only the magnitude of the change, whereas μ also considers the direction of the change and can therefore include cancelling out of effects. The μ, μ* and σ results are also presented graphically on a μ-σ axis in Figure 4-8. Noticeably there are only a few input variables that show any importance to the estimation of yield. These variables are labelled in Figure 4-8, whereas the remaining variables that have negligible μ, μ* and σ results in comparison are not labelled for clarity.

The streamflow variable is clearly the most important input variable, with the reliability, consecutive months and the upper RRC showing noteworthy effects on the yield estimate, as indicated from the position along the μ/μ*-axis. The consecutive months threshold, reliability threshold and streamflow show large interaction or non-linearity behaviours as indicated by the large σ indices.

Interestingly, most of the input variables show a negative input to output behaviour, this is revealed by the difference between μ and μ*. When these indices are equal but opposite (note that μ* will always be positive) it shows that the variable have a monotonically negative input to output behaviour. When they are not equal but μ is still negative it shows that the input to output behaviour is non-monotonic but tends to be negative. The streamflow, consecutive months threshold and the rainfall variables all produce equal μ and μ* indices. This indicates that when they have been perturbed they produce an output change in the same direction, i.e. when they are increased, the yield estimate also increases, when they are decreased, the yield decreases.

There are a number of variables that show a zero influence on the output. These include the initial storage volume variables, the target curve points 3 and 4, stage 3 and stage 4 percentage restrictable, stage 2 and stage 3 relative position, and the worst restriction stage threshold. The initial storage volume variables are zero due to the iterative handling procedure used. The zero measures for the stage 2 (and 3) relative position and stage 3 percentage restrictable suggests that the worst severity restriction stage that is triggered is stage 2 restrictions, i.e. the system never triggers a stage 3 restriction. This is also indicated by the zero results for the worst restriction stage, which itself suggests that total system storage does not drawdown enough for the worst restriction stage (either stage 3 or 4) to be the critical threshold. The zero effects of the target curves points shows that changing these points does not effect the yield estimate.

Table 4-11. Combined Results of the Morris Method Experiments.

Factor μ μ* σ μ* Ranking

Streamflow 6151 6151 996 1

Rainfall 852 852 189 5

Evaporation -629 640 234 9

Evaporation Factor A for Reservoir A -658 713 391 8 Evaporation Factor A for Reservoir B -728 728 192 7 Evaporation Factor B for Reservoir A -294 295 244 12 Evaporation Factor B for Reservoir B -315 315 211 11 Volume to Surface Area Relationship 200 205 209 14 Temporal Disaggregation Factors -433 444 264 10

Climate Index -737 737 270 6

Upper RRC Position -1169 1195 692 3

Lower RRC Position 44 44 145 16

Base Demand Position -204 210 241 13 Stage 1 Percentage Restrictable 108 129 204 15 Stage 2 Percentage Restrictable 23 31 93 18 Stage 3 Percentage Restrictable 0 0 0 20 Stage 4 Percentage Restrictable 0 0 0 20 Stage 1 Relative Position -31 31 103 18 Stage 2 Relative Position 0 0 0 20 Stage 3 Relative Position 0 0 0 20 Consecutive Months in Restriction 1190 1190 2063 4 Worst Severity Restriction Stage 0 0 0 20 Supply Reliability -3891 3891 1438 2 Target Storage Curves – Point 2 -31 31 102 17 Target Storage Curves – Point 3 0 0 0 20 Target Storage Curves – Point 4 0 0 0 20 Initial Volume of Reservoir A 0 0 0 20 Initial Volume of Reservoir B 0 0 0 20

Figure 4-8. Combined Results of the Morris Method Experiments.

The following points summarise the findings from the Morris method experiments:

1. The most influential variables in the estimation of yield for the hypothetical urban water supply system are the streamflow, reliability of supply threshold, the upper RRC position and the consecutive months in restriction.

2. Interactions and/or non-linearity behaviour exists, primarily in the consecutive months, reliability of supply and streamflow variables.

3. The most severe restriction stage imposed on this system is stage 2. This effectively negates the use of several variables, namely the worst restriction stage (which had a range of 3-4), stage 3 and stage 4 percentage restrictable, and stage 2 and stage 3 relative position.

4. The target curve points 2 and 3 are not influential, either suggesting that the storages never fill past curve point 2 (65,000 Ml total storage) for any Morris method experiment, or that the yield is not sensitive to changes of target curve point 3 and point 4.

5. The Morris method has successfully been applied to test the input variables used in the estimation of yield of an urban water supply system. It has efficiently identified

variables with negligible influential to the estimation of yield and also revealed system behaviour, such as highlighted in the points above.

The Morris method identified several variables that have zero influence on the estimation of yield and should therefore be neglected from further SA experiments to decrease the number of simulations required. Nevertheless, all variables as listed in Table 4-8 will be used in preliminary experiments using the variance based FAST and eFAST methods to confirm the results of the Morris method. Once these results are confirmed, the FAST, eFAST and Sobol’ methods will be used a reduced number of variables.