Sensitivity Analysis: Variance Based Methods

Two variance based methods are employed in this study: the Fourier Amplitude Sensitivity Test (FAST) and Sobol’ method of sensitivity. The following four sensitivity indices are used in this section to assess the sensitivity of the estimation of yield to changes in the input variables:

Si - The first-order sensitivity effects of the i-th input variable, free of any higher-

order or interaction effects. Si can be calculated via the classic FAST, eFAST

and the Sobol’ SA techniques. These indices should be positive and ΣSi≤ 1.

STi - The total-order sensitivity effects of the i-th input variable which includes the

effects of all possible combinations that the i-th input variable is included in. STi can be calculated via the eFAST and the Sobol’ SA techniques. STi ≥ Si for

the same variable.

Sij - A measure of the first-order interaction effects of the i-th and j-th input

variables, free of the effects of all other interactions and individual effects of the i-th and j-th input variables. Sij

c

ij ij i j

S =S − −S S

can only be calculated using the Sobol’ method and is determined by: . Sij

c ij

S

should always be positive.

- The ‘closed’ (Saltelli, 2002a) effect of the i-th and j-th input variables. This is a measure of effects of the i-th and j-th input variables, including the individual effects (Si and Sj

c ij

S

), and the interaction effect of the i-th and j-th input variables. can only be calculated using the Sobol’ method and should always be positive.

Note that the Sobol’ technique has two commonly used algorithms, Sobol’s (1993) own algorithm and a more accurate and computationally efficient algorithm developed by Saltelli

(2002b). The calculation of the sensitivity indices are done using the same equations, as presented in Section 3.5.3, but the sampling design differ in the two methods. For this study the Saltelli algorithm is used as it provides the same Si and STi

Using all variables given in

indices as the original Sobol’ algorithm but at a lower computational cost, and can also calculate higher-order sensitivity indices which the original Sobol’ method cannot.

Table 4-8, ten preliminary FAST/eFAST experiments were used to confirm the results of the Morris method experiments given in Section 4.4. Table 4-12 details the settings of these experiments. Both the classic FAST and the eFAST algorithms are used to confirm the Morris method results, as well as a grouped experiment using the eFAST algorithm. The accuracy of the FAST and eFAST techniques increases as the number of simulations increases, therefore increasing accuracy experiments were considered until sufficient convergence was reached. Different seeds are used to provide two experiments for each of the same resolution experiments (i.e. same or similar number of model simulations) which are then averaged. Due to the eFAST grouping algorithm the different random seeds produce slightly different number of required model simulations. This can be seen in different number of simulations between experiments 7 and 8 and between experiments 9 and 10. Only 40,000 run classic FAST experiments were performed as this was expected to be sufficiently accurate. The results of these experiments (experiments 1 and 2) are confirmed with the eFAST experiments (experiments 3 to 6). For experiments that have the resolution of sampling (i.e. the same or similar number of model simulations), the results are averaged and presented.

Table 4-12. Settings of the Preliminary FAST Experiments. Experiment Number FAST Algorithm Number of Variables/Groups Number of Simulations Random Seed 1 Classic 28 Variables 40000 9825169 2 Classic 28 Variables 40000 3584381 3 Extended 28 Variables 9884 9825169 4 Extended 28 Variables 9884 3584381 5 Extended 28 Variables 19964 9825169 6 Extended 28 Variables 19964 3584381 7 Grouped 7 Groups 9615 8974561 8 Grouped 7 Groups 9559 3584381 9 Grouped 7 Groups 19627 8974561 10 Grouped 7 Groups 19571 3584381

Table 4-13 presents the averaged first-order sensitivity indices (Si) for the classic FAST

experiments 1 and 2. Acceptable results are gained from these experiments as the sum of Si

is not greater than one. The sum of Si indicates the degree of additivity of the model; the

closer Si is to unity the more additive the model, where the sum of Si is exactly 1 for a

completely additive model. It is clear that the first-order measure is dominated by the streamflow, followed by the reliability threshold. The upper RRC position, consecutive months threshold and the rainfall variables are then the most important, with the remaining variables showing negligible difference in their importance. These results of experiments 1 and 2 confirm the Morris method ranking for the 12 most important input variables in the estimation of yield. Also corresponding with the Morris method are most of the zero importance variables. However, three variables, the lower RRC position, the relative position of intermediate curve 1 and target curve point 2, show a zero Si

Four experiments using eFAST (experiments 3, 4, 5 and 6) were also performed on the individual variables shown in

results whereas their μ values were not.

Table 4-8 to confirm the accuracy and the results of the Morris method experiments. The averaged first-order and total-order sensitivity indices of experiments 3 and 4 are presented in Table 4-14 and the averaged results of experiments 5 and 6 are presented in Table 4-15. The rankings of the 10 most important variable in experiment 5 and 6 correspond to the ranking of the Morris method experiments, confirming the accuracy of the Morris method at screening for important variables. However, errors are present. The results of experiments 3 and 4 sum to less than one, however the averaged results of experiments 5 and 6 sum to greater than one. This is counter to the theory that an increased number of model simulations should provide a more accurate estimation of the sensitivity indices. The source of this error is unknown, it could be due to aliasing or interference between frequencies or due to the issues handling discretely distributed variables. Nevertheless, the results will be used with caution. Between the two sets of experiments, the streamflow, reliability threshold, upper RRC position, consecutive months threshold and the rainfall variables have the same Si ranking and the sensitivity indices have

same order of magnitude. The magnitude of STi and the difference between Si and STi

Table 4-14

from and Table 4-15 indicate that there are high-order effects in all input variables, specifically in the streamflow, reliability threshold, volume to surface area and the evaporation factor A of Reservoir B.

Table 4-13. First-Order Indices (Si

Variable

) for FAST Experiments 1 and 2 (Averaged).

Si Ranking

Streamflow 0.6286 1

Rainfall 0.0112 5

Evaporation 0.0058 9

Evaporation Factor A for Reservoir A 0.0074 8 Evaporation Factor A for Reservoir B 0.0077 7 Evaporation Factor B for Reservoir A 0.0015 12 Evaporation Factor B for Reservoir B 0.0017 11 Volume to Surface Area Relationship 0.0008 13 Temporal Disaggregation Factors 0.0035 10

Climate Index 0.0091 6

Upper RRC Position 0.0284 3 Lower RRC Position 0 17 Base Demand Position 0.0007 14 Stage 1 Percentage Restrictable 0.0003 15 Stage 2 Percentage Restrictable 0.0001 16 Stage 3 Percentage Restrictable 0 17 Stage 4 Percentage Restrictable 0 17 Stage 1 Relative Position 0 17 Stage 2 Relative Position 0 17 Stage 3 Relative Position 0 17 Consecutive Months in Restriction 0.0164 4 Worst Restriction Stage 0 17 Supply Reliability 0.2273 2 Target Storage Curves – Point 2 0 17 Target Storage Curves – Point 3 0 17 Target Storage Curves – Point 4 0 17 Initial Volume of Reservoir A 0 17 Initial Volume of Reservoir B 0 17

Some irregularities and limitations of the eFAST technique become apparent between the two sets of experiments (experiments 3 and 4 and experiments 5 and 6). The most obvious, and previously discussed, is the sum of Si for experiments 5 and 6 is greater than one. There

are also numerous variables that have a zero Si

Table 4-14

in Table 4-15 whereas they are non-zero in . Another observation is that the variables with high Si (streamflow and reliability

threshold) in experiments 3 and 4 increase in experiment 5 and 6, while most other variables decreases. A significant finding from the point of the performance of the eFAST method is the non-zero Si results for the initial storage volumes.

Table 4-14. First-Order Indices (Si) and Total-Order (STi

Variable

) for eFAST Experiments 3 and 4 (Averaged).