Sobol’ sensitivity analysis

Let’s first deal with the variance-based method and its results. As stated above, it gives to the final users the first and total order sensitivity index with quite fair computational costs. At higher costs, it gives also the second order index, referred to each possible couple of variables. Recalling eq. 5.20, that constrains the sum of all the sensitivity index to be equal to 1. Though, this is not true for the total sensitivity parameter, since it includes for each variable all its interactions with the other ones1; it is important to remember these facts to properly understand and read the result of the analysis.

Variable S1 S1,Confidence Lev. ST ot S_{T ot,Confidence Lev.}2

x1 0.3980 0.0320 0.5485 0.0384 x2 0.3919 0.0371 0.5527 0.0423 x3 0.0037 0.0040 0.0063 0.0006 x4 0.0028 0.0044 0.0059 0.0006 x5 0.0060 0.0045 0.0064 0.0005 x6 0.0039 0.0041 0.0063 0.0005 x7 0.0013 0.0036 0.0063 0.0005 x8 0.0032 0.0039 0.0062 0.0006 x9 0.0044 0.0040 0.0063 0.0006 x10 0.0049 0.0044 0.0067 0.0005 . . . .

Table 6.1: Results of Sobol’ sensitivity analysis for DTLZ1, from appendix A.2 on test function. It reports the values of the variables from the first to the tenth and for the first objective function only.

From table 6.1 it can be seen the result given by sensitivity analysis. Here are reported just 10 out of the 14 variables of problem DTLZ1, however it is quite evident which are the most influential variables: x1 and x2. In fact, observing just the first two

order indexes, they contribute for almost 80% of the total objective function variance, while dealing with the total sensitivity, each one contributes for about 55%, hence considering their interaction, one would almost completely get the behaviour of the real test function. Table reports also the confidence level of each index evaluation: here with confidence level it is denoted the maximum deviation from the mean value of a given prediction, which is calculated with a confidence parameter of 95%. Hence to properly measure the goodness of given results it is necessary to analyse at the same time both the sensitivity index and its confidence level. Running several times the code

1_{Hence, summing up all of the total order sensitivity index will give a value larger than one, until} the function is a linear relation

2_{The Confidence Level is the maximum deviation possible from the mean value of a given prediction,} calculated with a selected confidence parameter.

75 6.1. Sensitivity analysis application

Figure 6.1: Histogram of Sobol’ sensitivity analysis referred to data in tab. 6.1.

with different parameter inputs, as described in section 5.3.3, just modifying the sample dimension3 to perform the analysis, this value can drastically change. Selecting a small sample, all the confidence levels increase a lot, therefore taking away the meaning of the analysis. On the other hand, choosing a much larger number of points would not realize better and better sensitivity evaluation: depending on the number of function variables one needs to choose properly the dimension sample.

Figure 6.2: Variability of Confidence Level and computational time versus sample dimension for DTLZ2 problem. Black symbols refer to the Confidence Level left axis, while the blue ones refer to the Computational Time right axis. Average values for the indexes are: S1 = 0.4491 and ST ot= 0.5558.

Dealing with the test function DTLZ2, as reported in fig. 6.2, it is evident how much steep is the variation of confidence for adding points in small samples than in

3_{The sample dimension (N ) term indicates the points selected for evaluating each variable variance.} Dealing with a k -dimensional test function, the total number of points in the sample will be k(N + 2), as already seen.

bigger ones. Doubling the sample size does not improve the estimation over the sen- sitivity index. On the other hand the computational time grows almost linearly with the sample dimension. Then it is better to choose a trade-off value to perform a good sensitivity estimation, without unnecessarily burden on the model, also depending on the complexity of evaluating objective functions.

Typically for the problems at hand this parameter has been set from 5 to 15 times the number of variables defining the function: as reported in A.2, DTLZ2 has 22 variables and a value of N = 250 ÷ 350 gives a good trade-off between accuracy and costs.

Interaction variables S2 S2,Confidence Lev.

x1 x2 0.1483 0.0700 x1 x3 0.0157 0.0546 x1 x4 0.0187 0.0556 x1 x5 0.0172 0.0553 . . . . x2 x3 -0.0021 0.0515 x2 x4 -0.0020 0.0519 x2 x5 -0.0010 0.0513 . . . . x3 x4 -0.0003 0.0066 x3 x5 -0.0012 0.0066 . . . .

Table 6.2: Results on second-order Sobol’ sensitivity analysis for the first objective function of DTLZ1, from ch. A.2 on test function; just few interaction terms are reported here.

Figure 6.3: Histogram of second order Sobol’ sensitivity analysis referred to data in tab. 6.2.

A further result can be obtained by the variance-based method of sensitivity analysis and it is the second-order index of the interaction between different variables by eq. 5.23, as reported in table 6.2. Also higher levels of interaction could be evaluated with

77 6.1. Sensitivity analysis application

further equations, but these would require much more computational efforts for kind of later useless knowledge: in fact, the higher is the interaction the lower the contribute to the objective function variance. Moreover, higher level of interactions needs much larger sample set to provide fairly acceptable results: data reported on second-order table is generated with a sample size N = 1500, and it presents a confidence level more or less doubled with respect to data in table 6.1, based on a sample size of N = 300. This happens because the eq. 5.23 brings with itself the errors of first order indexes, leading to larger deviation.

However, observing first, second and total-order sensitivity indexes of the variables x1 and x2 and summing up the first two, the resulting value is, with quite good ap-

proximation, the total index. All the second order indexes have much lower values with respect to the first one, even if this could be not always true for other kind of functions. Although it can be found in table 6.2 negative values, these are results of numerical and approximation errors, again due to the second-order indexes equation; all of them can be view as null, considering also the relative confidence level.

All the Sobol’ analysis performed on the test function described in appendix did not really required the second-order indexes calculation, or because their contribution is negligible and it does not bring useful knowledge to the behaviour of the chosen function, due to their definitions.