Initial Application of Importance Sampling

Initial Application of Importance Sampling

Sections 4.3.3 to 4.3.5 will consider how much work (i.e. how many snapshot evalua-tions) are required to estimate mean annual constraint costs to a given level of precision for a year 1 and year 6 power system background. Results are given when basing the weights for importance sampling on snapshot demand level and the mean of constraint costs in each snapshot. For comparison, results are also given when basing the impor-tance sampling weights on the standard deviation of constraint costs in each snapshot or when evaluating each snapshot (i.e. not using importance sampling).

However, in Section 4.3.1 it was noted that importance sampling methods based on snapshot mean or snapshot standard error require prior estimates of these quantities before importance sampling can be applied. Therefore, the results of these sections can be considered to be a pilot study, where it is assumed these quantities can be known accurately in advance in order to show the potential benefit of importance sampling if appropriate weights, ω, were already known accurately. For this pilot study, weights are calculated using 1000 initial evaluations of each snapshot. In practice, it would be very expensive and inefficient to first estimate importance sample weights using 1000 full simulator evaluations, so Sections 4.4.1 and 4.4.2 will go on to consider applications of importance sampling which use a small number of initial full simulations to estimate weights.

It is possible that no constraint costs are expected to occur in a given snapshot (i.e.

the mean constraint costs of a snapshot are zero), which would result in ω_τ for the snapshot being calculated as 0 via Equation 4.3.1. However, Equation 4.2.5 requires

4.3. Initial Application of Importance Sampling to Estimate Mean

Constraint Costs 73

ω_τ to be greater than zero to apply importance sampling. Therefore, a minimum value of ω_τ was set to 0.001.

Metric for Work Done

Naively, to compare the different importance sampling methods, it may be considered how many evaluations of Equation 4.2.5 are necessary to acquire an estimate of mean annual constraint costs to a given level of precision. However, Section 4.2.2 and Ap-pendix A.1 indicate that this is a misleading metric as it does not account for the amount of snapshots actually evaluated (actual work done) to acquire the estimate.

For example, 20 evaluations from importance sampling which evaluate on average 2000 snapshots actually expects to evaluate fewer snapshots than 3 simulations which eval-uate each snapshot.

Therefore, λ, a metric which accounts for the amount of snapshots actually simulated, will be considered instead. λ is a measure of how many snapshots were actually evalu-ated in terms of the 17520 half hour snapshots in a single year. For example, suppose 3 yearly simulator runs using importance sampling are performed which evaluate 11000, 9950 and 11572 snapshots respectively. Then for these three evaluations the total work done is

λ = 11000 + 9950 + 11572

17520 = 32522

17520 = 1.856279 EFSE

The unit used for λ in this thesis is equivalent full simulator evaluations (EFSE).

Precision of Estimates

Results of this section will consider how much work must be done to acquire an es-timate of mean annual constraint costs with an accuracy of at least 1%. Suppose c = (c₁, ..., c_n) represents a vector of n estimates of annual constraint costs utilising importance sampling via Equation 4.2.5. An accuracy of one percent will be defined as when the ratio of the standard error of the estimate of annual constraint costs to the mean of the estimate of annual constraint costs is 0.01 or less. That is, if µ_c = _n^1Pn_i=1c_i and s_c =

q 1 n−1

Pn

i=1(ci−µc)²

√n an accuracy of 1% requires _µ^sc

c to be 0.01 or less.

However, for a year 6 power system background it was noted that mean annual con-straint costs are £246,000,000. Therefore, an estimate where the standard error of the estimate is less than 1% of the mean of the estimate will expect to have a stan-dard error of £2,460,000, which is very large in real terms. Therefore, an alternative criterion will also be considered where the standard error of the estimate, s_c, must be less than £100,000. Whilst this may still seem quite high, £100,000 will later be defined in Section 5.1 as the equivalent cost of making a very small increase of 1 MW to transmission capacity [5, 78].

It is also important to recall that simulations of annual constraint costs acquired from importance sampling (and the average of many simulations) is still a random variable.

Therefore, the amount of work required to reach a given level of accuracy will not be perfectly consistent if one were to repeat the process. For example, it could be that one estimate of mean annual constraint costs with an accuracy of 1% is acquired after the equivalent work of 53.24 full simulator evaluations. However, if the process was repeated for the same power system background, an estimate with an accuracy of 1%

may take more or less work than the equivalent of 53.24 full simulator evaluations to acquire.

Therefore, this section will perform 200 separate repetitions of how much work is required for an accuracy of 1%. The mean amount of work required for an accuracy of 1% across these repetitions will be calculated to give an estimate of the expected work required to reach an estimate of mean annual constraint costs with an accuracy of 1%.

Boxplots of the work required for all 200 repetitions will also be given to illustrate how even for a fixed importance sampling technique there will be variation in the amount of work required to acquire an estimate of given precision.

Accuracy of Estimates

As well as considering the work required to acquire an estimate of mean annual con-straint costs, it is also important to give consideration to how the resulting estimates of mean annual constraint costs compare to the actual mean of the full simulator. This is considered in Section 4.3.6 with further details being given in Appendix A.2, which

4.3. Initial Application of Importance Sampling to Estimate Mean

Constraint Costs 75

compare the estimates that will be acquired in Sections 4.3.3 to 4.3.5 to highly accurate estimates of the simulator mean.

This is because whilst all estimates of mean annual constraint costs are unbiased (as Equation 4.2.6 verifies), it is possible that if too few simulator evaluations are used to acquire an estimate of constraint costs then the resulting estimate will typically differ quite substantially from the actual mean of the simulator. Therefore, the goal is not necessarily to minimise the expected work required to acquire an estimate of mean constraint costs, but to find a feasible balance between the work required to acquire an estimate of mean constraint costs and the accuracy one could expect in the resulting estimate.