The Use of Importance Sampling to Estimate Mean Constraint Costs
10.13 EFSE 293.8 EFSE Snapshot Mean of Alternative Power System
4.3.7 Why Importance Sampling Has Relatively Little Benefit for a Year 6 Power System Backgroundfor a Year 6 Power System Background
In Sections 4.3.3 and 4.3.5 it was shown that when estimating mean annual constraint costs for a year 1 power system background to a given level of precision, basing the
4.3. Initial Application of Importance Sampling to Estimate Mean
Constraint Costs 95
importance sampling weights on the mean constraint costs in each snapshots can sub-stantially reduce the amount of work required to reach a given level of precision in the estimate. However, it was also shown that for a year 6 power system background there was relatively little benefit from an application of importance sampling in comparison to using full simulator evaluations, with the best results being a 4.8% reduction in EFSE when basing importance sampling weights on the mean constraint costs of each snapshot.
As Section 4.2.2 and Appendix A.1 showed that basing importance sampling weights on the mean constraint costs of each snapshot minimises the variance of the resulting esti-mate (when evaluating an equivalent number of snapshots) it may have been expected that a greater reduction in the expected work required would have been observed for a year 6 power system background.
However, it is important to consider that just because a certain selection of weights minimises the variance of an estimate, it does not necessarily mean that the variance of the estimate is substantially reduced in comparison to simply evaluating every snap-shot. To illustrate this, 1000 further full simulator evaluations (i.e. evaluating each snapshot) were taken. Then, these simulations were used to acquire an estimate of how the variance varies depending on whether or not importance sampling was used.
In Section 4.2.2 it was noted that applying importance sampling using Equation 4.2.5 results in an increase in the variance of evaluations of annual constraint costs for all choices of importance sampling weights, ω, as the work required to acquire an estimate of annual constraint costs is not accounted for and instead the alternative importance sampling methodology detailed in Section 4.2.1 must be used to compare variance esti-mates when an equivalent amount of work (number of snapshots) have been evaluated.
Using the alternative importance sampling method outlined in Section 4.2.1, Ap-pendix A.1 states that the variance of a particular choice of importance sampling weights can be calculated as
σ2q(τ )= Eq(τ )
fc,T(τ )p(τ ) q(τ ) − µ
!2
(4.3.2)
where fc,T(τ ) was defined in Equation 4.2.7 as an estimate of annual constraint costs based on a single simulation of snapshot τ ; p(τ ) is the probability mass function used
when treating the snapshot, τ , as if it were a random variable (defined in Section 4.2.2 as 175201 for all snapshots) and q(τ ) is the alternative density used to sample values of τ to simulate, with Equation 4.2.9 defining an equivalent q(τ ) for given values of ωτ as
q(τ ) = ωτ
P
τωτ
The variance detailed in Equation 4.3.2 can be estimated using the additional 1000 full simulations as
ˆ
s2q(τ ) = 1 1000
1000
X
i=1 17520
X
τ =1
q(τ ) fc,T ,i(τ )p(τ ) q(τ ) − ˆµ
!2
(4.3.3) where fc,T ,i(τ ) is the ith simulated value of fc,T(τ ) for snapshot τ and ˆµ is the mean value of fc,T ,i(τ ) across all 17520 snapshots and 1000 repetitions.
Year 6 Results Year 1 Results Estimate of variance when evaluating each
snapshot (call this ˆs2p(τ ))
6.37×1017 1.31×1017 Estimate of variance when using importance
sampling with weights based on snapshot mean (call this ˆs2q(τ ))
6.02×1017 8.27×1015
ˆ s2q(τ ) ˆ
s2p(τ ) 0.945 0.063
Table 4.10: Table detailing how variance estimates vary depending whether or not importance sampling is used.
Table 4.10 details how the estimate of the variance from Equation 4.3.3 varies depend-ing on whether or not importance sampldepend-ing is used (with importance sample weights based on snapshot mean constraint costs). As can be seen, for a year 1 power system background the variance of the estimate is reduced by 93.7% when using importance sampling with weights based on the mean constraint costs of each snapshot in com-parison to not using importance sampling, which is in line with the reductions of over 93% in the amount of work required to estimate to a given level of precision in Ta-bles 4.3 and 4.6. However, for a year 6 power system background, the variance estimate is only reduced by around 5.5% when using importance sampling, which is also in line with the reductions in expected work required to acquire an estimate to a given level of precision previously observed.
Further evidence of this can be seen by considering Figure 4.10 (a), which illustrates
4.3. Initial Application of Importance Sampling to Estimate Mean
1 1948 5841 9734 13627 17520
0.0e+004.0e+068.0e+061.2e+07
Snapshot Mean Rank
Simulated Constraint Costs (£)
(a) Simulated constraint costs.
●
1 1948 5841 9734 13627 17520
0.0e+001.0e+072.0e+073.0e+07
Snapshot Mean Rank
Weighted Simulated Constraint Costs (£)
(b) Weighted simulated constraint costs.
Figure 4.10: Boxplots to illustrate the variation in simulated constraint costs for 1000 simulations for 10 snapshots for a year 1 power system background.
boxplots for 1000 simulator evaluations of constraint costs for 10 snapshots for a year 1 power system background. The snapshots chosen for illustration in Figure 4.10 (a) are 10 equally spaced snapshots when snapshots are ranked from the lowest mean of constraint costs to the greatest mean of constraint costs.
As can be seen, the snapshot with the greatest mean of constraint costs dominates the graph, with the costs in the other 9 snapshots being negligible in comparison. In-tuitively, it would therefore be expected that this snapshot is most important to the estimate of mean annual constraint costs, and when using importance sampling this snapshot should be sampled much more frequently than others. This is reflected in Figure 4.11, which illustrates how importance sampling weights, ω, vary from small-est weight to largsmall-est for a year 1 power system background when basing importance sampling weights on the mean constraint costs of each snapshot. As can be seen, the vast majority of snapshots have very little weight due to the mean of constraint costs in these snapshots being very low, with only 520 (3.02%) of the snapshots having a weight greater than 0.01.
Figure 4.10 (b) illustrates boxplots for the 1000 simulations of each of the snapshots of Figure 4.10 (a) when each of the simulations of constraint costs has been weighted using Equation 4.2.5 (i.e. the value that would be used when estimating constraint costs if
0 5000 10000 15000
0.00.20.40.60.81.0
Snapshot
Snapshot Weight
Year 1 Power System Background Year 6 Power System Background
Figure 4.11: Plot to compare importance sampling weights for a year 1 and year 6 power system background, when basing the weights on the mean constraint costs of each snapshot.
that particular snapshot was simulated when using importance sampling). As can be seen, the plots for each snapshot are now much more similar to one another, with no particular snapshot dominating. This illustrates the benefit of importance sampling for a year 1 power system background, as snapshots less relevant to the estimate of annual constraint costs are sampled much less frequently, with the weighted estimates of constraint costs from Equation 4.2.5 accounting for this when the snapshot is actually simulated.
Figure 4.12 displays the equivalent graphs to Figure 4.10 for a year 6 power system back-ground (i.e. boxplots for 1000 simulator evaluations of constraint costs for 10 equally spaced snapshots when snapshots are ranked from the snapshot with the lowest mean of constraint costs to the snapshot with the greatest mean of constraint costs). As can be seen in Figure 4.12 (a), even without an application of importance sampling the distribution of constraint costs across all snapshots is somewhat similar, with the con-straint costs from the snapshot with the greatest mean not dominating simulated costs from the other snapshots, unlike what was seen for a year 1 power system background.
Further, the weighted estimates of constraint costs from these simulations displayed in Figure 4.12 (b) do not appear to be a great improvement on Figure 4.12 (a). In particular, it can be seen that for the snapshot with the lowest mean of constraint costs, due to the small weight of this snapshot (0.069) several simulator evaluations
4.3. Initial Application of Importance Sampling to Estimate Mean
1 1948 5841 9734 13627 17520
0e+001e+052e+053e+054e+05
Snapshot Mean Rank
Simulated Constraint Costs (£)
(a) Simulated constraint costs.
●
1 1948 5841 9734 13627 17520
050000015000002500000
Snapshot Mean Rank
Weighted Simulated Constraint Costs (£)
(b) Weighted simulated constraint costs.
Figure 4.12: Boxplots to illustrate the variation in simulated constraint costs for 1000 simulations for 10 snapshots for a year 6 power system background.
which gave a larger than average evaluation of constraint costs for this snapshot result in very large weighted estimates of constraint costs.
Further evidence to this can be seen in Figure 4.11, which also displays how importance sampling weights, ω, vary from smallest weight to largest for a year 6 power system background when basing importance sampling weights on the mean constraint costs of each snapshot. For a year 6 power system background it can be seen that whilst only a small amount of snapshots are given weight greater than 0.8 or 0.7 (1.0% and 3.9% of snapshots respectively) the weights are much more evenly distributed amongst snapshots in comparison to a year 1 power system background. In particular, a yearly simulation of constraint costs which uses importance sampling would expect to evaluate just 74.68 snapshots for a year 1 power system background (0.43% of the year) in comparison to the 5220 that would be expected to be evaluated for a year 6 power system background (29.80% of the year).