Expected Cost Minimization

The results obtained from executing the MRP on our GEP-EC optimization model are shown in Table 2-9. For each MRP parameterization, the table shows results for both a

single, arbitrary MRP trial and aggregated statistics taken over all five MRP trials. Units for entries associated with and the 95% confidence interval (CI) are given in billions USD.

Table 2- 9 Results of Applying the Multiple Replication Procedure (MRP) to the GEP- EC optimization model. The MRP Input Parameters are Reported in the Columns Labels , , and n. Outputs are Reported for the MRP as well as Summary Statistics

taken over Five MRP Trials to Verify the Stability of the Procedure.

n MRP results Verification: five MRP trials 95% CI std. dev. 95% CI std. dev. 70 2 465 17.362 0.021 0.42 0.22 70 5 186 17.362 0.055 0.42 0.19 70 10 93 17.362 0.051 0.42 0.20 70 20 46 17.362 0.124 0.42 0.19 70 40 23 17.362 0.290 0.42 0.19 140 2 430 17.214 0.026 0.34 0.05 140 5 172 17.214 0.070 0.34 0.04 140 10 86 17.214 0.074 0.34 0.03 140 20 43 17.214 0.101 0.34 0.02 140 40 21 17.214 0.241 0.34 0.03 280 2 360 17.151 0.053 0.39 0.03 280 5 144 17.151 0.074 0.39 0.03 280 10 72 17.151 0.101 0.39 0.03 280 20 36 17.151 0.241 0.39 0.02 280 40 18 17.151 0.452 0.39 0.03 420 2 290 17.377 0.017 0.24 0.04 420 5 116 17.377 0.039 0.24 0.05 420 10 58 17.377 0.110 0.24 0.03 420 20 29 17.377 0.224 0.24 0.03 420 40 14 17.377 0.522 0.24 0.02 560 2 220 17.377 0.020 0.14 0.04

Table 2- 9 (continued)

560 5 88 17.377 0.063 0.14 0.04 560 10 44 17.377 0.103 0.14 0.05 560 20 22 17.377 0.198 0.14 0.07 560 40 11 17.377 0.571 0.14 0.07

We begin by considering the results obtained for one replication of the MRP, whose outputs are reported in the columns labeled “ ”and “95% CI”. Recall that we use (for a given MRP trial) a fixed random seed across experiments involving different combinations of and . This seed is used to randomize the list of scenarios, which are then sequentially partitioned into the baseline set containing scenarios and the validation groups. Consequently, the value of is identical in all trials in which only is varied. We first observe that the cost of the baseline solution is remarkably stable as is varied. This is consistent with the experiments reported in Section 2.7.2. Further, we would expect (and indeed observe) more stability in Table 2-9 because scenarios are accumulated as is increased—due to the use of a common random number seed across the individual MRP trials, and the sequential partitioning of the scenarios.

Next, we analyze the widths of the 95% confidence interval on the cost as and are varied, considering a single MRP trial. The largest CI width obtained is approximately 571 million USD, representing less than 3.5% of corresponding total cost of 17.377 billion USD. These results strongly suggest that accurate estimates of the optimal cost to our GEP- EC optimization model can be obtained using a remarkably small number of scenarios— especially relative to the full scenario tree. Unexpectedly, we observe that CI widths for a

fixed are generally increasing in ; the tightest confidence intervals appear at = 2, despite the trivial number of degrees of freedom in the resulting t test. This behavior is partially explained by the fact that for even modest , the number of samples in each group is too small to observe stability in . However, this explanation fails to account for the fact that the pattern holds even when the baseline solution is obtained with = 70. Fundamentally, the increase in the number of degrees of freedom in the t distribution is not offset by the stability in the total costs obtained from the validation scenario groups. Recall that the CI width is proportional to both the standard deviation of the sample optimality gap and the value of the t -statistic, and is inversely proportional to the square root of the number of scenario groups. The CI width monotonically increases as a function of for the GEP- EC optimization model (this behavior is not universal, but depends on both the specific optimization problem under consideration and the particulars of the MRP parameterization) because the sample optimality gap variance overcomes the benefit of increased and the number of degrees of freedom in the t distribution. Given a fixed , CI widths are reasonably consistent across different . Remarkably, the lowest CI widths obtained represent less than 0.5% of .

Finally, we consider the variability over multiple trials of MRP results, given fixed and . Such variability is generated by varying the random seed used to order the scenarios prior to partitioning them into the baseline and validation groups. The standard deviations of reported in Table 2-9 indicate that the stability of the MRP increases—as expected— with growth in ; taking a larger proportion of the N = 1000 scenarios in Ω will necessarily

decrease variability. However, even with modest , the standard deviations are comparatively small, indicating less than 5% deviation from the single-trial MRP result. The variability in CI widths obtained by the MRP also rapidly decreases as is increased, roughly leveling off once . Fundamentally, the five-trial MRP results indicate that the most accurate results are obtained using a large number of scenarios to form the baseline solution , yielding minimal negligible variability in both and CI width.

Overall, the results presented in Table 2-9 illustrate that very high-quality estimates of the minimal total cost for the GEP-EC optimization model can be obtained by using surprisingly small numbers of scenarios. Specifically, the computed CI widths range from less than 0.2% to roughly 3.5% of the baseline solution cost . Given the nature of long- term planning models—particularly the modeling assumptions employed and the range of uncertainties not explicitly considered—such tight confidence intervals strongly suggest that further efforts should be focused on improving the optimization model and uncertainty characterization, rather than using larger scenario samples in the existing GEP-EC model.

Table 2- 10 Results of Applying the Multiple Replication Procedure (MRP) to the GEP- CVAR Optimization Model. The MRP Input Parameters are Reported in the Columns Labels , , and n. Outputs are Reported for the MRP as well as Summary Statistics

taken over Five MRP Trials to Verify the Stability of the Procedure.

n MRP Verification: five MRP trials 95% CI std. dev. 95% CI std. dev. 70 2 465 23.341 0.265 0.70 1.97 70 5 186 23.341 0.365 0.70 1.78 70 10 93 23.341 0.509 0.70 1.93 70 20 46 23.341 0.784 0.70 1.93 70 40 23 23.341 1.260 0.70 1.98

Table 2- 10 (continued) 140 2 430 23.575 0.113 0.55 1.36 140 5 172 23.575 0.199 0.55 1.19 140 10 86 23.575 0.349 0.55 1.15 140 20 43 23.575 0.635 0.55 1.12 140 40 21 23.575 1.240 0.55 1.26 280 2 360 23.534 0.338 0.50 0.25 280 5 144 23.534 0.454 0.50 0.22 280 10 72 23.534 0.557 0.50 0.17 280 20 36 23.534 0.975 0.50 0.19 280 40 18 23.534 1.588 0.50 0.21 420 2 290 23.630 0.487 0.38 0.29 420 5 116 23.630 0.461 0.38 0.28 420 10 58 23.630 0.755 0.38 0.18 420 20 29 23.630 1.113 0.38 0.23 420 40 14 23.630 1.817 0.38 0.18 560 2 220 23.907 0.150 0.13 0.26 560 5 88 23.907 0.282 0.13 0.15 560 10 44 23.907 0.548 0.13 0.16 560 20 22 23.907 0.825 0.13 0.18 560 40 11 23.907 1.551 0.13 0.19