Estimation Methods with Control Groups 26

Another way to estimate demand reductions is by using a control group of customers that is not curtailed during the event. In essence, the electricity demand patterns by the group that was not

curtailed are used to infer what the usage patterns of the curtailment group would have been absent the curtailment.

However, on its own, using a control group does not guarantee more accurate results. A good control group has customers that, on average, look like and behave in the same manner as the customers whose AC loads were controlled except for the curtailment event. To eliminate alternative explanations for differences in electricity use, it is critical that the only systematic difference between the two groups is the fact that one group had their AC units curtailed while the other group did not. To put it differently, if two groups behave almost exactly the same during all hours of the year except for the hours when AC units were curtailed, it is reasonable to conclude the difference in electricity demand is due to the curtailment event.

The best way to ensure there are no systematic differences between the two groups is to randomly assign customers to the curtailment and control groups and use large sample sizes. This approach is known as an experiment or a Randomized Control Trial. It is widely regarded as the best evaluation method. Because of random assignment, on average, both groups can be expected to have similar characteristics such as household size and to experience the same weather, economic conditions and occupancy patterns. That is, random assignment of AC load control events eliminates alternative explanations for changes in demand by eliminating

systematic differences between the two groups. The only systematic difference between the two groups is whether or not they were curtailed. However, differences between the two groups can arise due to random variation in sampling. With larger groups, it is less likely that substantive differences exist.

Using control groups with random assignment has several benefits. It ensures the demand reduction estimates are unbiased and precise, provided large samples are used. More over the process is extremely easy to understand and allows the use of calculation methods that are highly transparent and simple to execute. It does not need to rely on a complex mathematical model or rules for selecting match days. In our assessment, the demand reductions were calculated in one of two ways:

 A simple comparison of means: With this approach, for each time period, demand reductions are estimated as the difference between the group that did not have their AC loads curtailed and one that did.

 A weather-matched difference-in-differences calculation: This approach is useful when sample sizes are smaller. The demand reduction is calculated as the difference between the two groups, but then adjusted with one additional step. We subtract out differences between the two groups during days without curtailments and similar weather. This nets out differences that are irrelevant and mainly due to sampling variation.

Figure 3-4 illustrates an example of calculating AC demand reductions with smart meter data, random assignment to operations and large sample sizes. The example is based on the 2011 SmartAC evaluation, which was still underway when this report was being written. In the evaluation, customers were randomly assigned to 1 of 10 groups. During each of nine test events, one of the randomly assigned groups was operated while the other nine groups acted as a control group. During actual program events, 9 of the 10 randomly assigned groups would be

operated. For the example day, a randomly assigned group of 10% of the devices, or 14,000 devices, was activated for curtailment. The remaining 90% or 124,000 devices acted as the control group and provided information about normal electricity use without the curtailment.

Figure 3-4: Example of Using a Control Group to Estimate Demand Reductions

Figure 3-4 shows actual metered loads for the two groups without any adjustments. It illustrates the accuracy and simplicity of the approach. Loads leading up to the event were identical for both groups and clearly diverged during the curtailment period. The large sample sizes effectively eliminated almost all random sampling error, while the randomized assignment ensured both groups were identical except for the load curtailments.

Not all utilities are able to measure AC demand reductions using random assignment with groups of similar magnitude, for reasons discussed below. Even within a larger utility like PG&E, demand reduction measurements are sometimes required at a more localized level, reducing sample sizes. Smaller sample sizes introduce the potential for differences due to random

variation in the sampling – that is, it can introduce differences that are unrelated to curtailments. With smaller sample sizes a difference-in-differences calculation can remove these irrelevant discrepancies and improve both the accuracy of the results.

Figure 3-5 illustrates the calculation using AC end-use data. In the example, 600 units were sampled and 300 were randomly assigned to the curtailment group and 300 were assigned to act as a control group.

The two groups have a clear difference in electricity use during non-event days and in the hours leading up to the event in the curtailment day. This difference is entirely the result of random sampling and the smaller sample sizes. Taking the simple difference between the two groups during the curtailment period clearly overestimates the demand reductions. To correct this, the differences observed during the non-event days with similar weather are subtracted out. In the example, the true demand reduction is 0.25 kW per AC unit. Without the adjustment, using the control group produces demand reductions of 0.40 kW. With the adjustment, the demand reduction estimate is 0.30 kW, a clear improvement.

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 1 4 7 10 13 16 19 22 kW Hour Ending

 Large samples effectively eliminate sampling error

 Actual average loads for a treatment group and control group on an event day

Roughly 14,000 (10%) devices dispatched Roughly 124,000 (90%)

participant devices not dispatched

Figure 3-5: Example of Weather-matched Difference-in-Differences Calculation

For many AC load control programs, it is not always feasible to implement randomly assigned control groups with large sample sizes. The first consideration is load control device

communications. Newer AC load control programs such as PG&E's SmartAC typically use systems that can transmit cycling instructions to individual AC units. For example, it is possible to instruct the load control device of a house to shed load and to instruct the load control devices at an adjacent house not to do so. Older AC cycling programs such as SCE's often rely on one- way communications where all units in a region respond to a radio signal. This has practical implications. For example, for a program like PG&E's SmartAC, it is possible to randomly assign the participants into 10 groups and withhold a group of over 14,000 accounts from being dispatched during each event, rotating the control group. For a program without load control devices that can be directly addressed, this is not feasible. To create a control group, they would need to install “placebo” or inactive load control devices for a subset of households. The second consideration is costs. With smart meters in place, the costs of deploying and using large

samples with several thousand customers to estimate load reductions is dramatically lower. The data also can be retrieved remotely and analyzed within days. Without smart meters in place, there is no such luxury and sample sizes are a legitimate concern. The ability to use extremely large sample sizes and ensure accurate representation of the population of interest are two of the most attractive features of smart meter household data. Utilities that had not yet deployed smart meters would need to install data collection devices, adding a substantial cost per unit included in

0.0 0.2 0.4 0.6 0.8 1.0 1.2 0 3 6 9 12 15 18 21 24 Av g.   kW   pe r   AC   unit Hour Ending Curtailment Day Control group Curtailment Group Control minus non‐ event day differences

0.0 0.2 0.4 0.6 0.8 1.0 1.2 0 3 6 9 12 15 18 21 24 Av g.   kW   pe r   AC   uni t Hour Ending

Non Event Days with Similar Weather Control Group Curtailment Group Difference due to  random sampling  and smaller  samples Difference during  non‐event days is  subtracted out

the samples. As a result, they would likely need to rely on substantially smaller samples of either household AC end-uses or households.