Comparison Group Designs — the Alternative to RCTs

Evaluations of energy programs typically use alternative evaluation strategies. The most common alternative to RCTs is the quasi-experimental method that involves a comparison group rather than a control group. The subjects in a

comparison group are not randomly assigned to a treatment condition, as are

participants in a control group. In other words, the treatment group is self- selected by characteristics that differentiate them from others in a population.

An example of a quasi-experimental design is the evaluation of the FEMP, using a customer survey (Hall et. al., 2001a; 2001b; 2002). In one part of the study, Energy Service Performance Contracting (ESPC) participants and non- participants (people with similar job titles) eligible to participate in ESPC, were assigned to a stage of diffusion (see Chapters 2 and 3) before and after participa- tion (or before and after an equivalent time period for non-participants) based on their responses to a series of questions in a survey. Table 4 shows the results. The percent of participants in the unaware, aware, and persuasion stages, declined by as much as 24 percent from before to after program participation (column 3). The percent of participants in the decision, implementation, and confirmation stages increased by an average of 16 percent. For non-participants, there was a substantial decline in those who were unaware (i.e., awareness increased), and modest increases among non-participants in the other stages. The bottom line is that participants moved rapidly in the direction of implementing and confirming behaviors following participation in the training program, while non-participants showed only small amounts of movement.

The alternative to RCTs that is most com- monly used in energy research is the comparison group design.

Table 4. Movement of FEMP ESPC Participants and Non-Participants through the Adoption Cycle

Percent of participants (N=101) Percent of aware non- True non-

participants (N=117) participants

(1) (2) (3) (4) (5) (6) (7)

Diffusion stage Before Since Change Before Since Change Percent of

hearing involvement from before hearing involve- from unaware non-

about with FEMP to after about ment with before to participants

FEMP FEMP FEMP after

Unaware 24 0 -24 21 0 -21 63 Aware 27 10 -17 31 40 +9 24 Persuasion 12 7 -5 9 10 +1 5 Decision - no 10 7 -3 14 16 +2 1 Decision - yes 3 21 +18 8 9 +1 1 Implementation 7 24 +17 4 6 +2 4 Confirmation 18 32 +14 14 18 +4 2

The big catch with comparison group designs is to identify the population from which the treatment group comes and then to draw a sample from it.

Source: Hall, et. al., 2002.

There are various strategies for selecting comparison groups. One method is to randomly select subjects for the comparison group from the same population as the treatment population. The catch phrase is “from the same population.” Suppose that the goal is to create a comparison group for a weatherization assis- tance study. Applicants for weatherization assistance are from the low-income population, but they are not representative of that population. They represent a self-selected group since they volunteer to participate in the program, the charac- teristics of which may not be apparent. They are typically, but not always, homeowners. They may be having problems with their utility bills. They have the contacts, know how, and willingness to reach out, or they have contacts that help them make connections with a Community Action Agency delivering weatherization services. These characteristics only define some low-income households, so sampling from all low-income households is not sampling from an equivalent population.

An example of this problem comes from a study of payment behaviors for a Maryland electric utility bill payment program (Lee, et. al., 2007). The program was managed by the Maryland Public Utilities Commission (PUC) and adminis- tered by an agency inside the Department of Human Services. The sample of

participants was not a problem, because the study was ordered by the PUC. The difficulty was constructing the comparison group. An obvious possibility was to use households applying for other types of social services. However, because of confidentiality rules, information about persons not in the electric bill payment program could not be obtained. The alternative used was to randomly select households from the same nine digit zip codes from which the participants came. This resulted in a non-participant sample whose energy consumption was nearly identical to that of the participants but who were determined, on the basis of a survey, to be different in important characteristics such as home ownership, length of time in residence, etc. Although weighting was not used because of the small sample sizes, the households in the non-participant sample could have been weighted to more closely reflect the characteristics of the participant population to make the two groups more comparable.

Thus, it is difficult to define an equivalent population. Also, once the equiva- lent population has been defined, one has to obtain a list of members of the population to draw a sample. The zip code strategy presented a good strategy, because near neighbors could be identified on the basis of address and data anonymously obtained from billing records.

Some evaluations of weatherization programs have constructed comparison groups (called lagged comparison groups) by using later sign ups to form a comparison group for earlier sign ups (Figure 16) (Hall 1998; Reed 1999b; 1997). This should help with equivalence problems, but there is no guarantee that the characteristics of participants who signed up for the program two years previously are similar to households who sign up two years later. This might be particularly true if the program has matured and is more effective in reaching a target audi- ence. The evaluator needs to determine the equivalence of the two groups on key characteristics.

There are many variations in developing a comparison sample. One approach is a sample matched on key characteristics. For the weatherization example, a

One way to skirt the equivalence problem is to draw samples of later participants and use them as a compari- son group for earlier participants. Random selection of non- participants followed by careful matching on key variables is another solution to the comparison

The cases in a random sample of participants for a comparison group can be weighted on key characteristics to reflect the treatment group.

the characteristics of households in the treatment group by income, age, number of householders, structure of the household, historical energy use, etc. A frequent difficulty with this strategy is that there are not enough data to do the matching a priori.

Another approach is weighting. The characteristics of the comparison group can be compared with the characteristics of the treatment group. Then, the distribution of individual characteristics of members of the comparison group can be weighted to match the distribution in the treatment group. To be done well, this requires substantial skill and some luck, as well as sufficiently sized samples.

Under certain circumstances, well matched comparison group studies can approach the rigor of RCTs. However, the use of comparison group studies does increase the risk of misleading results because of the difficulty in eliminating bias in the selection of the comparison group. For example, differences between the participant groups and comparison groups that affect their changes in energy use between the pre- and post-participation periods will bias the estimate of net

program savings (selection bias). Awareness of this risk is crucial to the design of such evaluations.