Evaluation Lessons Learned: Challenges

While the pilots enjoyed many observed successes, a pilot project would not be a pilot, by definition, without the opportunity to learn lessons from design and delivery choices. DNV KEMA uses the following section to summarize challenges we observed within the areas of pilot participation, customer energy use, customer attitudes and behavior (including program marketing), and project processes and evaluation.

Participation

Saving money was the most important reason for participating in the pilots. Throughout the pilot, DNV KEMA repeatedly heard that saving money (or the promise of saving money) was the most important motivator for participants (or potential participants). In the follow up surveys, 60 percent of Plover, 37 percent of Allouez, and 36 percent of Brillion cited saving money as the most important reason they participated in the pilots. The survey asked 185 households in all three pilot communities that did not participate in time of use rates how much savings they would need to expect to justify switching to time of use rates. Over two-thirds (69%) cited $25 or more per month. Another 23 percent cited between $15 and $20, and another five percent cited a number between $5 and $10.

the 31 percent who did provide a reason, about one-fourth (26%) said they did not want to lose control of their home’s energy use. Another 26 percent said they did not think participation in the rates would save enough money.

Home owners were more likely to participate in the pilots than renters. Eighty-five percent of the pilot participants reported living in a single-family detached home compared to 75 percent of the non-participants. Over 95 percent of the respondents living in single-family detached homes said they owned rather than rented. Renters have little motivation (and often lack the authority) to spend a lot of money on durable energy saving home improvements. As explained in the Databases subsection later, it is also difficult to accurately measure energy savings impacts for renters.

One out of the three communities participated enough to attain their full community involvement goal. Plover was the only community to achieve its full community participation target, due in large part to the success of the default rate assignment process. The other communities received pro-rated

community rewards based on how close they got to the goal. Pilot implementers used the community participation goals only to provide a target for achieving the community participation award. Because of the novelty of the pilots, program implementers had to set these goals without enough information to make confident predictions about eventual participation. Thus, they had to make an educated guess at a participation level that would represent a challenging target.

Energy Use

The final billing analysis concluded that residential participants reduced their energy use by an average of about three percent. This was the average savings of households across all three

communities that participated in one or more of the following offer categories: Focus on Energy, Time-of-Use rates, Cool Rewards, or one of the Tools and Technology offers. There were too few commercial participants to make robust savings estimates.

When combined with energy efficiency measures (EEM), participation in other offers only resulted in an additive savings effect. Except for the combination of time of use rates and tools and technology offers discussed earlier, the savings from participating in multiple offers was about equal to the sum of the savings for participating in each of the offers individually.

DNV KEMA found little evidence that pilot community residents changed the time of day they used electricity. Several of the pilot electricity rates involved the use of price signals to encourage participants to shift their electricity use to different times of day. Using hourly billing data, DNV KEMA found evidence that Allouez residents shifted some electricity use off of about one percent of winter peak hours.

We did not find any evidence of Brillion or Plover residents shifting electricity use off of summer, winter, or shoulder-season peak hours.

Despite increasing Focus on Energy participation, residential audits did not appear to directly reduce energy use. As mentioned in the successes section, audit participation increased participation in Focus on Energy. DNV KEMA also conducted a billing analysis that compared the audit households (regardless of Focus participation) with those in the pilot communities that did not receive audits and

those in the control communities. This billing analysis did not show statistically significant energy savings for the audit participants.

The combination of no tangible energy savings from audits themselves, but increased participation in Focus from audit participants, is somewhat contradictory. A possible explanation is that about half of the audit participants installed Focus measures, but half did not participate beyond the audit. The billing analysis averaged together these two categories, which had the effect of reducing the average savings per audit household. Similarly, some of the non-audit and control households installed Focus measures, which the billing analysis would have also averaged into that category. This would effectively raise the per household savings for the comparison groups. The combination of these two effects may have reduced the difference enough that it was no longer statistically significant. Other possible explanations include:

weather leveled out the differences, or the increased Focus participation inspired by the audits did not actually result in energy savings.

Process

The number of stakeholders created project management challenges. DNV KEMA repeatedly heard from interviews with program implementers that the number of organizations involved in delivery of the pilots created communication challenges. These interviewees asserted the value of clearly defining inter-organization responsibilities and communication protocols. WPS’s final report includes a project management conclusion that it is essential to define, document, and communicate roles and responsibilities in a timely manner across all the organizations involved in pilot delivery.

Focus on Energy rebate participants voiced a preference to work with a local contractor with whom they already had a relationship, and were unsatisfied if their contractor was not Focus on Energy-approved. The program may have been able to engage more local contractors by offering multiple opportunities to enter the program, reaching out personally to the contractors to invite their participation, and/or explaining contractor benefits through participation more clearly.

The speed of rolling out Allouez and especially Plover made it difficult to apply lessons learned from the previous pilot community. More time between rollouts would have provided the

implementation teams with the chance to make more effective changes before the next pilot rollout. This might also have simplified the communication and evaluation efforts.

Two years may be the optimum pilot duration. We received feedback – either directly from customers or via the participation data – that the various pilot durations in the communities may have not always been ideal. In Brillion, where the pilots ran for three years, some residents expressed “marketing fatigue”

where community members ignored marketing materials at best, or were even annoyed by them. On the

interest. There was another instance where demand exceeded auditor availability for a couple of months.

Deadlines and offer endings are another time when program implementers should plan for a surge in customer interest.

Not all pilot customers had equal opportunity to access all marketing channels and receive all possible information. Customers lacking high-speed Internet access did not necessarily have easy and ready access to the iCanConserve microsites, built to support and provide key education about the pilot program and its offerings to customers. In particular, we found evidence that Brillion residents had lesser internet access than those in other pilot communities and the average WPS customer. Sixty percent of Brillion baseline survey respondents said they had high-speed internet access, compared to 73 percent of Allouez, 79 percent of Plover, and 77 percent of WPS customers in general².

The iCanConserve pilot had several initial program offerings that did not generate customer response or excitement. Key stakeholders, Energy Advocates, and our evaluation findings noted that iCanConserve customers did not choose to participate in two initial pilot offerings – Community Supported Financing and the Business Staffing Grant. The redesign or reallocation of the funding dedicated to these programs was listed earlier as an example of successful flexibility on the part of program implementers.

Databases

Pilot outcomes and conclusions are only as reliable as the data inputs. Databases were set up to record point-in-time information rather than allow a retrospective view of the actions of pilot participants.

In order to accurately track customer participation and assess energy savings impacts, creating solid, consistent database infrastructures in advance of a pilot launch is paramount. Data choices made before the iCanConserve launch (i.e., utilizing pre-existing databases not optimized for tracking specific pilot activity or energy efficiency actions) and migration to a new database mid-pilot made it difficult to track and compare the actions of participants in the pilot and control communities. Some specific issues included: lack of consistent identification numbers across the four residential and three commercial databases, lack of measure codes for the pilot-specific measures that were distinct from general Focus on Energy measures, and databases contained records for different numbers of customers or premises depending on when they were queried.

DNV KEMA’s decision to evaluate the program at the customer-at-a-premise level resulted in several challenges. DNV KEMA considered the same premise with two different customers to be two different units of analysis. Furthermore, because DNV KEMA only evaluated customer-premise combinations that existed from the beginning to the end of the pilot in each community, any changes to the pair during the pilots resulted in the loss of a record. The decision to analyze at the paired level seemed the most logical at the beginning of the pilots because the pilots were interested in attitude and behavioral changes (customer-level variables) as well as energy efficiency equipment installations (a

2 Based on the 2010 WPS territory wide survey.

premise-level variable). This decision did result in several challenges, including handling customers with multiple premises (such as landlords) and premises where the customer changed. In the latter case, these changes could have resulted from a family moving out of the premise or from divorces, legal name changes, or other situations that might not represent a whole family moving. However, it was beyond the scope of the evaluation to determine the precise reason for a change of customer associated with a premise. About one-fourth (24%) of the customer-premise combinations that existed at the beginning of the pilots did not exist at the end, and DNV KEMA was unable to analyze these records.