Evaluation Recommendations

Examining the key pilot successes and challenges of the iCanConserve pilot provides us the opportunity to document recommendations about what should be considered or implemented in future pilots or when attempting to transfer pilot activities to a broader territory. DNV KEMA uses this report section to offer our recommendations to optimize energy efficiency pilot planning and programming, based on lessons we learned during this program evaluation.

Participation

Provide high incentives coupled with a deadline for participation. The restructuring of the

commercial rebate programs in early 2012 resulted in a significant increase in commercial participation, especially just before incentive deadlines. Program staff attributed the increased participation to high incentives and the deadlines themselves. The pilots experienced another surge in participation right before the final deadline that signaled the end of the pilots.

Use default rate assignment processes to increase rate adoption. The default rate assignment process resulted in over ten times the participation levels as the opt in process. We found evidence of reduced customer satisfaction immediately following the default rate assignment. Very few of the customers reached immediately following the rate assignment provided a specific reason for their satisfaction rating.

However, customer satisfaction reported in the follow-up surveys was about the same as was reported in the baseline surveys. This suggests that if satisfaction did decrease after the default rate assignment process, it rebounded within a year.

Minimize the amount of information potential participants need to process, and wherever possible, customize the information and include payback periods. Some participants (residential and

commercial) expressed being overwhelmed by the information in the audit reports. Organize reports so that they have a high-level summary with additional details if the recipient wants to drill down. In addition, some participants expressed frustration that the information in the reports was not applicable to their particular situation.

Brillion audits also required participants to implement all of the audit recommendations in order to receive a fully discounted audit. This created another barrier because the sum of the recommendations represented overwhelming capital outlays for some people. The best structure attempted during the pilot projects was a single, full audit for a nominal fee, coupled with direct install measures and a high bonus rebate for implementing the top three audit recommendations.

Use demographic information within customer data to predict participation and target marketing for future energy efficiency programming. Certain demographic characteristics (single-family home, central air conditioner, previous energy efficient actions, and community involvement) are better predictors of program participation than other demographics that are easier to come by (income, education, age).

Energy Use

Residential customers participating in voluntary critical peak pricing rates could benefit from technology that allows them to (a) receive notifications of the events via cellphone or text message, and (b) control their home remotely over the Internet. About half of the Response Rewards customers interviewed by DNV KEMA reported doing nothing to change electricity use during the critical periods.

Despite having the option to select their preferred method of notification from phone, text, email, or fax, many said they were not aware of the events until after the fact. In other cases, even when the respondent received timely notification of the event, they could not do anything about it because they were not home and lacked technology to remotely control their home. Devices that can receive the events and automate control, such as at least one model of smart thermostat available through the pilots, could be helpful as well.

Attitude / Behavior Changes

Secure buy-in from key community leaders and enlist their help to increase the likelihood of pilot success. The pilot communities experienced various levels of community engagement which reflected engagement levels of local community leaders. Program implementers reported to DNV KEMA that the pilot was easier to implement in communities where the pilot had greater support from community leaders. Community leaders may or may not be elected officials. Examples of the types of community leaders important to the iCanConserve pilots included a mayor, a village administrator, and a parks department head.

Increase opportunities for face-to-face contact. Throughout the pilots, DNV KEMA heard positive comments from residents and businesses in the pilot communities and pilot implementers about the Energy Advocates who performed the audits. Similarly, DNV KEMA heard similar customer satisfaction sentiments based on personal communication experiences within studies on the Business Staffing grant and the School-to-Home grant, respectively.

Process

Consider pilot launch timing carefully. Defaulting customers to time-of-use rates during times of increased energy usage (i.e. the middle of summer) can adversely affect customer adoption or satisfaction.

Also, be sure to allow adequate time between pilot launches to give implementers a chance to fully integrate previous lessons learned.

Design tracking databases to capture pilot participation and activity exclusively. In order to accurately track customer participation and assess energy savings impacts, creating a solid database infrastructure in advance of a pilot launch is paramount. Unique pilot databases should also include thoughtful, consistent identifiers that allow easy linking between pilot participant data and other data used for analysis (i.e. billing / metering data). Databases should track enough information to provide a clear retrospective of when and what each relevant household did throughout the pilot.

Produce and maintain detailed and accurate documentation for all major decisions. Many decisions will be made during the course of a pilot project or program. The longer a program or project endures, and the more organizations involved, the likelihood increases that natural personnel changes will remove key knowledge-holders from the project. Timely and clear documentation of all major program decisions will help ensure the maintenance of institutional knowledge.

Simplify staff communications and pilot program processes as much as possible. Pilot programs, by nature, explore new territory. Keeping the pilot communications between involved staff and delivery components as simple as possible keeps the focus on the pilot delivery, its customers, the results.

Plan for surges in customer demand. The pilot repeatedly enjoyed rapid response at the end of the program or at the end of an offer (i.e., Focus on Energy incentives) – especially when they informed customers that the end was near. The increase in customer interest is predictable; prepare and/or increase staffing to handle program- or offer-end surges to avoid customer disappointment.

Build in feedback mechanisms and the flexibility to change in response. The ability and willingness of pilot implementers to adapt to early results and feedback improved the pilot outcomes. Several of the iCanConserve pilot components were redesigned while the pilot was ongoing based on low participation and / or customer feedback.

Utilize a variety of marketing channels to maximize the number of customers reached. WPS had customers with a wide variety of technology access and internet savvy. Pilot marketing occasionally had to rely on lower-tech solutions to reach these individuals.

Make efforts to include local contractors. Enlisting the help of contractors from the pilot area, if possible, increases the feeling of community and feeds a local economy. Use of local contractors likely also increases scheduling options for participants.

impacts of individual components difficult. When multiple offers are packaged, it is usually impossible to attribute effects to single offers within packages - the best that evaluation can say is whether the package as a whole had effects. Pilots with fewer offers would facilitate more specific evaluations.