Customer-Premise Pairing

DNV KEMA considered the same premise with two different customers to be two different units of analysis. Furthermore, because DNV KEMA only evaluated customer-premise combinations that existed from the beginning to the end of the pilot in each community, any changes to the pair during the pilots resulted in the loss of a record. The decision to analyze at the paired level seemed the most logical at the beginning of the pilots because the pilots were interested in attitude and behavioral changes (customer-level variables) as well as energy efficiency equipment installations (a premise-(customer-level variable). This decision did result in several challenges, including handling customers with multiple premises (such as landlords) and premises where the customer changed. In the latter case, these changes could have resulted from a family moving out of the premise or from divorces, legal name changes, or other situations that might not represent a whole family moving. However, it was beyond the scope of the evaluation to determine the precise reason for a change of customer associated with a premise. About one-fourth (24%) of the customer-premise combinations that existed at the beginning of the pilots did not exist at the end, and DNV KEMA was unable to analyze these records.

quarterly deliveries of these data. Rate changes and tools and technology participants were tracked by WPS and reported to the evaluation via monthly iCanConserve eligibility reports.

The primary tracking of participation in the iCanConserve programs was done using databases designed for other uses. Pilot rebate and audit participation was tracked in Focus on Energy databases, rates were tracked using WPS’s billing system and tools and technology participation was tracked in dedicated Excel spreadsheets. During the three years of pilot program activity, the Focus on Energy program changed both implementers and tracking systems. Table 6-2 shows the different areas of pilot participation and the tracking systems used to record participation.

Table 6-2: Program Participation Tracking Systems

Sector Type of

Participation Tracking System Used Dates

Residential Rebates and Audits WECC Access

database Pre-program – 3/31/2012

Residential Rebates and Audits SPECTRUM database 4/1/2012 – 12/31/2012

Residential Rates WPS Billing system All

Residential Tools and

Technology Eligibility Report All

Commercial Rebates and Audits WISEERTS database Pre-program – 12/31/2011

Commercial Rebates and Audits SPECTRUM database 1/1/2012 – 12/31/2012

Evaluation activities required identifying which eligible WPS customers participated in the Pilot or other programs and how they participated. Linking data across databases required the creation and manual addition of common IDs to each dataset. The evaluation created IDs for this purpose based on the transformed customer and premise IDs in WPS’s tracking systems. These IDs only existed for customer and premise combinations that were eligible for the Pilots at the time that the Pilot evaluation launched, allowing the evaluation to focus only on community members who lived in the Pilot and control communities both prior to and throughout the evaluation.

As the pilot programs evolved, the evaluation encountered several challenges with the tracking systems.

Those challenges are as follows:

 “Common IDs” were not added to rebate and audit records as they were created: this caused a labor intensive manual process to add the IDs with each data delivery.

 The rebate and audit databases often had conflicting information, were not clear, and changed over time, which made it challenging to identify:

 Pilot measures separate from Focus measures

 When an audit took place

 What type of audit was completed (when more than one type was available in a community)

 Matching aggregated totals of participation (counts, savings, and incentives) to program reported participation for a time period was challenging due to dates and multiple tracking systems employed by the program.

 Migration to a new database mid-pilot resulted in significant issues in accounting for program effects.



 The databases WPS used to track the Plover rate assignment process were not optimized for understanding specific actions taken by customers. WPS maintained a rate assignment database for several months that tracked whether they should switch a premise during the mass switch in July, 2011. This database was optimized to give WPS a list of the premises they should switch in July. It did not track specific dates of specific decisions associated with a premise nor the

decision maker. This made it difficult for the evaluation to determine critical customer actions such as the decision to opt out or opt into a pilot rate or which premises experience a change in customer during the lead-up period. WPS maintained a second database to track customer actions after the mass switch occurred. The variables used as keys in the pre-switch database were different than those used in the post-switch database, which further complicated the evaluation team’s ability to determine customer actions.

 Participation in upstream lighting programs could not be accounted for due to a lack of customer information from their purchase.

 Installation of energy savings measures from the School to Home program could not be initially accounted for due to a failure to compile the returned program surveys.

Several of these challenges were unavoidable during the course of this pilot. However, a few of the challenges could have been reduced or eliminated by choosing (or obtaining permission to use) alternative data tracking options before the pilot began. DNV KEMA’s recommends that tracking participation for future pilot programs of this scope weight the importance of the following while designing their customer programming:

 Add (and consistently utilize) specific fields to flag program participation as separate from participation in other programs when multiple programs are tracked in the same database (or when one program is a subset of another).

 Add (and consistently utilize) specific fields to identify differences in program participation: for example, if two different audits are offered, ensure it is easy to identify each as distinct in the database.

 Ensure that a common ID is present in all relevant databases to link records if multiple databases are used to track participation.