Consolidated Edison of New York: Residential Direct Install Program:
Process Evaluation Summary
Evaluation Conducted by: DNV KEMA as subcontractor to Navigant Consulting March 13, 2013
PROGRAM SUMMARY
Con Edison designed the Residential Direct Installation (RDI) program as an entry point for residential customers to evaluate their home’s energy performance and identify energy savings opportunities. The target market is residential customers defined as single family homes and multi-family homes with 2 to 4 units. The program offers a home energy survey for $50. The purpose of the survey is to evaluate the customer’s home for energy performance, document existing equipment and then identify and recommend energy efficiency upgrades.
Following the energy survey, the program installs up to nine low-cost, energy efficient measures. The energy efficiency measures include: six compact fluorescent lamps (CFLs), one smart strip, hot water pipe insulation, low-flow showerheads, weather stripping, door sweeps, window air conditioner timers, faucet aerators and setting back the water heater temperature. The program allots up to 10 CFLs per participant.
EVALUATION OBJECTIVE AND HIGH LEVEL FINDINGS
Table ES1 summarizes the RDI program savings goals and reported achievements for Con Edison from program initiation in 2010 through July 2011 (when the evaluation team fielded its surveys of participants and non-participants)1. The table also contains cumulative program achievement information as of 2011 year-end. The program achieved roughly one-third of its survey goal (34%); at 20%, it fell short of its energy savings goals.
Table ES1. Con Edison–RDI Goals and Reported Achievements
Parameter Program Goal 2010 – 2011 Progress through July 2011 July Percent of Goal Achieved Progress through Year-End 2011 Year-End Percent of Goal Achieved Number of Surveys 8,252 1,848 22% 2,832 34% Savings (MWh) 6,425 868 13% 1,295 20%
Coincident Peak Savings
(MW) 1.56 0.05 3% .13 9%
The overall objective of the RDI process evaluation is to assess the effectiveness and efficiency of program design, delivery and implementation processes. Specifically, the evaluation addressed six key program processes: program planning, infrastructure development, marketing and customer acquisition, program delivery, satisfaction with the program and interactions with other programs.
The RDI program is intended to serve as an entry point for residential customers wishing to identify opportunities for improved efficiency in their home. Con Edison contracts with Honeywell to deliver the RDI program on their behalf. The RDI program did not achieve its 2010-2011 program goals. It achieved 20 percent of its MWh savings goal and 9 percent of MW savings goal by the end of the year. The RDI Program generated its highest totals of savings by month in its first few months of operation (October 2010 through December 2010). The program posted lower monthly savings in 2011.
2
Although the program operated reasonably well, the achievement of program goals was greatly hindered by its inability to generate new customer interest as well as never achieving the average savings per participating home necessary. Program satisfaction levels were generally high among participants and program tracking was deemed sufficient. There are several areas for improvements noted below, all intended to improve the chance of program success. However, the core issues holding back program success are related to marketing and the lack of sufficient measure installation quantities at participating homes.
EVALUATION RECOMMENDATIONS AND PROGRAM ADMINISTRATOR RESPONSE The following recommendations were made by the evaluators conducting this study. Consolidated Edison’s initial response to these recommendations is also summarized below and will be tracked over time.
Recommendation 1: Program Planning and Design
a) Communicate and focus on increasing the number of installed measures per visit, or review program offerings and consider measure mix options that might have better installation rates than those currently provided.
b) Market aggressively to program-eligible customer groups which currently have low program participation compared to their population.
c) Aim to and make any needed staffing changes to complete more home energy surveys through increased marketing and subsequent customer program demand.
Response to Recommendation 1:
Since this research was conducted, Con Edison’s Marketing efforts have been centralized, greatly enhanced and better aligned, as a result of our full time Marketing Section Manager coming on board. Staffing changes are directly related to volume, and our Implementation Contractor has said that when demand increases, they would have no issue with increasing staff.
Recommendation 2: Infrastructure and Development
a) Consolidated Edison should revisit quality control processes regarding savings calculations and the manner in which values are transferred from the database to the scorecard reports. Consolidated Edison should also consider adding a measure-level identification (ID) field to both the customer installs table in the database and the scorecard reports and consider tracking multiple phone numbers in separate fields, either by type of phone (home, work, mobile) or order of preference (phone number 1, phone number 2, etc).
Response to Recommendation 2:
While this program continues to operate, the program’s future remains in doubt, and will be assessed in the future.
Recommendation 3: Marketing and Customer Acquisition
a) Develop an agreement between Con Edison and Honeywell about the roles that each entity will be responsible for in marketing the RDI Program.
b) Target future marketing in a manner that will allow technicians to work more efficiently in a smaller geographic area.
c) Provide clear and direct communication about specific program offerings throughout the program’s website and direct mail materials.
d) Streamline website customer call-to-action efforts to reduce customer confusion, and increase the likelihood of contracting the program through the website.
e) Ensure Con Edison has fully capitalized on the opportunity to market the RDI program at naturally occurring touch points such as customer call centers.
f) Create a baseline on number of page views, time on page, and click-through rates as a basis for regular review of web activity before undertaking significant website revisions.
g) Continue to emphasize saving energy and money in RDI marketing messages.
h) As program activity increases, consider automating customer information / survey requests between the Con Edison website and the Honeywell program schedulers.
i) Energy survey technicians should continue to request participants tell friends/family members if they are satisfied with the program.
Response to Recommendation 3:
Any website revisions will be contingent upon the continuation of the program.
Individual program marketing plans are currently being developed in 2013. These plans should be ready sometime in the latter portion of Q1 of 2013. However, a specific marketing plan and funds for this program will be limited and embedded in cross marketing efforts, as benefits from the other residential programs could enhance this program as well.
Recommendation 4: Program Delivery
a) Con Edison and Honeywell should review the newly implemented HomeSTAR survey report tracking log and customer service courtesy call procedures in the first half of 2012 to assure that it is
functioning properly.
b) Regularly review the capacity of program staffing levels to meet program goals and consider hiring additional staff as appropriate.
c) Clearly message the need for Honeywell to increase the number of installed measures per visit, whenever possible, and monitor whether the program change to install more smart strips or CFLs translates into additional program savings.
d) Continue to emphasize to energy survey technicians that program measures must be installed at participant homes.
e) Implement program changes and marketing material updates simultaneously. f) Consider emailing Home Energy Survey participants their results.
g) Con Edison branding in the field should be strengthened through improved use of Con Edison program identification in the field.
h) Consider streamlining program payment processes by offering customers more payment options (e.g., PayPal, Square, etc.).
Response to Recommendation 4:
While the program remains operational, any revisions made will be contingent upon the continuation of the program, and the status of the petition filed by the Company. Recommendation 5: Program Satisfaction
a) We recommend that Con Edison verify customer receipt of the HomeSTAR report as part of the ongoing Honeywell online survey effort among participants.
Response to Recommendation 5:
While the program remains operational, any revisions made will be contingent upon the continuation of the program, and the status of the petition filed by the Company.
4
a) Con Edison should ensure that all opportunities to broker other programs as an integral part of the home energy survey are fully leveraged. Energy survey technicians should be promoting additional Con Edison energy efficiency programs during every Home Energy Survey. Improving RDI program satisfaction may be the program’s strongest marketing and promotional asset.
Response to Recommendation 6:
While the program remains operational, any revisions made will be contingent upon the continuation of the program, and the status of the petition filed by the Company.
DETAILED PROCESS EVALUATION FINDINGS
Consolidated Edison was not successful in meeting its savings targets. The average achieved savings per visit is considerably less than what was anticipated in planning. The study also noted opportunities to boost program awareness and participation within the boroughs (such as Queens and Brooklyn) with the largest program-eligible customer population (e.g., Brooklyn) and among residents of multi-family (2-4) buildings. Program operations were noted to be generally
successful at addressing energy efficiency barriers as outlined in the program’s logic model. Through increased marketing and subsequent customer program demand, and ultimately, by executing more home energy surveys, the program will increase its energy savings achievements. The BBCS database appears to be sufficient for implementation and evaluation needs, and
comprehensively tracks complete information for each site, including measures installed during the survey, recommended measures, and tips for behavior changes to save energy. However, savings totals from the database do not match those in the scorecard reports, savings in the database do not use the same per unit savings values for measures of the same type and several records have greater than expected measures savings and/or quantities.
The “Power of Green” (Green Team) marketing campaign utilized many marketing vehicles to communicate Con Edison’s suite of available energy efficiency programs and emphasize its “green” brand. The vast majority of RDI program participants indicated they initiated program contact by phone and reported that saving money and energy was a major reason for program participation, which is consistent with the appeal in current program marketing. The lack of leads has resulted in less than optimal survey schedules. Green Team materials promoting the RDI program (including the website) are generally adequate, but do not provide information that implementers believe are more appropriate for presentation in-person and on-site, such as
program participation fees, and often do not list measures other than CFLs and smart strips. Only about one-quarter of RDI participants and even fewer non-participants (10%) report hearing about the program through the Con Edison website or using it in the past year. Both RDI program participants and non-participants most frequently mention hearing about the RDI program through a Con Edison bill insert. Word-of-mouth was not frequently named as a source of program awareness, although, field technicians were observed asking participants to talk to their family and/or friends about the program if they agreed it was a positive experience.
Honeywell is generally servicing customers who request a Home Energy Survey promptly. Honeywell has internal meetings that are effective and provide opportunity for best practices to be shared and for QA/QC issues to be resolved and emphasized as necessary. Technicians were observed to consistently perform the survey elements for each participant with the exception of some leave behind materials, which differed slightly. The technicians were able to answer most customer questions directly and adequately, or properly referred the question to the quality control coordinator.
Nearly three-fourths of participants recalled the energy survey technician discussing the energy survey results with them after completing the survey and nearly two thirds were able to recall receiving specific recommendations. A majority of participants indicated they received a written report of their results in three weeks or less, although fewer than 54% of them reported that the recommendations in the report were helpful. Installed measure satisfaction rates were reasonably
high, with the top three most frequently installed rated a 7 or higher. However, twenty-nine (13%) participants who received a smart strip indicated the technician did not plug equipment into the smart strip. The average amount of measure activity installed was significantly short of the average necessary to meet overall program savings goals.
Participant satisfaction with the program is high, with fifty-five percent ranking their satisfaction at 8 or higher on a 10-point satisfaction scale, although twenty percent of participants rated their program satisfaction lower than 5. This latter group most commonly indicated the program was not helpful to them, lacked information or were frustrated by not recalling having received a written report from the program. Energy survey technician ratings were very high and most participants strongly agreed that energy survey technicians explained the program and answered their questions. Nearly three out of four (73%) participants would recommend the program to a friend or family member.
Roughly one out of every five program participants recalled receiving Con Edison program recommendations from their energy survey technician and sixty-five percent of participants indicated their survey technician did not refer them to other Con Edison energy efficiency programs. Nearly two-thirds of program participants who confirmed receipt of program recommendations reported a referral to the Appliance Bounty program. RDI participants were more likely to participate in other Con Edison energy efficiency programs (Appliance Bounty, Res HVAC , and/or Room AC) compared to non-participants although roughly two-thirds of both RDI program participants and non-participants indicated no awareness of other energy efficiency programs outside of Con Edison offerings.
EVALUATION METHODS AND SAMPLING
The research and the findings expressed in this report are based upon the following evaluation activities:
Review of program planning and marketing materials,
Review of program tracking system, data, and other program delivery documents, In-depth interviews with:
o Con Edison staff (two completed, June 2010 and February 2011, respectively) o Honeywell staff (six completed, July 2010 to August, 2011)
o Honeywell subcontracted energy survey technicians (four completed, August, 2011 to October, 2011)
o Program participants (five completed, May and June 2011) Customer telephone surveys with:
o Program participants (293 completed, October and November 2011)
o Program non-participants (297 completed, November and December 2011), and
Field and office observations of implementation staff and energy survey technicians (August 18th and 19th, 2011).
