Post mortem: Dissecting an underperforming behavior program
Residential energy consumption accounted for 21% of total primary energy consumption in the United States (EIA 2015). In recent years, the average US household has become more energy efficient. While the residential sector currently is the largest electricity-consuming sector, residential energy intensity is expected to decline, with the average purchased electricity per household projected to fall 11.3% from 2015 to 2040. Utility energy efficiency programs, federal efficiency standards for most major end uses, and ambitious state and local building energy codes are the main reasons for this projected decline.
Behavior programs, like those offered by OPower and C3Energy, are increasingly viewed as an important tool in utilities’ energy efficiency program arsenal. These programs seek to engage levers such as feedback, social norms, and rewards to engage customers, influence energy use behavior, and achieve savings.
Savings from such behavior programs on average range from 1% to 3%. While these savings levels appear modest, they have proven to be consistent and typically represent a cost-effective option for utility program managers, with a cost of saved energy ranging from $0.01 per kilowatt-hour (kWh) to $0.08 per kWh according to a 2013 study of ten such programs.
An underperforming behavior program
DNV GL recently conducted an evaluation of the New York State Electric & Gas (NYSEG) and Rochester Gas and Electric (RG&E) Energy Saver program. Although the Energy Saver Program included incentives such as coupons or gift cards, the level of savings achieved fell short of ex-ante program estimates. Response to a web survey conducted as part of a parallel process evaluation among 5,400 program participants corroborated that finding – while some participants stated that their monthly bill was reduced as a result of program participation, the majority of customers did not report any change in their monthly bills.
The Energy Saver program which ran from February 2013 to April 2014 was an opt-in behavior based residential energy efficiency program
Customers also received nominal rewards for actual energy savings. The free online energy efficiency program for residential energy customers provided personalized energy-saving recommendations, which factored in location, weather, and home profile, and rewarded customers with points based on the amount of energy saved. With verified reductions in energy use, participants received two points for every kWh of electricity saved and twenty points for every therm of natural gas saved compared to the previous year. The points could be redeemed by customers for discounts from online retailers.
Customers could log in to their account and pledge energy saving actions. They were also sent email messages each month showing how much energy they saved and how many reward points they earned.
Underperformance along each dimension
DNV GL’s process evaluation brought together survey data with usage, cross-program participation, and program tracking data to extract layered insights on customer engagement with the program. We examined factors that could have an influence on program performance such as customer satisfaction, interest, engagement, elements of program design and website design.
Potential contributing factors to underperformance of the Energy Saver program included:
- Satisfaction/Interest. Although all participants opted in to the program, nearly half did not express a high level of interest in the program and two-fifths did not indicate a high level of satisfaction.
- Engagement as measured by interaction with the program website to pledge energy saving actions is poor, with less than one-sixths indicating that they did so.
- Incentives/Points. While over half of the participants indicate satisfaction with the mechanics of point redemption, an analysis of the verbatim comments indicates that respondents indicate room for improvement in the type of rewards offered and life of the points. This could translate into a negative impact on the program results given that rewards were the key and only mechanism to drive behavior change on this program.
- Over two-fifths of respondents’ self-reported changes in bills that were not aligned with actual levels of electricity consumption. The opacity of the relationship between occupant activities and choices and their energy use/energy bill has long been theorized as a hurdle to energy efficiency.
Opportunities to improve future programs
These findings provide an insight into opportunities for targeting, further engagement, and education in future programs.
High-yield customer segment? Customers who had participated in other EE programs performed better in the Energy Saver program relative to those who had not participated in other EE programs in terms of program satisfaction, engagement, and performance.
This provides directional evidence that cross-selling energy efficiency to participants in other NYSEG and RG&E EE programs could yield increased savings. As such, this segment of customers who have participated in multiple EE programs represent an attractive target with a higher likelihood of engagement for new programs from NYSEG and RG&E.
Perfunctory to persistent engagement. For most participants the engagement is perfunctory and/or transactional rather than proactive and persistent. That nearly half of the website visitors were looking for energy savings tips demonstrates that they were motivated to increase their energy savings. However, the fact that most of the participants logged into the website a few times annually or only at the beginning of the program indicates that the information provided was not necessarily compelling enough to warrant repeat visits and increased engagement. That the participants had already shown their interest in saving energy by opting in to the program, but became less satisfied and engaged over time shows that they may require more specific feedback from the program about their usage and energy savings activities.
Findings point to an inability to draw customers to the website and use it effectively as a channel for communication. Exploring the effectiveness of other channels of communications, such as text messages and smart phone apps, and meeting the customer where they are could yield fruit with respect to engagement.
Real-time feedback. A significant number of program participants did not have an accurate perception of their usage from their monthly bill and/or progress updates via email from the program.
This disconnect between customer perception of usage and their actual usage, as found here, is one of the reasons utilities explore alternatives like dynamic or real-time feedback mechanisms to provide customers with a true picture of their actual consumption so that they can see how it varies based on usage.
Ms. Sadhasivan will be presenting this work in greater detail at the ACEEE Summer Study conference at Asilomar, CA in August 2016. Click here to learn more about DNV GL’s Customer Intelligence and Analytics services.
(this in turn is derived from: Mazur-Stommen, S., & Farley, K. (2013). ACEEE Field Guide to Utility-Run Behavior Programs. ACEEE Report No. B132, p. 32. Available at: http://www.aceee.org/research-report/b132)