Controlling for self-selection effects in energy efficiency program evaluation
Evaluation of energy efficiency programs must answer two critical questions:
1. How did the targeted customers’ behavior (including energy use) change after participating in or being exposed to the program?
2. How and to what extent were any observed changes influenced by they program? Or, put another way: What would the targeted customers have done in the absence of the program?
Because time travel is not available in the evaluator’s toolkit, question #2 cannot be directly answered. In energy efficiency program evaluation—as in medical research and the social sciences, randomized controlled trials (RCT), in which potential participants are randomly selected to receive program services or messages—are considered to be the preferred approach to characterizing program effects. The observed differences in outcomes between customers randomly assigned to the receive program services (the treatment group) and customers assigned to the control group represent the effects of the program.
Unfortunately, RCT methods are feasible only for evaluation of a limited range of programs. For example, RCT methods are logistically feasible when exposure to the program can be tightly controlled, as is the case in comparative feedback programs such as those offered by OPower and C3. Similarly, if value of program information or incentives is low, then it is feasible from a regulatory and customer relations standpoint to use random assignment methods. However, where program success requires broad publicity efforts and access to valuable incentives cannot be restricted, random assignment is not available. This is the case for most programs.
Many evaluations of energy efficiency programs in which participation was voluntary use the comparison of outcomes among participants to outcomes experienced by customers who did not participate (non-participants) to represent program effects. Such evaluation designs are vulnerable to the effects of self-selection bias, an issue long recognized in social science research. People who choose to participate in a program are likely to differ systematically from those who do not in regard to factors that directly affect observed differences in targeted behaviors such as thermostat control or purchase of energy efficiency appliances and, ultimately, on changes in energy consumption. Customers who voluntarily participate in an energy program are likely to have better awareness of energy efficiency benefits than non-participants, as well as stronger intrinsic motivation and greater resources. Thus, participants are more likely than non-participants to have implemented energy efficiency measures without the program, and a direct comparison of outcomes between those groups would likely overrepresent program results.
Statistical techniques to control for possible self-selection effects that can be applied to survey-based studies can provide a cost-effective way to improve the validity and accuracy of energy efficiency program evaluations based on participant/nonparticipant comparisons. DNV GL used such techniques to evaluate the energy impacts of an efficiency advocating home improvement show named PowerHouse™ that Interstate Power and Light Company, an Alliant Energy company, has produced for 16 years. In addition to the television show, PowerHouse has a website that provides supplemental articles, energy-related facts, energy savings calculators, and links to other informational websites such as those with information on Alliant Energy rebates. PowerHouse provides strong brand identification for Alliant Energy and a vehicle for customer outreach.
In this case, DNV GL proceeded from the principle that the customer attributes and beliefs associated with PowerHouse viewership would be similar to or the same as those associated with undertaking energy efficiency measures. The primary data that was collected consisted of surveys completed with 600 PowerHouse viewers and 605 non-viewers collected viewing status, energy efficiency knowledge and attitudes, and whether respondents undertook over 50 different energy-saving actions.
We found that PowerHouse viewers did score higher on a scale measuring energy efficiency knowledge and attitudes. We, therefore, used logistic regression modeling techniques to estimate a propensity to be a viewer for each sample customer, so that effects of program viewing on energy saving behaviors could be determined controlling for this pre-disposition.
The statistical models resulted in estimated savings attributable to the PowerHouse show of 10 kWh annually and 40 watts per viewing household—roughly equivalent to the installation of one CFL—after controlling for differences in viewing propensity and other characteristics. All attributable savings occurred from energy saving actions that were not supported by other programs. Although savings per viewer were modest, viewership of the program is sufficiently large to result in total savings that match those achieved by more conventional incentive programs. The results of this study show promising applications for evaluation of other “behavioral” programs that cannot be deployed with random assignment. These methods will become more important as program sponsors add voluntary features to behavioral programs.
DNV GL will be presenting the results of this study at the IEPEC conference in Chicago in mid-August. For more details, download the IEPEC paper at www.iepec.org. You can also check out the PowerHouse website and view show clips at http://www.powerhousetv.com/.
 Alliant Energy operates in Iowa, Wisconsin, and Minnesota, and the PowerHouse show airs in six television markets across those three states. However, the evaluation described in this paper focused only on Iowa customers and viewers.