State and Local Climate and Energy Program
Determining Policy and Program Impacts
- Developing an Action Plan
- Developing a GHG Inventory
- Identifying and Analyzing Policy Options
- Designing and Implementing Programs
- Choosing a Clean Energy Financing Program
- Leading by Example in Government Operations
- Engaging Stakeholders
- Determining Policy and Program Impacts
- Assisting Local Governments
- Types of Impacts
- Importance of Measuring Impacts
- Key Steps for Measuring Impacts
- Approaches to Program Evaluation
- Tools and Resources
Types of Impacts
States are implementing many policies and programs using energy efficiency, renewable energy, combined heat and power, and clean distributed generation. These policies and programs are helping states meet a range of environmental, energy, and economic goals because they can:
- Lower demand for energy
- Improve electric system reliability
- Reduce power plant emissions and improve people's health
- Promote economic development
Importance of Measuring Impacts
By measuring the impacts of existing policies and programs, states are able to:
- Compare costs to benefits to identify cost-effective options
- Demonstrate how clean energy policies and programs help achieve multiple state energy, environmental, and economic objectives
- Design or select new or expanded clean energy options that offer greater energy, environmental, and economic benefits
- Identify opportunities where clean energy can be used to support energy system, environmental, and/or economic development planning strategies across the state
- Build support for additional clean energy policies and programs
Additionally, measuring impacts across a spectrum of outcomes provides states with timely information to estimate policy and program impacts and improve implementation. This information can help states answer questions such as:
- Is the policy or program achieving its objectives? If so, how and why?
- How well has the policy or program worked? What is the magnitude of the impact?
- How reliable is the policy or program? Will it continue to generate benefits into the future?
- What changes are needed to improve the policy or program?
- Should the policy or program be expanded, adjusted, or cancelled?
By answering these questions, states can identify the most effective approaches, determine how to improve future policies or programs, and decide where to focus for greater impacts.
Reporting impacts and communicating them to key audiences can help states achieve:
- Transparency by documenting progress being made towards goals and enabling others to provide feedback
- Accountability to funding sources, helping to obtain continued support
- Improvement by providing recommendations for designing better programs
Key Steps for Measuring Impacts
Four steps are involved in measuring the impacts of policies and programs:
- Plan – Establish goals, define performance indicators, specify evaluation and reporting approaches, and allocate resources. The level of rigor and analysis required will depend on who needs the information and for what purpose.
- Track/Benchmark – Develop a tracking system, establish baseline reporting conditions and reporting period, and collect and organize performance data.
- Evaluate – Conduct impact, process, and/or market evaluations to measure benefits.
- Report – Report and promote evaluation findings, assess results, and modify the program as needed. Information on Engaging Stakeholders is available to assist states.
More details are available in EPA's Model Energy Efficiency Program Impact Evaluation Guide (PDF) (152 pp, 1.44M, About PDF).
Evaluation, Measurement and Verification for Energy Efficiency Programs
The terms evaluation, measurement, and verification (EM&V) encompass a wide range of industry-standard analyses used to assess clean energy programs. EM&V efforts are generally applied to energy efficiency programs administered by utilities, states, and third-party energy efficiency providers, as well as state-mandated renewable energy policies and programs funded by ratepayers.
State public utility commissions typically initiate and oversee EM&V activities for the purpose of:
- Determining whether overall objectives are being achieved
- Identifying any necessary program improvements
- Assessing program cost-effectiveness
- Estimating impacts and their persistence over time
- Providing the basis for capturing energy (kWh) and demand (kW) impacts in energy planning
Three types of EM&V, each with a different purpose, are frequently undertaken to assess program effectiveness.
|Evaluation Types||Purpose||Information Derived|
|Impact Evaluations||Quantifies the direct and indirect benefits of a program or project using measured savings or deemed savings methods.||Determines the quantity of energy and/or demand saved, the monetary value of these savings; can include the amount of emissions reductions and other non-energy benefits.|
|Process Evaluations||Indicates how to improve the structure and delivery of a program or project. These evaluations typically survey program stakeholders, analyze their feedback, and use this information to identify opportunities for program improvement.||Determines how well program or project processes are performing and provides recommendations for how these systems can be improved.|
|Market Effects Evaluations||Indicates how a program affects the structure or functioning of a market — or the behavior of participants in a market — that result from one or more program efforts.||Determines changes that have occurred in state operations and/or private markets, and evaluates how the marketplace is different as a result of the program.|
Determining the impacts of climate and energy policies and programs can be difficult because not everything that a state may want to measure is easily documented. Approaches to overcoming these challenges are presented on the linked pages.
- Calculating Energy Savings involves comparing observed presents a challenge of estimating what did happen relative to what "would have" happened.
- When Assessing Air Quality, GHG, and Public Health Benefits, air pollutants are often measured but can also be calculated based on emission factors.
- Assessing Electric System Benefits is complex because there are many interconnected pieces to an energy system and impacts from relatively minor changes in energy use are difficult to pinpoint.
- When Quantifying Economic Benefits, states must account for the many factors at work in economic systems. Assumptions of various levels of rigor are used, and states must decide what types and portions of economic impacts will be calculated (e.g., environmental externalities, jobs, etc.).
Tools and Resources
General EM&V Resources
U.S. DOE State and Local Energy Efficiency Action Network (SEE Action) Evaluation, Measurement, and Verification Working Group
SEE Action EM&V Resources Portal
US EPA EM&V Webinar Website:
International Energy Program Evaluation Conference (IEPEC)
Select EM&V Reference Documents
SEE Action Energy Efficiency Program Impact Evaluation Guide:
http://www1.eere.energy.gov/seeaction/pdfs/emv_ee_program_impact_guide.pdf (PDF) (180 pp, 2.8 MB, About PDF)
SEE Action Evaluation, Measurement, and Verification (EM&V) of Residential Behavior-Based Energy Efficiency Programs: Issues and Recommendations
http://www1.eere.energy.gov/seeaction/pdfs/emv_behaviorbased_eeprograms.pdf (82 pp, 1.96 MB, About PDF)
California Energy Efficiency Evaluation Protocols:
http://www.calmac.org/publications/EvaluatorsProtocols_Final_AdoptedviaRuling_06-19-2006ES.pdf (PDF) (34 pp, 220 KB, About PDF)
Regional EM&V Resources
Northeast Energy Efficiency Partnerships (NEEP) Regional Evaluation, Measurement and Verification Forum (EM&V Forum)
The Regional Technical Forum (RTF) of the Northwest Power and Conservation Council