In late 2014, Dr. Arik Levinson, an economics professor at Georgetown University, requested that Energy Innovation review his forthcoming paper, “How Much Do Building Energy Codes Really Save? Evidence from California.” On January 8, 2015, we sent the following response to Dr. Levinson, detailing our feedback on his analysis of California’s energy savings from building codes. The take-away is that Dr. Levinson’s analysis is incomplete and therefore inconclusive. The most conspicuous reason for this conclusion is that his analysis omits natural gas, the most common energy source in California buildings.
Dear Professor Levinson,
Recently, you contacted Energy Innovation LLC and requested that we review a draft of your paper, “How Much Energy Do Building Energy Codes Really Save? Evidence from California.” Thank you for providing us with the chance to review your paper. We are always eager to understand the real-world impacts of energy policy, and too few take the time to do this important work.
Unfortunately, we believe that the analysis described in the paper has a number of flaws that undermine its ability to support your conclusions. These issues are briefly outlined below. We believe that significant revisions to the analysis procedure would be necessary in order to accurately measure the impacts of building energy codes.
First, most of your discussion only considers electricity, not all forms of energy. This is problematic because the vast majority of homes in California use natural gas for space heating and water heating. Building codes have a disproportionate effect on energy used for HVAC and water heating, relative to their impacts on other energy uses. By excluding natural gas from much of your analysis, you miss much of the energy use improvement from building codes, because you are not measuring the energy source that is primarily impacted.
We note that you did include natural gas in a few models, though you only break it out individually in one. In the single model where you do break out natural gas, the results (Table 3) show statistically significant and increasingly negative coefficients for homes built after building codes were implemented. These results suggest that in fact, building codes have led to decreased energy use in new homes.
Compounding this problem, the RASS dataset used in your analysis omits homes with electric space and water heating. This means that even if we consider only electricity (and not natural gas), the buildings that are most likely to have seen reductions in energy usage have been excluded. This is because, as noted above, building codes have much less impact on electricity use from lighting, appliances, personal electronics, etc. than on electricity use from HVAC and water heating systems. This problem is exacerbated by the fact that plug loads have increased significantly since building codes were first implemented. For example, a comparison of RECS data in California from 1993 and 2009 shows that over those years the amount of electricity used for appliances in the average California home increased from 12.5 MBtu/year to 15.3 MBtu/year, an increase of 22 percent. In essence, your analysis is checking for a statistically significant relationship between building codes and total electricity use, including things such as lighting, television sets, and computers in addition to air conditioning (which many California households don’t even have). Accordingly, a failure to find a statistically significant relationship, especially given the increasing electricity load from consumer electronics, doesn’t shed much light on the effectiveness of building codes.
A third issue is the use of climate zone as a proxy variable for heating and cooling demand [in the Strategy 1 – RASS model). Climate zones are very rough, and this roughness contributes to the low r-squared value of 0.36 (for the case in which all controls are applied). A more accurate proxy for demand would be to use heating degree days (HDD) and cooling degree days (CDD). The use of a more accurate demand metric would reduce noise and make it easier to observe statistically significant trends. Your results, though they were not statistically significant, indicate that electricity use in new homes decreased after the first building codes were implemented. This is evidenced in Figure 5 and in the decreasing values of your building period coefficients after 1977. Fixing the issues identified in previous paragraphs and using a more accurate demand metric would likely clarify this trend and may result in statistically significant relationships.
Finally, you compare buildings’ energy use in California to buildings’ energy use in other states, using other states as a baseline for overall energy trends. That is, the energy use of buildings in other states is used as an estimate of how the energy use of California’s buildings would have changed, had California not enacted its strong building codes. This approach does not account for the spillover effects of California’s building and appliance codes, nor does it account for efficiency policies implemented in other states. As you note in California energy efficiency: Lessons for the rest of the world, or not?, published in November 2014 in the Journal of Economic Behavior and Organization, after the implementation of California’s first building and appliance codes, “appliance manufacturers quickly began meeting California’s energy efficiency standards nationwide, rather than designing and producing two sets of products.” Also, other states “soon followed California’s lead, in some cases mimicking or adopting California’s standards outright.” Thus, part of the effect of California’s building codes is incorporated into your baseline, and thus the energy savings resulting from those effects are not properly attributed to the building codes that were responsible.
In summary, we recommend the following approach:
· Include all relevant building fuels (especially natural gas and electricity) in your analysis.
· Do not exclude buildings that use electricity for space and water heating. More broadly, avoid excluding any class of buildings that is likely not to be representative of all buildings affected by building codes, to avoid skewing the analysis.
· Rather than climate zone, use HDDs and CDDs to estimate heating and cooling demand.
· Extract non-heating and cooling electricity use from your analysis before you run any correlations so that plug load changes are neutralized.
· If other states’ buildings’ energy consumption is to be used as a baseline, it is necessary to factor out the influence that California’s building codes have had on those states’ building codes, as well as the behavior of manufacturers who sell more efficient products or use more efficient building practices in those other states as a result of California’s building codes.
We appreciate that estimating the impacts of building codes is a challenging and important task, and we thank you for the opportunity to comment on your draft paper. We think that the corrections to the analysis methodology that we have suggested will greatly clarify the effect of building codes and enhance your paper.
Hal Harvey and Robbie Orvis