Understanding the Energy Transition with a Machine Learning Toolbox

Project Information

ENECML

Grant agreement ID: 101001732

DOI

10.3030/101001732

EC signature date 26 May 2021

Start date 1 September 2021

End date 31 August 2026

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 1 679 125,00

EU contribution

€ 1 679 125,00

Coordinated by

AGENCIA ESTATAL CONSEJO SUPERIOR DE INVESTIGACIONES CIENTIFICAS
Spain

Periodic Reporting for period 3 - ENECML (Understanding the Energy Transition with a Machine Learning Toolbox)

Reporting period: 2023-09-01 to 2025-02-28

Summary of the context and overall objectives of the project

This project aims to build tools to better understand the economic impacts of the rapid transformation of electricity markets and to help design electricity markets going forward. The project develops and implements novel statistical tools and structural models that contribute to our understanding of this rapid transformation. The research focuses on firm strategic responses and investment (supply-side), consumer behavior, and welfare and distributional impacts (demand-side). The proposal presents several projects and methodologies that examine these issues in detail with unique high-frequency micro-data on firms and households.

The tools and models developed in this proposal can help understand the impacts of the energy transition, both on the supply and demand side. Among the expected methodological contributions, the project combines machine learning tools with more standard structural modeling. On the supply side, the proposal emphasizes the need to understand how strategic behavior interacts with market design in the presence of intermittent resources. On the demand side, the proposal highlights the importance of understanding the distributional implications of such changes with special attention to the residential sector and the most vulnerable socio-economic households.

In examining these questions in the context of electricity markets, I use recent developments in machine learning. Classification algorithms, as well as predictive approaches in data science, provide an auxiliary toolbox to the analysis of electricity markets with high-frequency data that are rapidly becoming essential and yet are not commonly used. Electricity markets lend themselves particularly well to this type of tools. Indeed, prediction algorithms have been used to operate these markets for decades due to the need to accurately predict demand to manage the electricity grid operation. The data revolution nowadays is access to not only aggregate demand data but also smart meter data from millions of households, which is opening the door to new avenues of research. I propose to develop a battery of procedures and methods to examine high-frequency data in these markets. The insights from analyzing electricity markets using these tools can also apply to other settings.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The work performed from the beginning of the project has focussed on two aims.

The first line of work has been devoted to studying the integration of renewable power in electricity markets by looking at detailed hourly data from markets with large renewable generation (supply side). Focusing on wind power, we have produced a new analysis that shows that the costs of integrating renewable power into the grid in the Iberian peninsula have remained small (Figure 1). We show new evidence of the decomposition of these costs and the behavior of the power plants providing reliable services. Focusing on solar power, we have examined how the expansion of transmission infrastructure in Chile has enabled the construction of solar plants in Atacama. We provide an economic framework to study the cost-benefit analysis of transmission lines while accounting for investment effects that could be anticipated. We use k-means tools to construct a simple but realistic model of the Chilean electricity market and find that investment effects are critical in assessing the cost and benefits of transmission infrastructure. On the opposite side of decarbonization, how to phase out coal and gas generation is a challenge. Using data from the US, we build a model of investment decisions to examine to which extent regulatory frameworks are delaying the phase-out of coal power.

The second line of work has been devoted to understanding the impacts of the energy transition on households, focusing on equity impacts (demand side). Using detailed smart meter data from millions of households, I have developed a new estimator of unobserved income to improve our understanding of winners and losers during the energy crisis. Using more aggregate consumer expenditure surveys from thousands of households, we have examined the equity implications from broader energy policies during the energy crisis that affect other goods (gasoline, natural gas). We have also used detailed consumption hourly data to understand how households respond to dynamic pricing, finding that consumers responded substantially to the implementation of time-of-use pricing in Spain.

In addition to these two lines of work, I have developed a conceptual framework to analyze climate policies, highlighting the trade-off between cost efficiency, effectiveness, fairness, and long-term considerations.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The project develops new tools and models to examine the energy transition. Each project provides new economic insights or tools that advance our ability to study the economic impacts of the energy transition and its related impacts on the supply and the demand side of electricity markets. Here, I summarize two of them.

On the supply side, the work on solar power integration in Chile provides general insights into how event-study methodologies might be ill-fitted to understand the impacts of transmission expansion projects. In particular, we provide general statements on when the event-study might over- or under-predict the benefits from increased integration. While the study focuses on electricity markets, this insight can broadly appeal to other applications.

On the demand side, the work on the equity impacts of dynamic pricing provides new econometric methodologies to estimate unobservable income at the household level. It expands the existing literature to consider a case in which the consumption set of households is highly multidimensional (every hour of the year) and shows how to exploit detailed smart meter data together with the income distribution at the zip code level to estimate income. These tools can be used beyond our application of interest, and have received attention in setting in which income is difficult to observe but subsidies are present (e.g. fuel subsidies).

Wind power increases operational costs modestly