Periodic Reporting for period 4 - FlexAnalytics (Advanced Analytics to Empower the Small Flexible Consumers of Electricity)
Reporting period: 2022-08-01 to 2024-01-31
The power grid is a colossal infrastructure that generates tons and tons of data. Furthermore, it is the backbone structure on which modern economies rest and the main gateway for the large-scale integration of renewable energy sources. As such, the power sector must lead the world’s economy transition to net-zero emissions. Within this context, the overall objectives of FlexAnalytics are the following:
1. The development of a novel mathematical framework for decision-making, through which we can fully exploit all the data generated by the power sector.
2. The utilization of said mathematical framework to devise a system whereby a pool of small flexible prosumers of electricity can participate in wholesale electricity markets on equal footing with large power producers and consumers. Activating the demand-side of the power sector will facilitate the establishment of renewables-dominated power systems, while bringing potential cost savings in the order of billions of euros to society.
3. The use of data-science technologies for the efficient, secure and sustainable operation of power systems.
In pursuing these objectives, we have leveraged and pioneered cutting-edge results in the fields of machine learning and optimization under uncertainty. These methodological advances have, in turn, allowed us to unlock the potential of demand-side flexibility and upgrade the current procedures by which power systems are operated so that they can take full advantage of the data they generate to produce cheaper electricity with a lower environmental and climate impact.
1. The development of a novel mathematical framework for decision-making, through which we can fully exploit all the data generated by the power sector.
2. The utilization of said mathematical framework to devise a system whereby a pool of small flexible prosumers of electricity can participate in wholesale electricity markets on equal footing with large power producers and consumers. Activating the demand-side of the power sector will facilitate the establishment of renewables-dominated power systems, while bringing potential cost savings in the order of billions of euros to society.
3. The use of data-science technologies for the efficient, secure and sustainable operation of power systems.
In pursuing these objectives, we have leveraged and pioneered cutting-edge results in the fields of machine learning and optimization under uncertainty. These methodological advances have, in turn, allowed us to unlock the potential of demand-side flexibility and upgrade the current procedures by which power systems are operated so that they can take full advantage of the data they generate to produce cheaper electricity with a lower environmental and climate impact.
In line with the three main objectives of FlexAnalytics, our work has been structured around the following three lines of action:
1. The design of novel schemes for data-driven decision-making under uncertainty. The key idea here is that the decisions we make are typically influenced by uncertain (random) phenomena. Our goal as decision-makers is thus to minimize the regret or cost that we foresee our decisions will entail. To this aim, besides, we usually gather information on all those factors that, we believe, can help us reduce the level of uncertainty we are faced with. This information is often known as “the context”. In this line, we have developed alternative schemes for decision-making under uncertainty with contextual information. More specifically, we have investigated both parametric and non-parametric approaches. In the former, we assume that the relation between the decisions we make and the context can be mathematically described by a member of a prespecified parametric family of functions, which is to be determined (by way of an optimization problem). In the latter, in contrast, the decisions are directly inferred from the data with no a-priori restriction on their relationship (which can be, therefore, of any nature). Logically, each of these two approaches has its pros and cons. Whereas the parametric scheme is easier to understand and interpret, and often leads to optimization problems that are computationally more tractable, it is often quite limited in the type of relations “context-decisions” that it can capture. On the contrary, the nonparametric scheme offers a superior modeling power, but is “data hungry”, more prone to “overfitting” (which can be mitigated via robustification), more difficult to interpret and more computationally demanding.
2. The development of a system for the participation of a pool of flexible loads in the wholesale electricity markets. We have primarily focused on two types of flexible loads of strategic importance to the future power sector, namely, a fleet of electric vehicles (EVs), with vehicle-to-grid capabilities, and a cluster of smart buildings. We have first built tools to mimic, as realistically as possible, the behavior of these two types of flexible loads and aggregations thereof, in order to assess the extent to which they can respond to the electricity price. Subsequently, we have developed technologies, based on data-driven inverse optimization, whereby we can encode the price-response of the loads in the form of a complex bid that can be directly submitted to wholesale electricity markets. Very importantly, the ability of this bid to predict the reaction of the flexible loads to the electricity price is equal to or superior than that of state-of-the-art forecasting techniques. However, the bid our system produces has the great advantage that it can be directly interpreted, used, and processed by the market, precisely because it is a bid.
3. New methods for data-driven power system operations. We have successfully applied the novel schemes for data-driven decision-making under uncertainty that we have developed to address a number of paradigmatic problems in power system operations, such as the participation of weather-driven renewable power producers in electricity markets, the optimal power flow problem, the networked-constrained unit commitment problem, etc. In parallel, we have also devised data-driven and computational algorithms to efficiently solve these problems by means, for example, of screening out superfluous constraints or the tightening of large constants, achieving substantial computational savings.
1. The design of novel schemes for data-driven decision-making under uncertainty. The key idea here is that the decisions we make are typically influenced by uncertain (random) phenomena. Our goal as decision-makers is thus to minimize the regret or cost that we foresee our decisions will entail. To this aim, besides, we usually gather information on all those factors that, we believe, can help us reduce the level of uncertainty we are faced with. This information is often known as “the context”. In this line, we have developed alternative schemes for decision-making under uncertainty with contextual information. More specifically, we have investigated both parametric and non-parametric approaches. In the former, we assume that the relation between the decisions we make and the context can be mathematically described by a member of a prespecified parametric family of functions, which is to be determined (by way of an optimization problem). In the latter, in contrast, the decisions are directly inferred from the data with no a-priori restriction on their relationship (which can be, therefore, of any nature). Logically, each of these two approaches has its pros and cons. Whereas the parametric scheme is easier to understand and interpret, and often leads to optimization problems that are computationally more tractable, it is often quite limited in the type of relations “context-decisions” that it can capture. On the contrary, the nonparametric scheme offers a superior modeling power, but is “data hungry”, more prone to “overfitting” (which can be mitigated via robustification), more difficult to interpret and more computationally demanding.
2. The development of a system for the participation of a pool of flexible loads in the wholesale electricity markets. We have primarily focused on two types of flexible loads of strategic importance to the future power sector, namely, a fleet of electric vehicles (EVs), with vehicle-to-grid capabilities, and a cluster of smart buildings. We have first built tools to mimic, as realistically as possible, the behavior of these two types of flexible loads and aggregations thereof, in order to assess the extent to which they can respond to the electricity price. Subsequently, we have developed technologies, based on data-driven inverse optimization, whereby we can encode the price-response of the loads in the form of a complex bid that can be directly submitted to wholesale electricity markets. Very importantly, the ability of this bid to predict the reaction of the flexible loads to the electricity price is equal to or superior than that of state-of-the-art forecasting techniques. However, the bid our system produces has the great advantage that it can be directly interpreted, used, and processed by the market, precisely because it is a bid.
3. New methods for data-driven power system operations. We have successfully applied the novel schemes for data-driven decision-making under uncertainty that we have developed to address a number of paradigmatic problems in power system operations, such as the participation of weather-driven renewable power producers in electricity markets, the optimal power flow problem, the networked-constrained unit commitment problem, etc. In parallel, we have also devised data-driven and computational algorithms to efficiently solve these problems by means, for example, of screening out superfluous constraints or the tightening of large constants, achieving substantial computational savings.
FlexAnalytics has produced results that go beyond the state of the art in at least three fields or topics, namely:
1. In our primary goal of designing a system that allows small, flexible power consumers to participate in wholesale electricity markets on equal terms with the rest of the market players, we have not only managed to produce a cutting-edge toolbox for that purpose, but also, during the process, we have contributed in a very remarkable way to the development of what is now known as data-driven inverse optimization, and to its study as a new form of machine-learning particularly suitable for inferring decisions by rational agents.
2. From a methodological point of view, FlexAnalytics has produced new theory, methods and models by which decisions can be directly estimated from available data, with a significant increase in the decision value and a decrease in the decision risk. These results together constitute a novel mathematical framework for decision-making under uncertainty with contextual information, which has contributed in a decisive manner to the emergence and development of a new subfield known as Conditional Stochastic Optimization, which is today a hot topic within the Operations Research community.
3. Spurred by the need to solve the complex optimization problems derived from the application of data-driven optimization and conditional stochastic optimization to challenging problems in the operation of power systems and smart energy grids, FlexAnalytics has produced numerous strategies to address the solution of such problems in an efficient and effective manner. Various of these strategies are particularly original in that they combine machine-learning techniques with mathematical programming methods (such as the inclusion of valid inequalities, bounds and large-constants tightening, and constraint screening).
1. In our primary goal of designing a system that allows small, flexible power consumers to participate in wholesale electricity markets on equal terms with the rest of the market players, we have not only managed to produce a cutting-edge toolbox for that purpose, but also, during the process, we have contributed in a very remarkable way to the development of what is now known as data-driven inverse optimization, and to its study as a new form of machine-learning particularly suitable for inferring decisions by rational agents.
2. From a methodological point of view, FlexAnalytics has produced new theory, methods and models by which decisions can be directly estimated from available data, with a significant increase in the decision value and a decrease in the decision risk. These results together constitute a novel mathematical framework for decision-making under uncertainty with contextual information, which has contributed in a decisive manner to the emergence and development of a new subfield known as Conditional Stochastic Optimization, which is today a hot topic within the Operations Research community.
3. Spurred by the need to solve the complex optimization problems derived from the application of data-driven optimization and conditional stochastic optimization to challenging problems in the operation of power systems and smart energy grids, FlexAnalytics has produced numerous strategies to address the solution of such problems in an efficient and effective manner. Various of these strategies are particularly original in that they combine machine-learning techniques with mathematical programming methods (such as the inclusion of valid inequalities, bounds and large-constants tightening, and constraint screening).