## Periodic Reporting for period 2 - FlexAnalytics (Advanced Analytics to Empower the Small Flexible Consumers of Electricity)

Reporting period: 2019-08-01 to 2021-01-31

The so-called Digital Age has remarkably increased our ability to make, compile, and record observations about our society and Nature in the form of data. This, in turn, has given us a unique opportunity to transform the different productive sectors of our modern economies, so that we can make a better and more sustainable use of the physical resources our planet provides us with. The power sector, in particular, must lead this transformation as the backbone structure on which human activities rest and the main gateway for the large-scale penetration of renewable energy sources. Within this context, the overall objectives of FlexAnalytics are the following:

1. The development of a novel mathematical framework for decision-making, through which we can fully exploit all the data generated by the power sector. This will tremendously facilitate the establishment of renewables-dominated power systems. To this end, we are leveraging and pioneering cutting-edge results in the fields of machine learning and optimization under uncertainty.

2. The utilization of said new mathematical framework for decision-making to devise a system whereby a pool of small flexible prosumers of electricity can participate in wholesale electricity markets on equal footing with large power producers and consumers. It is well known already that activating the demand-side of the power sector can bring potential cost savings in the order of billions of euros to society, while increasing the capability of the electricity sector to accommodate larger amounts of renewable energy sources.

3. The use of data-science technologies for the efficient, secure and sustainable operation of power systems. Our aim is to upgrade the current procedures by which power systems are operated so that they can take full advantage of the data they generate to produce cheaper electricity with a lower environmental and climate impact.

1. The development of a novel mathematical framework for decision-making, through which we can fully exploit all the data generated by the power sector. This will tremendously facilitate the establishment of renewables-dominated power systems. To this end, we are leveraging and pioneering cutting-edge results in the fields of machine learning and optimization under uncertainty.

2. The utilization of said new mathematical framework for decision-making to devise a system whereby a pool of small flexible prosumers of electricity can participate in wholesale electricity markets on equal footing with large power producers and consumers. It is well known already that activating the demand-side of the power sector can bring potential cost savings in the order of billions of euros to society, while increasing the capability of the electricity sector to accommodate larger amounts of renewable energy sources.

3. The use of data-science technologies for the efficient, secure and sustainable operation of power systems. Our aim is to upgrade the current procedures by which power systems are operated so that they can take full advantage of the data they generate to produce cheaper electricity with a lower environmental and climate impact.

In keeping with the three main objectives of FlexAnalytics, the work performed so far under the framework of this project has been structured around the following three lines of action:

1. The design of novel schemes for data-driven decision-making under uncertainty. The key idea here is that the decisions we make are typically influenced by uncertain (random) phenomena. Our goal as decision-makers is thus to minimize the regret or cost that we foresee our decisions will entail. To this aim, besides, we usually gather information on all those factors that, we believe, can help us reduce the level of uncertainty we are faced with. This information is often known as “the context”. In this line, we have developed and explored two alternative schemes for decision-making, namely, a parametric and a non-parametric approach. In the former, we assume that the relation between the decisions we make and the context can be mathematically described by a member of a prespecified parametric family of functions, which is to be determined (by way of an optimization problem). In the latter, in contrast, the decisions are directly inferred from the data with no a-priori restriction on their relationship (which can be, therefore, of any nature). Logically, each of these two approaches has its pros and cons. Whereas the parametric scheme is easier to understand and interpret, and often leads to optimization problems that are computationally more tractable, it is often quite limited in the type of relations “context-decisions” that it can capture. On the contrary, the nonparametric scheme offers a superior modeling power, but is “data hungry”, more prone to “overfitting”, more difficult to interpret and more computationally demanding. To overcome some of these drawbacks, we have made use of distributionally robust optimization.

2. The development of a system for the participation of a pool of flexible loads in the wholesale electricity markets. In the first two years and a half of this project, our work in this regard has been primarily focused on two types of flexible loads of strategic importance to the future power sector, namely, a fleet of electric vehicles (EVs), with vehicle-to-grid capabilities, and a cluster of smart buildings. We have first built tools to mimic, as realistically as possible, the behavior of these two types of flexible loads and aggregations thereof, in order to assess the extent to which they can respond to the electricity price. Subsequently, for the case of the fleet of electric vehicles, we have developed a technology, based on data-driven inverse optimization, whereby we can encode the price-response of the EV-fleet in the form of a complex bid that can be directly submitted to the wholesale electricity market. Very importantly, the ability of this bid to predict the reaction of the EV-fleet to the electricity price is equal to or superior than that of state-of-the-art forecasting techniques. However, the bid our system produces has the great advantage that it can be directly interpreted, used, and processed by the market, precisely because it is a bid.

3. New methods for data-driven power system operations. We have first used the parametric approach for decision-making under uncertainty mentioned in point 1) above to improve the competitive edge of renewable power producers in electricity markets. Interestingly, our approach results in a computationally inexpensive linear optimization problem that can be efficiently run in any off-the-shelf solver, while substantially increasing the profitability of the renewable energy source by more than two percentage points in terms of average opportunity loss. We have also devised a data-driven procedure that leverages historical information to screen out inactive network constraints in the transmission-constrained unit-commitment problem, which is a widely used problem for power system operations that is very difficult to solve to global optimality in real-life instances. This procedure achieves substantial computational savings, in the range from 70% to 98% in terms of solution time depending on the congestion level of the power system and its size.

1. The design of novel schemes for data-driven decision-making under uncertainty. The key idea here is that the decisions we make are typically influenced by uncertain (random) phenomena. Our goal as decision-makers is thus to minimize the regret or cost that we foresee our decisions will entail. To this aim, besides, we usually gather information on all those factors that, we believe, can help us reduce the level of uncertainty we are faced with. This information is often known as “the context”. In this line, we have developed and explored two alternative schemes for decision-making, namely, a parametric and a non-parametric approach. In the former, we assume that the relation between the decisions we make and the context can be mathematically described by a member of a prespecified parametric family of functions, which is to be determined (by way of an optimization problem). In the latter, in contrast, the decisions are directly inferred from the data with no a-priori restriction on their relationship (which can be, therefore, of any nature). Logically, each of these two approaches has its pros and cons. Whereas the parametric scheme is easier to understand and interpret, and often leads to optimization problems that are computationally more tractable, it is often quite limited in the type of relations “context-decisions” that it can capture. On the contrary, the nonparametric scheme offers a superior modeling power, but is “data hungry”, more prone to “overfitting”, more difficult to interpret and more computationally demanding. To overcome some of these drawbacks, we have made use of distributionally robust optimization.

2. The development of a system for the participation of a pool of flexible loads in the wholesale electricity markets. In the first two years and a half of this project, our work in this regard has been primarily focused on two types of flexible loads of strategic importance to the future power sector, namely, a fleet of electric vehicles (EVs), with vehicle-to-grid capabilities, and a cluster of smart buildings. We have first built tools to mimic, as realistically as possible, the behavior of these two types of flexible loads and aggregations thereof, in order to assess the extent to which they can respond to the electricity price. Subsequently, for the case of the fleet of electric vehicles, we have developed a technology, based on data-driven inverse optimization, whereby we can encode the price-response of the EV-fleet in the form of a complex bid that can be directly submitted to the wholesale electricity market. Very importantly, the ability of this bid to predict the reaction of the EV-fleet to the electricity price is equal to or superior than that of state-of-the-art forecasting techniques. However, the bid our system produces has the great advantage that it can be directly interpreted, used, and processed by the market, precisely because it is a bid.

3. New methods for data-driven power system operations. We have first used the parametric approach for decision-making under uncertainty mentioned in point 1) above to improve the competitive edge of renewable power producers in electricity markets. Interestingly, our approach results in a computationally inexpensive linear optimization problem that can be efficiently run in any off-the-shelf solver, while substantially increasing the profitability of the renewable energy source by more than two percentage points in terms of average opportunity loss. We have also devised a data-driven procedure that leverages historical information to screen out inactive network constraints in the transmission-constrained unit-commitment problem, which is a widely used problem for power system operations that is very difficult to solve to global optimality in real-life instances. This procedure achieves substantial computational savings, in the range from 70% to 98% in terms of solution time depending on the congestion level of the power system and its size.

In the second half of the project lifetime, we plan to develop further the three lines of actions that has been previously mentioned. Next, we briefly discuss how.

1. Most existing parametric approaches for decision-making under uncertainty in the presence of context information have two major drawbacks. First, they can only address some particular decision-making problems and second, they are often solved using ad-hoc gradient-based algorithms. Against this background, we expect to devise a general-purpose mathematical framework that can be used for a much wider class of decision-making problems and whose solution can be tackled through off-the-shelf optimization solvers. This framework will be formulated using bilevel programming in order to find the member of the user-specified parametric family of functions that delivers the best decisions. On a different front, we also plan to develop a novel nonparametric approach for decision-making with contextual information that, unlike existing ones, is able to protect the decision-maker against the ambiguity about the relation “context-decisions”.

2. We will first extend our data-driven inverse optimization approach for market bidding to the case of a pool of smart buildings. This extension is particularly challenging due to the thermal inertia of buildings, which have a direct impact on their power consumption and hence, on their ability to respond to the electricity price. We plan to combine our data-driven inverse optimization approach with a geometric strategy to account for the buildings’ thermal dynamics when estimating the market bid to be submitted to the electricity market. Second, we will also extend our methodology even further to capture the price-response of a whole distribution network (containing active prosumers). This will represent a major step forward in the field, since existing methodologies either require complex DSO-TSO interaction schemes or need to have access to (private) detailed information on the distribution network and its components. Our approach, in contrast, can be seamlessly applied under the current DSO-TSO architecture and only necessitates data at the substation level (price and power), aside from some optional and readily available contextual information such as weather conditions, calendar effects, etc. Last but not least, we plan to develop dedicated algorithms to efficiently solve our inverse-optimization models.

3. While machine-learning techniques have already been used to learn, for example, the optimal solution to classical power system operational problems such as the economic dispatch and the unit commitment problem, the error inherent to the learning process may lead to solutions that cannot be implemented in practice. In this respect, we plan to design novel strategies to certificate and/or guarantee that the output of the learning tasks results in a feasible, and hence implementable solution. Another shortcoming of existing data-driven approaches to improve power system operations is the lack of reliable methods to select, among all the available data, those pieces of information that contribute most to the learning task. For this reason, we will propose new algorithms for discriminating relevant information, with the ultimate purpose of making data-driven methods for power system operations more efficient and interpretable.

1. Most existing parametric approaches for decision-making under uncertainty in the presence of context information have two major drawbacks. First, they can only address some particular decision-making problems and second, they are often solved using ad-hoc gradient-based algorithms. Against this background, we expect to devise a general-purpose mathematical framework that can be used for a much wider class of decision-making problems and whose solution can be tackled through off-the-shelf optimization solvers. This framework will be formulated using bilevel programming in order to find the member of the user-specified parametric family of functions that delivers the best decisions. On a different front, we also plan to develop a novel nonparametric approach for decision-making with contextual information that, unlike existing ones, is able to protect the decision-maker against the ambiguity about the relation “context-decisions”.

2. We will first extend our data-driven inverse optimization approach for market bidding to the case of a pool of smart buildings. This extension is particularly challenging due to the thermal inertia of buildings, which have a direct impact on their power consumption and hence, on their ability to respond to the electricity price. We plan to combine our data-driven inverse optimization approach with a geometric strategy to account for the buildings’ thermal dynamics when estimating the market bid to be submitted to the electricity market. Second, we will also extend our methodology even further to capture the price-response of a whole distribution network (containing active prosumers). This will represent a major step forward in the field, since existing methodologies either require complex DSO-TSO interaction schemes or need to have access to (private) detailed information on the distribution network and its components. Our approach, in contrast, can be seamlessly applied under the current DSO-TSO architecture and only necessitates data at the substation level (price and power), aside from some optional and readily available contextual information such as weather conditions, calendar effects, etc. Last but not least, we plan to develop dedicated algorithms to efficiently solve our inverse-optimization models.

3. While machine-learning techniques have already been used to learn, for example, the optimal solution to classical power system operational problems such as the economic dispatch and the unit commitment problem, the error inherent to the learning process may lead to solutions that cannot be implemented in practice. In this respect, we plan to design novel strategies to certificate and/or guarantee that the output of the learning tasks results in a feasible, and hence implementable solution. Another shortcoming of existing data-driven approaches to improve power system operations is the lack of reliable methods to select, among all the available data, those pieces of information that contribute most to the learning task. For this reason, we will propose new algorithms for discriminating relevant information, with the ultimate purpose of making data-driven methods for power system operations more efficient and interpretable.