Periodic Reporting for period 2 - MORE (Management of Real-time Energy Data)
Período documentado: 2022-04-01 hasta 2023-12-31
The energy industry and especially in the Renewable Energy Sources (RES) is one of the sectors which creates huge volumes of sensor data. The technical challenges are staggering: a single wind turbine has 7000 sensors that, ideally, we would like to report in subsecond periods (infeasible now due to data management limitations), while manufacturers monitor thousands of turbines. Moreover, the impact on the economy is huge: the RES industry was estimated at 1,469 billion USD in 2017 and even small increases in the productivity and the competitive advantage of the EU companies in the sector will have a significant impact on EU economy. MORE focuses on creating the software tools for data analytics tailored to the needs of RES stakeholders. The main objectives of MORE are:
Objective 1: Creating a platform that can consume billions of streams and 100s of petabytes of data.
MORE will address this challenge with a twofold strategy: a) edge analytics to locally perform fast simple event detection and b) time series modelling and compression, supported by the edge processing. We plan a system where data will be modelled and summarized in the whole data processing pipeline and analytics will be directly applied to the summaries.
Objective 2: Accurate prediction, forecasting and diagnostics.
MORE will achieve its aim for accuracy in prediction and diagnostics by focusing on incremental machine learning algorithms that can scale better than most existing approaches and are updated continuously. Moreover, it will also work on highly parallelizable pattern extraction methods, by focusing on motif extraction techniques which are developed especially for time series data. By extracting motifs from the collected historical data, we can identify patterns that are linked to important events in the RES installation or reveal properties of the components
Objective 3: Reduce the human effort for building complex learning models. Limited computational resources or lack of sophisticated techniques is not the only obstacle for extracting the desired information from time series data. A significant problem is the human effort required to parametrize a machine learning model. AutoML is a set of automation services that help the end user perform feature extraction, model tuning and other time consuming and complex tasks. However, existing AutoML approaches do not cover IMLA algorithms or ML algorithms for time series.
Objective 4: Have a tangible result on RES management
In MORE we strive not to just beat the benchmarks, but also provide solutions validated by our industry partners. The industry partners in the consortium have identified several important goals for RES data analysis. Their goal setting derives not only from their own efforts in data analysis, but also from several other RES sector stakeholders, who need feedback from monitored data analysis. The patterns and features that the RES industry desires, indicate conditions with important operational and financial impact.
MORE has achieved all its objectives and has delivered an open-source platform with software that implements tools for managing RES data with capabilities beyond the state-of-the-art.
Objective 1: Creating a platform that can consume billions of streams and 100s of petabytes of data.
MORE will address this challenge with a twofold strategy: a) edge analytics to locally perform fast simple event detection and b) time series modelling and compression, supported by the edge processing. We plan a system where data will be modelled and summarized in the whole data processing pipeline and analytics will be directly applied to the summaries.
Objective 2: Accurate prediction, forecasting and diagnostics.
MORE will achieve its aim for accuracy in prediction and diagnostics by focusing on incremental machine learning algorithms that can scale better than most existing approaches and are updated continuously. Moreover, it will also work on highly parallelizable pattern extraction methods, by focusing on motif extraction techniques which are developed especially for time series data. By extracting motifs from the collected historical data, we can identify patterns that are linked to important events in the RES installation or reveal properties of the components
Objective 3: Reduce the human effort for building complex learning models. Limited computational resources or lack of sophisticated techniques is not the only obstacle for extracting the desired information from time series data. A significant problem is the human effort required to parametrize a machine learning model. AutoML is a set of automation services that help the end user perform feature extraction, model tuning and other time consuming and complex tasks. However, existing AutoML approaches do not cover IMLA algorithms or ML algorithms for time series.
Objective 4: Have a tangible result on RES management
In MORE we strive not to just beat the benchmarks, but also provide solutions validated by our industry partners. The industry partners in the consortium have identified several important goals for RES data analysis. Their goal setting derives not only from their own efforts in data analysis, but also from several other RES sector stakeholders, who need feedback from monitored data analysis. The patterns and features that the RES industry desires, indicate conditions with important operational and financial impact.
MORE has achieved all its objectives and has delivered an open-source platform with software that implements tools for managing RES data with capabilities beyond the state-of-the-art.
MORE worked using real world requirements and data to provide tools that unlock the potential from the data collected by RES installation monitoring. The consortium industry partners provided data from various solar parks and wind turbines and supported different problems and use cases. Working with the vast amounts of data that our industry partners have revealed several challenges that could not have been completely foreseen. One of them is the variety of the data; each installation produces data that describe a different production mechanism, i.e. each source is unique, due to technological differences, differences in placement or due to age factors, etc. The most important challenge posed by data variety is the lack of labelled data, i.e. data that have explicitly been associated with different conditions, e.g. soiling.
The consortium developed a platform, which implements an edge-cloud model for data analytics, and for prediction and forecasting models that fully exploit the compression offered by ModelarDB. The architecture supports lightweight analytics on the edge and allows transferring large amounts of data to the cloud for resource-intensive analytics. MORE platform incorporates 4 main innovations:
ModelarDB, a time series data management system, that native supports compression of data.
The SAIL ( Streams And Incremental Learning ) library which offers incremental learning ML algorithms with AutoML features.
The RES Health monitoring toolkit, which offers tools that allow diagnosing and predicting problems in RES operation.
Α Self Service Visualization Platform for Renewable Energy Analytics, that supports big geo-located timeseries data from renewable energy sources, providing real-time analytics at macro and micro scales.
The work performed in the project has provided solutions with unpreceded scalability and accuracy capabilities, that address the needs of the RES analytics industry and offer techniques that can solve problems in all fields where analysis of streams and time series is needed.
The consortium developed a platform, which implements an edge-cloud model for data analytics, and for prediction and forecasting models that fully exploit the compression offered by ModelarDB. The architecture supports lightweight analytics on the edge and allows transferring large amounts of data to the cloud for resource-intensive analytics. MORE platform incorporates 4 main innovations:
ModelarDB, a time series data management system, that native supports compression of data.
The SAIL ( Streams And Incremental Learning ) library which offers incremental learning ML algorithms with AutoML features.
The RES Health monitoring toolkit, which offers tools that allow diagnosing and predicting problems in RES operation.
Α Self Service Visualization Platform for Renewable Energy Analytics, that supports big geo-located timeseries data from renewable energy sources, providing real-time analytics at macro and micro scales.
The work performed in the project has provided solutions with unpreceded scalability and accuracy capabilities, that address the needs of the RES analytics industry and offer techniques that can solve problems in all fields where analysis of streams and time series is needed.
In the second period, MORE has built on the tangible outcomes of the first period (libraries, tools, evaluation results) and has produced 4 innovations for time series and stream management tailored to the needs of the RES industry. These innovations are incorporated into a platform that supports the whole data processing flow from the RES park to the business user. Emphasis was given on scalability, ensuring that the implemented data mining and machine learning methods are efficiently executed in distributed settings, on tens of thousands of RES modules.
MORE widely disseminated its results and consulted multiple stakeholders in the RES and IT industries to ensure that actual industry needs are met, and that its innovations have the widest possible impact. The result was a platform that can perform meaningful analytics on RES data, accurately solve problems currently unhandled, and produce real value (e.g. in terms of increased production, or optimized maintenance procedures) to RES stakeholders. MORE’s data-driven techniques will have a large socio-economic impact on the field of RES, by increasing the efficiency of RES parks, reducing the cost of producing RES energy and enhancing the stability and predictability of the RES energy production.
MORE widely disseminated its results and consulted multiple stakeholders in the RES and IT industries to ensure that actual industry needs are met, and that its innovations have the widest possible impact. The result was a platform that can perform meaningful analytics on RES data, accurately solve problems currently unhandled, and produce real value (e.g. in terms of increased production, or optimized maintenance procedures) to RES stakeholders. MORE’s data-driven techniques will have a large socio-economic impact on the field of RES, by increasing the efficiency of RES parks, reducing the cost of producing RES energy and enhancing the stability and predictability of the RES energy production.