Periodic Reporting for period 3 - EOPEN (EOPEN: opEn interOperable Platform for unified access and analysis of Earth observatioN data)
Okres sprawozdawczy: 2020-01-01 do 2020-10-31
"EOPEN is a platform targeted at non-expert users who want to easily access big volume of Copernicus data, to successfully address the Big Data challenges.
EOPEN main objective is to ensure scalability of the data standardization, fusion and exchange methods, combining also non-EO data (e.g. meteorological and social media (tweets)) and metadata annotation. EOPEN combines mature ICT solutions, building on top of existing European High-Performance Computing (HPC) infrastructure.
An inherent difficulty in using EO data is the variety of data sources and its usage without the knowledge of the underlying algorithms; application development requires the support of IT specialists and a relevant effort for integration.
Fostering interoperability aims at removing obstacles to facilitate the exchange of information between applications, databases, and other computer systems. Among the challenges are the heterogeneous hardware infrastructures changing in time; the scalability to cope with increasing processing needs, as it may occurs; the ability to integrate proprietary servers, cloud computing resources and the possibility to implement various algorithms with different needs in terms of programming language, processing resources (RAM, Disk, GPU, …), data collections, etc.
The EOPEN platform has tackled interoperability by integrating user defined algorithms and executing them in a ""federated platform"", by interacting with local and remote systems.
Three teams have collaborated to build-up the EOPEN platform: a technical and a scientific team for the development of the core framework and its extra components, and a service providers' team for setting up services for the stakeholders targeted in the Pilot Use Cases.
EOPEN can be accessed in two ways - either to visualize layers of information (EOPEN data and products) through a GIS viewer (OpenSphere) (general users); or for using/creating workflows through a graphical interface (expert users) (Fig. 3). The User interface is available in English, Italian, Finnish and Korean."
EOPEN main objective is to ensure scalability of the data standardization, fusion and exchange methods, combining also non-EO data (e.g. meteorological and social media (tweets)) and metadata annotation. EOPEN combines mature ICT solutions, building on top of existing European High-Performance Computing (HPC) infrastructure.
An inherent difficulty in using EO data is the variety of data sources and its usage without the knowledge of the underlying algorithms; application development requires the support of IT specialists and a relevant effort for integration.
Fostering interoperability aims at removing obstacles to facilitate the exchange of information between applications, databases, and other computer systems. Among the challenges are the heterogeneous hardware infrastructures changing in time; the scalability to cope with increasing processing needs, as it may occurs; the ability to integrate proprietary servers, cloud computing resources and the possibility to implement various algorithms with different needs in terms of programming language, processing resources (RAM, Disk, GPU, …), data collections, etc.
The EOPEN platform has tackled interoperability by integrating user defined algorithms and executing them in a ""federated platform"", by interacting with local and remote systems.
Three teams have collaborated to build-up the EOPEN platform: a technical and a scientific team for the development of the core framework and its extra components, and a service providers' team for setting up services for the stakeholders targeted in the Pilot Use Cases.
EOPEN can be accessed in two ways - either to visualize layers of information (EOPEN data and products) through a GIS viewer (OpenSphere) (general users); or for using/creating workflows through a graphical interface (expert users) (Fig. 3). The User interface is available in English, Italian, Finnish and Korean."
Three Pilot Use Cases (PUCs) have been implemented to demonstrate the platform, namely, flood risk monitoring, food security and climate change - Fig.1 illustrates the implementation concept.
Initially the data requirements have been identified in terms of data and products available either from external sources or generated in the platform - the Joint Decision and Information Governance approach has been adopted to analyse the decision process and identify data needs and possible improvements.
One of the main objectives of EOPEN is to fuse Sentinel data with multiple, heterogeneous and big data sources; processing of large streams of data is based on open-source and scalable algorithms of change detection, event detection, data clustering.
A variety of data has been added to the platform to serve the Use Cases; Copernicus satellite data is provided through an “Umbrella Hub”, scouting data on various Copernicus hubs (including ONDA-DIAS); meteorological and climatological data are normally available from the Finnish Meteorological Institute and from the Copernicus Land Services; other products are available from other sources. Moreover, social media data (Tweets) can be filtered, depending on the topic and displayed on satellite maps, allowing for communities/event detection.
Prototyping has allowed the tuning to stakeholders' needs - the matching of the data and functionalities' requirements has been monitored continuously during the Use Case implementation; the implemented functionalities can be adapted to different contexts (Fig. 2 illustrates a conceptual scheme of the platform).
Training material aimed at general users and developers has been produced (videos; slides, webinars, etc.), in addition to user and developer guides which can be also accessed through the platform.
In the following some aspects of the implemented modules are mentioned.
Several methods have been applied for the delineation of surface water and flooded areas mapping to serve the Flood Use Case, ranging from thresholding to AI methods (e.g. Recursive Neural Network - RNN; Deep Neural Network - DNN; Deep Convolution Neural Network - DCNN); they make use of Sentinel 1/Sentinel 2 data, or both, in addition to other data (e.g. Digital Elevation Models).
AI methods have been also applied to generate updated Land cover maps (outlier detection), and paddy rice mapping (RNN; Random Forest classifier), based on Sentinel 2 and/or Sentinel 1 data, to serve the food security Use Case, focused on paddy rice crop monitoring and yield estimation in South Korea.
Weather and climate data have been provided to serve the Climate Change Use Case, focused on Finland, targeting reindeers' herders and researchers and the Finnish transportation agency (e.g. HIRLAM forecast, the FMI GlobSnow and the SMOS Freeze/Thaw and climate projections); also, several tools for statistical and time-series analysis, historical data browsing, etc. have been implemented.
Initially the data requirements have been identified in terms of data and products available either from external sources or generated in the platform - the Joint Decision and Information Governance approach has been adopted to analyse the decision process and identify data needs and possible improvements.
One of the main objectives of EOPEN is to fuse Sentinel data with multiple, heterogeneous and big data sources; processing of large streams of data is based on open-source and scalable algorithms of change detection, event detection, data clustering.
A variety of data has been added to the platform to serve the Use Cases; Copernicus satellite data is provided through an “Umbrella Hub”, scouting data on various Copernicus hubs (including ONDA-DIAS); meteorological and climatological data are normally available from the Finnish Meteorological Institute and from the Copernicus Land Services; other products are available from other sources. Moreover, social media data (Tweets) can be filtered, depending on the topic and displayed on satellite maps, allowing for communities/event detection.
Prototyping has allowed the tuning to stakeholders' needs - the matching of the data and functionalities' requirements has been monitored continuously during the Use Case implementation; the implemented functionalities can be adapted to different contexts (Fig. 2 illustrates a conceptual scheme of the platform).
Training material aimed at general users and developers has been produced (videos; slides, webinars, etc.), in addition to user and developer guides which can be also accessed through the platform.
In the following some aspects of the implemented modules are mentioned.
Several methods have been applied for the delineation of surface water and flooded areas mapping to serve the Flood Use Case, ranging from thresholding to AI methods (e.g. Recursive Neural Network - RNN; Deep Neural Network - DNN; Deep Convolution Neural Network - DCNN); they make use of Sentinel 1/Sentinel 2 data, or both, in addition to other data (e.g. Digital Elevation Models).
AI methods have been also applied to generate updated Land cover maps (outlier detection), and paddy rice mapping (RNN; Random Forest classifier), based on Sentinel 2 and/or Sentinel 1 data, to serve the food security Use Case, focused on paddy rice crop monitoring and yield estimation in South Korea.
Weather and climate data have been provided to serve the Climate Change Use Case, focused on Finland, targeting reindeers' herders and researchers and the Finnish transportation agency (e.g. HIRLAM forecast, the FMI GlobSnow and the SMOS Freeze/Thaw and climate projections); also, several tools for statistical and time-series analysis, historical data browsing, etc. have been implemented.
In the following some foreseen areas of improvement are briefly illustrated.
“Enable value adding services on generic data and information storage and processing facilities which can allow public and commercial users effective production environment to interact with and serve their user base without deploying their own storage and processing facilities.”
> Users can select any DIAS for executing their processes.
“Make access to the Copernicus data and information easy and user friendly through scalable dissemination and exploitation software based on international standards.”
“Foster the establishment of interoperable access facilities to all EU Member States.”
> Interoperability has been tackled as mentioned above.
“Establish a link with other big data initiatives.”
>> Project partners established contacts with other big data projects (e.g. BETTER and CANDELA) during a Big Data Hackathon, organized by the Commission (November 2019) resulting in publications; collaboration will be pursued beyond project duration.
“Provide user community tools including best-practices”
> Implemented functionalities, methods of analysis and modules can be adapted to different contexts; the available promotional and training material will be usefull for exploitation.
“Ensure resilience of the overall dissemination and exploitation system.”
> Being system agnostic with reference to data sources and execution environments it can adapt to changes.
“Optimize the use of Copernicus data by non-traditional user communities to meet societal challenges.”
> EOPEN product/services are complimentary to the stakeholder’s systems; non-expert users showed interest in the system at the info-days and webinars organized by each Pilot Use Case.
“Synergies among EOPEN and other consortia will broaden the end user community”.
The EOPEN platform has been promoted in several ways and on various occasions, as mentioned below:
- H2020 EOPEN MOSES poster presented at ESA MWBS 2018
- EOPEN presented to a wide non-sectorial audience at COWM 2018
- EOPEN-MOSES CWD-ONDA combined usage video produced in cooperation with MOSES partners (https://eopen-project.eu/puc/)
- ESA Phi Weeks (September 2019 & 2020)
- Asian Conference of Remote Sensing (ACRS) (October 2019) - session on EOPEN
- EC Big Data Hackathon (November 2019)
- EC EO Big Data Conference (part of the Final Review Meeting), November 2020.
“Enable value adding services on generic data and information storage and processing facilities which can allow public and commercial users effective production environment to interact with and serve their user base without deploying their own storage and processing facilities.”
> Users can select any DIAS for executing their processes.
“Make access to the Copernicus data and information easy and user friendly through scalable dissemination and exploitation software based on international standards.”
“Foster the establishment of interoperable access facilities to all EU Member States.”
> Interoperability has been tackled as mentioned above.
“Establish a link with other big data initiatives.”
>> Project partners established contacts with other big data projects (e.g. BETTER and CANDELA) during a Big Data Hackathon, organized by the Commission (November 2019) resulting in publications; collaboration will be pursued beyond project duration.
“Provide user community tools including best-practices”
> Implemented functionalities, methods of analysis and modules can be adapted to different contexts; the available promotional and training material will be usefull for exploitation.
“Ensure resilience of the overall dissemination and exploitation system.”
> Being system agnostic with reference to data sources and execution environments it can adapt to changes.
“Optimize the use of Copernicus data by non-traditional user communities to meet societal challenges.”
> EOPEN product/services are complimentary to the stakeholder’s systems; non-expert users showed interest in the system at the info-days and webinars organized by each Pilot Use Case.
“Synergies among EOPEN and other consortia will broaden the end user community”.
The EOPEN platform has been promoted in several ways and on various occasions, as mentioned below:
- H2020 EOPEN MOSES poster presented at ESA MWBS 2018
- EOPEN presented to a wide non-sectorial audience at COWM 2018
- EOPEN-MOSES CWD-ONDA combined usage video produced in cooperation with MOSES partners (https://eopen-project.eu/puc/)
- ESA Phi Weeks (September 2019 & 2020)
- Asian Conference of Remote Sensing (ACRS) (October 2019) - session on EOPEN
- EC Big Data Hackathon (November 2019)
- EC EO Big Data Conference (part of the Final Review Meeting), November 2020.