Periodic Reporting for period 2 - BETTER (Big-data Earth observation Technology and Tools Enhancing Research and development)
Periodo di rendicontazione: 2019-05-01 al 2021-01-31
In the last decades the European Space Agency (ESA) and the European Commission (EC), in partnership with other institutions and service operators such as EUMETSAT, have been developing a sustainable ecosystem of Earth Observation (EO) satellite missions, data providers, distributors, storage and processing centres, added value service providers and most importantly end users. Among these initiatives, Copernicus ecosystem has been consolidated through ground-breaking initiatives such as the launch of the Sentinel satellites constellation, providing operational satellite data, as well as the implementation of the Copernicus Services in different thematic areas, and is complemented by ESA’s own efforts in the Thematic Exploitation Platforms and other initiatives, making Europe the worldwide leader in the EO field.
In parallel to the improved data capabilities available in Europe, the downstream sector has been developing capabilities towards using EO to support decision making processes. From pipeline infrastructure monitoring to forestry management, EO-based services have been under development and are reaching operational status in many business domains. Satellite imagery can have enough resolution to monitor movements along borders, coast flooding, crop productivity, detect vessels, identify key urban features and infrastructures or monitor the evolution of construction works for licensing purposes.
The objective of the BETTER project was to tap into all these resources, capabilities and know-how established through past initiatives in the Space and ICT sector to develop the intermediate data handling layers (i.e. Data Pipelines), based on Big Data tools and technologies. In this way, the intermediate layers leveraged existing European EO-based services and infrastructures to: a) integrate Copernicus EO imagery and products with other high value datasets such as citizen observatories, mobile communication records, news and social media data; and b) maximise information and knowledge content within those integrated heterogeneous datasets to provide added-value EO-based products to a wide variety of users with diverse needs.
The service ecosystem constructed by the BETTER partners provides the ideal setup for the main objective of the BETTER project: the development of a Data Pipeline service layer. Data Pipelines are simply sequential components responsible for data access, integration, transformation, analytics and visualization. They are able to extract the required data or information from a wide array and large volumes of data and deliver it to different users, according to their exact specifications.
During the three years of the project, the project objectives were fully achieved by the development of BETTER as a Service based on: a) user requirements coming from 36 challenges launched by a wide range of promoters, from from international agencies to research institutes, universities and companies; b) development and integration in the Ellip platform of data pipelines that require the systematic processing of large heterogeneous EO based datasets; c) the implementation of on-demand applications that explored the data streams provided by the different developed pipelines; d) the definition of a robust business model, based on the experience gained during the project, to explore the developed pipelines or to develop new pipelines for future customers.
In parallel to the improved data capabilities available in Europe, the downstream sector has been developing capabilities towards using EO to support decision making processes. From pipeline infrastructure monitoring to forestry management, EO-based services have been under development and are reaching operational status in many business domains. Satellite imagery can have enough resolution to monitor movements along borders, coast flooding, crop productivity, detect vessels, identify key urban features and infrastructures or monitor the evolution of construction works for licensing purposes.
The objective of the BETTER project was to tap into all these resources, capabilities and know-how established through past initiatives in the Space and ICT sector to develop the intermediate data handling layers (i.e. Data Pipelines), based on Big Data tools and technologies. In this way, the intermediate layers leveraged existing European EO-based services and infrastructures to: a) integrate Copernicus EO imagery and products with other high value datasets such as citizen observatories, mobile communication records, news and social media data; and b) maximise information and knowledge content within those integrated heterogeneous datasets to provide added-value EO-based products to a wide variety of users with diverse needs.
The service ecosystem constructed by the BETTER partners provides the ideal setup for the main objective of the BETTER project: the development of a Data Pipeline service layer. Data Pipelines are simply sequential components responsible for data access, integration, transformation, analytics and visualization. They are able to extract the required data or information from a wide array and large volumes of data and deliver it to different users, according to their exact specifications.
During the three years of the project, the project objectives were fully achieved by the development of BETTER as a Service based on: a) user requirements coming from 36 challenges launched by a wide range of promoters, from from international agencies to research institutes, universities and companies; b) development and integration in the Ellip platform of data pipelines that require the systematic processing of large heterogeneous EO based datasets; c) the implementation of on-demand applications that explored the data streams provided by the different developed pipelines; d) the definition of a robust business model, based on the experience gained during the project, to explore the developed pipelines or to develop new pipelines for future customers.
The BETTER project has been divided into 3 consecutive cycles of around 1-year duration. At each of the different pre-defined cycles of the project, three challenges per thematic promoter (WFP, SatCen and ETHZ) are defined and implemented in their respective areas of expertise (Food Security, Geospatial Intelligence and GeoHazards), focused on activities that require Big Data handling capabilities. At the end of the project, the target of 36 successful challenges has been achieved.
Regarding the Design of the Data Pipelines, Challenge Break Down Workshops with the promoters are used for gathering the user stories and technical requirements necessary, which have been documented in the 3 Pipeline Requirement documents. In total, 64 Data Pipelines were considered achieved by the users and only one was not assessed by the external promoter.
The BETTER Collaboration and External Initiatives team has engaged with ESA, GEO, Copernicus and ICT sector initiatives and institutions. A total of 63 stakeholders for engagement were identified. During the project, most of them were successul, only 11 were not feasible.
The Website has been active and extended in line with our communication targets. Online presence has been established afterwards on a number of social media platforms. The project branding material, logo and the theme have been utilised for all the designed material, in addition to the social media pages and the website. BETTER contributions have been presented in 9 conferencies, such as ESA Phi week, Big Data from space or Living Planet Symposium 2019.
Regarding the Design of the Data Pipelines, Challenge Break Down Workshops with the promoters are used for gathering the user stories and technical requirements necessary, which have been documented in the 3 Pipeline Requirement documents. In total, 64 Data Pipelines were considered achieved by the users and only one was not assessed by the external promoter.
The BETTER Collaboration and External Initiatives team has engaged with ESA, GEO, Copernicus and ICT sector initiatives and institutions. A total of 63 stakeholders for engagement were identified. During the project, most of them were successul, only 11 were not feasible.
The Website has been active and extended in line with our communication targets. Online presence has been established afterwards on a number of social media platforms. The project branding material, logo and the theme have been utilised for all the designed material, in addition to the social media pages and the website. BETTER contributions have been presented in 9 conferencies, such as ESA Phi week, Big Data from space or Living Planet Symposium 2019.
BETTER main outcome was the development of an Big Data handling intermediate layer as a service (BETTER as a service), able to handle with large volume and heterogeneous datasets to boost the effective use of the different European EO data storage and processing facilities by the wider range research, service provider and other end users communities and maximise the information and knowledge extraction potential behind these datasets.
BETTER as a Service facilitates the handling of large volumes of data and support innovative service creation compatible with the main EO data access and processing infrastructures of NextGEOSS, Copernicus DIAS or the Collaborative Ground Segment infrastructure. The cloud based architecture of the platform in which the pipelines are developed, the experience of the consortium in developing and leading many of these initiatives (e.g. NextGEOSS and ESA’s TEPs) ensures the compatibility of technologies and standards used for discovery, access and processing and deep integration into this encompassing ecosystem.
BETTER had a specific WP with significant allocated effort dedicated to engaging with other EO and ICT Big Data related initiatives. The focus of its activities were on establishing active collaborations that go beyond joint communication activities and to actually define and coordinate technical initiatives to achieve mutually beneficial goals. This led to the integration of additional data streams, technologies and tools into the BETTER service layer, and eventually in other European and International projects (e.g. Copernicus Integrated Ground Segment (IGS) and Services, ESA’s TEPs, GEO initiatives, flagships and AIPs, and others). A good example of this is the collaboration between BETTER and EOpen in developing a pipeline that took advantage of an API developed by EOpen that made available Twitter data related to natural disasters. That data was used to develop an automatic trigger a pipeline that processed Sentinel-1 earthquake derived data whenever an earthquake took place.
BETTER as a Service facilitates the handling of large volumes of data and support innovative service creation compatible with the main EO data access and processing infrastructures of NextGEOSS, Copernicus DIAS or the Collaborative Ground Segment infrastructure. The cloud based architecture of the platform in which the pipelines are developed, the experience of the consortium in developing and leading many of these initiatives (e.g. NextGEOSS and ESA’s TEPs) ensures the compatibility of technologies and standards used for discovery, access and processing and deep integration into this encompassing ecosystem.
BETTER had a specific WP with significant allocated effort dedicated to engaging with other EO and ICT Big Data related initiatives. The focus of its activities were on establishing active collaborations that go beyond joint communication activities and to actually define and coordinate technical initiatives to achieve mutually beneficial goals. This led to the integration of additional data streams, technologies and tools into the BETTER service layer, and eventually in other European and International projects (e.g. Copernicus Integrated Ground Segment (IGS) and Services, ESA’s TEPs, GEO initiatives, flagships and AIPs, and others). A good example of this is the collaboration between BETTER and EOpen in developing a pipeline that took advantage of an API developed by EOpen that made available Twitter data related to natural disasters. That data was used to develop an automatic trigger a pipeline that processed Sentinel-1 earthquake derived data whenever an earthquake took place.