Skip to main content

Big-data Earth observation Technology and Tools Enhancing Research and development

Periodic Reporting for period 1 - BETTER (Big-data Earth observation Technology and Tools Enhancing Research and development)

Reporting period: 2017-11-01 to 2019-04-30

In the last decades the European Space Agency (ESA) and the European Commission (EC), in partnership with other institutions and service operators such as EUMETSAT, have been developing a sustainable ecosystem of Earth Observation (EO) satellite missions, data providers, distributors, storage and processing centres, added value service providers and most importantly end users. Among these initiatives, Copernicus ecosystem has been consolidated through ground-breaking initiatives such as the launch of the Sentinel satellites constellation, providing operational satellite data, as well as the implementation of the Copernicus Services in different thematic areas, and is complemented by ESA’s own efforts in the Thematic Exploitation Platforms and other initiatives, making Europe the worldwide leader in the EO field.
In parallel to the improved data capabilities available in Europe, the downstream sector has been developing capabilities towards using EO to support decision making processes. From pipeline infrastructure monitoring to forestry management, EO-based services have been under development and are reaching operational status in many business domains. Satellite imagery can have enough resolution to monitor movements along borders, coast flooding, crop productivity, detect vessels, identify key urban features and infrastructures or monitor the evolution of construction works for licensing purposes.
The objective of the BETTER project is to tap into all these resources, capabilities and know-how established through past initiatives in the Space and ICT sector to develop the intermediate data handling layers (i.e. Data Pipelines), based on Big Data tools and technologies. In this way, the intermediate layers will leverage existing European EO-based services and infrastructures to: a) integrate Copernicus EO imagery and products with other high value datasets such as citizen observatories, mobile communication records, news and social media data; and b) maximise information and knowledge content within those integrated heterogeneous datasets to provide added-value EO-based products to a wide variety of users with diverse needs.
The service ecosystem constructed by the BETTER partners provides the ideal setup for the main objective of the BETTER project: the development of a Data Pipeline service layer. Data Pipelines are simply sequential components responsible for data access, integration, transformation, analytics and visualization. They are able to extract the required data or information from a wide array and large volumes of data and deliver it to different users, according to their exact specifications. The pipelines will be designed and implemented based on the requirements of the project's Thematic Promoters: the World Food Programme (WFP), representing the Food Security societal challenges, the European Union Satellite Centre (SatCen), responsible for the Geospatial Intelligence, and the Swiss Federal Institute of Technology Zurich (ETHZ), working in the GeoHazards research field.
The BETTER project has been divided into 3 consecutive cycles of around 1-year duration. At each of the different pre-defined cycles of the project, three challenges per thematic promoter (WFP, SatCen and ETHZ) are defined and implemented in their respective areas of expertise (Food Security, Geospatial Intelligence and GeoHazards), focused on activities that require Big Data handling capabilities. An additional set of three thematic challenges were planned to each cycles through invitations to external partner institutions, however it was decided during the 1s cycle to postponed them, having four in the 2nd cycle and 5 to the 3rd.
For the 1st challenge cycle of BETTER, the Challenge Achievement started late for part of the data pipelines and therefore completed at the month 16. Also, it was due to a learning process from the challenge promoters to use the BETTER framework to exploit the pipeline results that took a bit longer than expected. The Challenge Definition of the 2nd cycle has been done in parallel with the Challenge achievement of the 1st cycle so that it has taken longer due to the overlapping tasks. Nevertheless, the CRM2 and ABM2 milestones of the Challenge breakdown stage have been passed on the 11th April 2019.

Regarding the Design of the Data Pipelines, Challenge Break Down Workshops with the promoters are used for gathering the user stories and technical requirements necessary, which have been documented in the First and Second Pipeline Requirements. There are 22 challenges defined and the 9 of the first cycle are already on production phase.

The WP4 Collaboration and External Initiatives team has started to map the data provision, ESA, GEO, Copernicus and ICT sector initiatives and institutions the team will engage during its duration. WIthin task 4.1 14 stakeholders for engagement are already identified, but the activities haven’t started officially yet. Within task 4.2 14 Stakeholders for Engagement are already identified, from which 4 engagement activities have already started. In the case of task 4.3 6 Stakeholders for Engagement are already identified, 3 engagement activities have already started.

The Website has been active since late March, and is still being optimised and extended in line with our communication targets. Online presence has been established afterwards on a number of social media platforms. The project branding material, logo and the theme have been utilised for all the designed material, in addition to the social media pages and the website. A brochure and poster have been designed and can already be used for dissemination activities.
BETTER intends to provide the intermediate layers or services able to handle with large volume and heterogeneous datasets to boost the effective use of the different European EO data storage and processing facilities by the wider range research, service provider and other end users communities and maximise the information and knowledge extraction potential behind these datasets.
BETTER will facilitate the handling of large volumes of data and support innovative service creation compatible with the main EO data access and processing infrastructures of NextGEOSS, Copernicus DIAS or the Collaborative Ground Segment infrastructure. The cloud based architecture of the platform in which the pipelines will be developed, the experience of the consortium in developing and leading many of these initiatives (e.g. NextGEOSS and ESA’s TEPs) ensures the compatibility of technologies and standards used for discovery, access and processing and deep integration into this encompassing ecosystem.
BETTER has a specific WP with significant allocated effort dedicated to engaging with other EO and ICT Big Data related initiatives. The focus of its activities will be on establishing active collaborations that go beyond joint communication activities and to actually define and coordinate technical initiatives to achieve mutually beneficial goals. This should lead to the integration of additional data streams, technologies and tools into the BETTER service layer, and eventually in other European and International projects (e.g. Copernicus Integrated Ground Segment (IGS) and Services, ESA’s TEPs, GEO initiatives, flagships and AIPs, and others).