Periodic Reporting for period 2 - C-SCALE (Copernicus - eoSC AnaLytics Engine)
Reporting period: 2022-07-01 to 2023-09-30
Through the provision of massive Earth Observation (EO) data, the EU Copernicus programme has established itself globally as the predominant spatial data provider. Research communities widely use these data to monitor and address global challenges supporting European policy initiatives, such as the Green Deal and others. To facilitate that, the C-SCALE (Copernicus - eoSC AnaLytics Engine) project federated European EO infrastructure services, such as ESA’s Sentinel Collaborative Ground Segment, the Copernicus DIASes (Data and Information Access Services under the EC), independent nationally-funded EO service providers, and European Open Science Cloud (EOSC) e-infrastructure providers, capitalising on EOSC's capacity and capabilities to support Copernicus research and operations. The project implemented and published the C-SCALE Federation in the EOSC Portal as a suite of complementary services that can be easily exploited. It consists of the Data Federation, a service providing access to a large EO data archive, the Compute Federation, and a set of Workflow Solutions. The C-SCALE Data Federation makes EO data providers under EOSC findable, their metadata databases searchable, and their product storage accessible. In addition, C-SCALE integrates heterogeneous resources into a “system of systems” that offers the users an interface that, in most cases, provides similar functionality and quality of service as a centralised, monolithic data archive would. At the same time, the C-SCALE Compute Federation provides access to a wide range of computing providers to enable the analysis of Copernicus and EO data under EOSC. The design of the federation allows users to deploy their applications using federated authentication mechanisms, find their software under a common catalogue, and have access to data using C-SCALE Data Federation tools. The federation relies on existing tools and services already compliant with EOSC. By abstracting the set-up of computing and storage resources from the end-users, it enables the deployment of custom workflows to generate results quickly and easily.
C-SCALE empowers European researchers, institutions and initiatives to easily discover, access, process, analyse and share Copernicus data, tools, resources and services through the EOSC Portal. To this end, C-SCALE delivers a federated compute and data infrastructure offering Copernicus and Earth Observation (EO) data on the EOSC Marketplace, including a seamless user experience where the complexity of resource provisioning and orchestration is abstracted away from the end-user. The service offer of C-SCALE includes three main services:
- The Federated Earth System Simulation and Data Processing Platform (FedEarthData) service brings together the providers of data and processing capacity so that EO products held in distributed archives across the federation can be easily discovered and seamlessly accessed and processed on batch as well as on interactive analytic platforms deployed on distributed computing resources anywhere across the federation.
- The Earth Observation Metadata Query Service (EO-MQS) makes Copernicus data distributed across partners within the federation discoverable and searchable. It is a STAC-compliant API that redistributes incoming queries among the federated sites and provides a consolidated response containing the list of aggregated results. The EO-MQS exposes all STAC collections available within the federation on a single endpoint and provides a search interface that accepts the core parameters of the STAC API Item Search specification.
- The openEO platform service provides intuitive programming libraries alongside a large EO data repository to simplify processing and data management. This large-scale data access and computation is performed on multiple infrastructures allowing use cases from exploratory research to large-scale production of EO-derived maps and information in an accelerated way.
In addition to these services, the project has delivered a set of Workflow Solutions and the C-SCALE Support:
- The Workflow Solutions are easily deployable workflows supporting monitoring, modelling and forecasting of the Earth system. They provide adaptable templates and examples, in the form of Jupyter Notebooks, Copernicus and EO data and analysis workflows enabling users to more easily arrange a processing pipeline to create results on the C-SCALE federation.
- The C-SCALE support includes all the documentation and training material generated for the engagement with existing and new stakeholders, including both researchers and service providers in Earth Observation.
- The Federated Earth System Simulation and Data Processing Platform (FedEarthData) service brings together the providers of data and processing capacity so that EO products held in distributed archives across the federation can be easily discovered and seamlessly accessed and processed on batch as well as on interactive analytic platforms deployed on distributed computing resources anywhere across the federation.
- The Earth Observation Metadata Query Service (EO-MQS) makes Copernicus data distributed across partners within the federation discoverable and searchable. It is a STAC-compliant API that redistributes incoming queries among the federated sites and provides a consolidated response containing the list of aggregated results. The EO-MQS exposes all STAC collections available within the federation on a single endpoint and provides a search interface that accepts the core parameters of the STAC API Item Search specification.
- The openEO platform service provides intuitive programming libraries alongside a large EO data repository to simplify processing and data management. This large-scale data access and computation is performed on multiple infrastructures allowing use cases from exploratory research to large-scale production of EO-derived maps and information in an accelerated way.
In addition to these services, the project has delivered a set of Workflow Solutions and the C-SCALE Support:
- The Workflow Solutions are easily deployable workflows supporting monitoring, modelling and forecasting of the Earth system. They provide adaptable templates and examples, in the form of Jupyter Notebooks, Copernicus and EO data and analysis workflows enabling users to more easily arrange a processing pipeline to create results on the C-SCALE federation.
- The C-SCALE support includes all the documentation and training material generated for the engagement with existing and new stakeholders, including both researchers and service providers in Earth Observation.
Throughout the course of 33 months, the project's consortium has successfully fulfilled its objectives, revolutionising the processing and analysis of Copernicus data. C-SCALE created a Big Copernicus Data Analytics platform through federating European data and compute providers and enabled and supported numerous researchers across Europe to access and leverage Copernicus data. More specifically, the C-SCALE federation has been established including 8 Cloud, HPC/HTC compute and data providers. Three new services have been onboarded into the EOSC Marketplace and four main workflows are also available from Zenodo, RoHub and therefore EOSC. In addition, C-SCALE has enabled a common interface for Copernicus/EO datasets and aligned them with other services that are available via the EOSC Marketplace. Merging these previously disconnected service markets and pooling of services and resources on a pan-European scale bring considerable benefits through network effects (and innovation catalysed by it) and economies of scale factors. For example, through the collaboration with EOSC DIH access European-based SMEs were also able to access the services.
Overall, C-SCALE has created impact in three different domains:
- Science, by creating high-quality new knowledge, having actively contributed to over 55 dissemination activities and publications in peer-reviewed journals, strengthening human capital in Research, having organized 11 workshops attended by more than 100 people followed by hundreds of online views, and Innovation and fostering the diffusion of knowledge and Open Science. As such, all materials created during the project, including documentation, workflows, and software, are openly available under permissive licenses (i.e. published under CC-BY when possible) and FAIRNESS of data has been a centre point of the project.
- Society, by Addressing EU policy priorities & global challenges through Research & Innovation, delivering benefits & impact via Research & Innovation missions, delivering benefits & impact via Research & Innovation missions. As such, the 21 use cases performed acting as a proof-of-concept for C-SCALE services, addressed socially relevant topics such as climate change effects, understanding of coastal ocean dynamics and water quality or forecasting water resource management.
- Economy, by generating innovation-based growth, leveraging investments in Research & Innovation and creating more and better jobs, having engaged, 8 SMEs and over 80 experts (among researchers, engineers and support teams) and activities continued under several follow-up projects such as openEO Platform, INTERFACE or interTwin projects.
Overall, C-SCALE has created impact in three different domains:
- Science, by creating high-quality new knowledge, having actively contributed to over 55 dissemination activities and publications in peer-reviewed journals, strengthening human capital in Research, having organized 11 workshops attended by more than 100 people followed by hundreds of online views, and Innovation and fostering the diffusion of knowledge and Open Science. As such, all materials created during the project, including documentation, workflows, and software, are openly available under permissive licenses (i.e. published under CC-BY when possible) and FAIRNESS of data has been a centre point of the project.
- Society, by Addressing EU policy priorities & global challenges through Research & Innovation, delivering benefits & impact via Research & Innovation missions, delivering benefits & impact via Research & Innovation missions. As such, the 21 use cases performed acting as a proof-of-concept for C-SCALE services, addressed socially relevant topics such as climate change effects, understanding of coastal ocean dynamics and water quality or forecasting water resource management.
- Economy, by generating innovation-based growth, leveraging investments in Research & Innovation and creating more and better jobs, having engaged, 8 SMEs and over 80 experts (among researchers, engineers and support teams) and activities continued under several follow-up projects such as openEO Platform, INTERFACE or interTwin projects.