Computing Strategy for Data-intensive Science Infrastructures in Europe

Project Information

SPECTRUM

Grant agreement ID: 101131550

DOI

10.3030/101131550

Project closed

EC signature date 29 September 2023

Start date 1 January 2024

End date 30 June 2026

Funded under

• Research infrastructures

Total cost

No data

EU contribution

€ 2 499 542,50

Coordinated by

STICHTING EGI
Netherlands

Periodic Reporting for period 1 - SPECTRUM (Computing Strategy for Data-intensive Science Infrastructures in Europe)

Reporting period: 2024-01-01 to 2025-03-31

Summary of the context and overall objectives of the project

The amount of data gathered, shared, and processed in frontier research is set to increase steeply in the coming decade, leading to unprecedented data processing, simulation, and analysis needs.
However, the current data and compute infrastructures are not equipped to handle this deluge of data. The research communities in HEP and RA are preparing to launch new instruments that require data and compute infrastructures several orders of magnitude larger than what is currently available, entering the Exascale era. The limitations of the current infrastructures pose significant challenges, including bottlenecks in data processing and analysis, which could hinder scientific discoveries.

This project aims to deliver a Strategic Research, Innovation and Deployment Agenda (SRIDA) and a Technical Blueprint for a European compute and data continuum.

By paving the way towards data-intensive scientific collaborations with access to a federated European Exabyte-scale research data federation and compute continuum, the project will enable researchers to process and analyze data more efficiently, leading to faster and more accurate scientific discoveries. Moreover, the project will contribute to Europe's position in the global research landscape, ensuring that European researchers have access to the most advanced data and compute infrastructures.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The main technical achievements are carried out within WP3 “Community of Practice Creation and 1st Consultation” and WP5 “Landscape, use cases, challenges and gaps”.

In WP3, the Community of Practice (CoP) was established, where different Working Groups were defined to streamline discussions. D3.1 was published including the results from the survey distributed to collect needs and ideas bottom up directly from the communities, a review of the Knowledge Hub, and trends and directions as extrapolated from all the collected material.

WP5 has been focused on identifying and analyzing representative scientific use cases from diverse RIs, studying the access policy recommendations, and analysing the landscape, best practices and gaps.

Deliverable 5.1 (D5.1) includes an analysis of requirements for 14 use cases with a focus on the HEP and RA, and a cross-cutting analysis of common themes and trends. The key findings identify a significant need to greatly expand the compute capabilities of the fields to process unprecedented volumes of data.

D5.2 documents the analysis of access policies of 20 e-Infrastructures. It compares and contrasts them with the requirements of research communities with a focus on the HEP and RA fields and the use cases described in D5.1,and provides recommendations for the future evolution of the access policies to meet the current and future needs of researchers.

D5.3 contains the results of a study of the technical characteristics of 20 e-Infrastructures which serve research communities and data-intensive use cases with significant computing requirements. The studied technical characteristics are compared to research community requirements, and recommendations formulated for the future evolution and improvement of European e-Infrastructures to meet the scientific user needs.

WP3 and 5 have provided both the community-driven context and the technical landscape analysis to inform the next stages for creating a blueprint and SRIDA for future e-Infrastructure development.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The establishment of the CoP (KER1) and the early deployment of the Knowledge Hub have created a collaborative foundation for sharing tools, methodologies, and technical solutions across scientific domains. It also joined efforts with the JENA Computing Initiative to conduct a survey to understand the current best practices and future needs in large-scale and data-intensive scientific computing in HEP and RA domains. The results of the survey are in D3.1 and have contributed to the European Strategy for Particle Physics.

The Compendium of Use Cases (KER2) covers the future compute needs in HEP, RA and other domains. Together with the Landscape Analysis (KER3) and the Access Policies compilation (KER4) in D5.2 and D5.3 they represent the top-down approach vision of gaps and future needs for the most relevant scientific experiments in the next decade.

The analysis of the state of play paves the way for future disruptive research and breakthrough science and innovation through cutting-edge, interconnected and sustainable RIs able to reinforce the scientific excellence and performance and efficiency of the ERA.

Continued research and technical work are required to translate the identified requirements into functional architecture and prototypes, which are currently being developed within WPs 6 and 7 for the generation of the Technical Blueprint Architecture (KER5) and the SRIDA (KER6), the roadmap documents that will be delivered at the end of the project. They have the potential to foster an optimized coordinated research infrastructure capacity among countries and regions while re-inforcing European R&D+i capacities.

Adoption of recommendations will enable capturing the scientific discovery of next generation experiments in RA and HEP communities powered by the necessary high-end computing and data infrastructure to tackle the challenges of HL-HLC, SKAO and other major endeavours. The consolidation of digital services and infrastructure needed for data-intensive science will offer new opportunities to players in all the supply chain, leading to the consolidation of the European advanced computing sector.