Periodic Reporting for period 1 - SPECTRUM (Computing Strategy for Data-intensive Science Infrastructures in Europe)
Periodo di rendicontazione: 2024-01-01 al 2025-03-31
The amount of data gathered, shared, and processed in frontier research is set to increase steeply in the coming decade, leading to unprecedented data processing, simulation, and analysis needs.
However, the current data and compute infrastructures are not equipped to handle this deluge of data. The research communities in HEP and RA are preparing to launch new instruments that require data and compute infrastructures several orders of magnitude larger than what is currently available, entering the Exascale era. The limitations of the current infrastructures pose significant challenges, including bottlenecks in data processing and analysis, which could hinder scientific discoveries.
This project aims to deliver a Strategic Research, Innovation and Deployment Agenda (SRIDA) and a Technical Blueprint for a European compute and data continuum.
By paving the way towards data-intensive scientific collaborations with access to a federated European Exabyte-scale research data federation and compute continuum, the project will enable researchers to process and analyze data more efficiently, leading to faster and more accurate scientific discoveries. Moreover, the project will contribute to Europe's position in the global research landscape, ensuring that European researchers have access to the most advanced data and compute infrastructures.
However, the current data and compute infrastructures are not equipped to handle this deluge of data. The research communities in HEP and RA are preparing to launch new instruments that require data and compute infrastructures several orders of magnitude larger than what is currently available, entering the Exascale era. The limitations of the current infrastructures pose significant challenges, including bottlenecks in data processing and analysis, which could hinder scientific discoveries.
This project aims to deliver a Strategic Research, Innovation and Deployment Agenda (SRIDA) and a Technical Blueprint for a European compute and data continuum.
By paving the way towards data-intensive scientific collaborations with access to a federated European Exabyte-scale research data federation and compute continuum, the project will enable researchers to process and analyze data more efficiently, leading to faster and more accurate scientific discoveries. Moreover, the project will contribute to Europe's position in the global research landscape, ensuring that European researchers have access to the most advanced data and compute infrastructures.
The main technical achievements are carried out within WP3 “Community of Practice Creation and 1st Consultation” and WP5 “Landscape, use cases, challenges and gaps”.
In WP3, the Community of Practice (CoP) was established, where different Working Groups were defined to streamline discussions. D3.1 was published including the results from the survey distributed to collect needs and ideas bottom up directly from the communities, a review of the Knowledge Hub, and trends and directions as extrapolated from all the collected material.
WP5 has been focused on identifying and analyzing representative scientific use cases from diverse RIs, studying the access policy recommendations, and analysing the landscape, best practices and gaps.
Deliverable 5.1 (D5.1) includes an analysis of requirements for 14 use cases with a focus on the HEP and RA, and a cross-cutting analysis of common themes and trends. The key findings identify a significant need to greatly expand the compute capabilities of the fields to process unprecedented volumes of data.
D5.2 documents the analysis of access policies of 20 e-Infrastructures. It compares and contrasts them with the requirements of research communities with a focus on the HEP and RA fields and the use cases described in D5.1,and provides recommendations for the future evolution of the access policies to meet the current and future needs of researchers.
D5.3 contains the results of a study of the technical characteristics of 20 e-Infrastructures which serve research communities and data-intensive use cases with significant computing requirements. The studied technical characteristics are compared to research community requirements, and recommendations formulated for the future evolution and improvement of European e-Infrastructures to meet the scientific user needs.
WP3 and 5 have provided both the community-driven context and the technical landscape analysis to inform the next stages for creating a blueprint and SRIDA for future e-Infrastructure development.
In WP3, the Community of Practice (CoP) was established, where different Working Groups were defined to streamline discussions. D3.1 was published including the results from the survey distributed to collect needs and ideas bottom up directly from the communities, a review of the Knowledge Hub, and trends and directions as extrapolated from all the collected material.
WP5 has been focused on identifying and analyzing representative scientific use cases from diverse RIs, studying the access policy recommendations, and analysing the landscape, best practices and gaps.
Deliverable 5.1 (D5.1) includes an analysis of requirements for 14 use cases with a focus on the HEP and RA, and a cross-cutting analysis of common themes and trends. The key findings identify a significant need to greatly expand the compute capabilities of the fields to process unprecedented volumes of data.
D5.2 documents the analysis of access policies of 20 e-Infrastructures. It compares and contrasts them with the requirements of research communities with a focus on the HEP and RA fields and the use cases described in D5.1,and provides recommendations for the future evolution of the access policies to meet the current and future needs of researchers.
D5.3 contains the results of a study of the technical characteristics of 20 e-Infrastructures which serve research communities and data-intensive use cases with significant computing requirements. The studied technical characteristics are compared to research community requirements, and recommendations formulated for the future evolution and improvement of European e-Infrastructures to meet the scientific user needs.
WP3 and 5 have provided both the community-driven context and the technical landscape analysis to inform the next stages for creating a blueprint and SRIDA for future e-Infrastructure development.
The establishment of the CoP (KER1) and the early deployment of the Knowledge Hub have created a collaborative foundation for sharing tools, methodologies, and technical solutions across scientific domains. It also joined efforts with the JENA Computing Initiative to conduct a survey to understand the current best practices and future needs in large-scale and data-intensive scientific computing in HEP and RA domains. The results of the survey are in D3.1 and have contributed to the European Strategy for Particle Physics.
The Compendium of Use Cases (KER2) covers the future compute needs in HEP, RA and other domains. Together with the Landscape Analysis (KER3) and the Access Policies compilation (KER4) in D5.2 and D5.3 they represent the top-down approach vision of gaps and future needs for the most relevant scientific experiments in the next decade.
The analysis of the state of play paves the way for future disruptive research and breakthrough science and innovation through cutting-edge, interconnected and sustainable RIs able to reinforce the scientific excellence and performance and efficiency of the ERA.
Continued research and technical work are required to translate the identified requirements into functional architecture and prototypes, which are currently being developed within WPs 6 and 7 for the generation of the Technical Blueprint Architecture (KER5) and the SRIDA (KER6), the roadmap documents that will be delivered at the end of the project. They have the potential to foster an optimized coordinated research infrastructure capacity among countries and regions while re-inforcing European R&D+i capacities.
Adoption of recommendations will enable capturing the scientific discovery of next generation experiments in RA and HEP communities powered by the necessary high-end computing and data infrastructure to tackle the challenges of HL-HLC, SKAO and other major endeavours. The consolidation of digital services and infrastructure needed for data-intensive science will offer new opportunities to players in all the supply chain, leading to the consolidation of the European advanced computing sector.
The Compendium of Use Cases (KER2) covers the future compute needs in HEP, RA and other domains. Together with the Landscape Analysis (KER3) and the Access Policies compilation (KER4) in D5.2 and D5.3 they represent the top-down approach vision of gaps and future needs for the most relevant scientific experiments in the next decade.
The analysis of the state of play paves the way for future disruptive research and breakthrough science and innovation through cutting-edge, interconnected and sustainable RIs able to reinforce the scientific excellence and performance and efficiency of the ERA.
Continued research and technical work are required to translate the identified requirements into functional architecture and prototypes, which are currently being developed within WPs 6 and 7 for the generation of the Technical Blueprint Architecture (KER5) and the SRIDA (KER6), the roadmap documents that will be delivered at the end of the project. They have the potential to foster an optimized coordinated research infrastructure capacity among countries and regions while re-inforcing European R&D+i capacities.
Adoption of recommendations will enable capturing the scientific discovery of next generation experiments in RA and HEP communities powered by the necessary high-end computing and data infrastructure to tackle the challenges of HL-HLC, SKAO and other major endeavours. The consolidation of digital services and infrastructure needed for data-intensive science will offer new opportunities to players in all the supply chain, leading to the consolidation of the European advanced computing sector.