Periodic Reporting for period 1 - ChEESE-2P (Center of Excellence for Exascale in Solid Earth - Second Phase)
Reporting period: 2023-01-01 to 2023-12-31
Problem statement. Solid Earth (SE) is extremely rich in computational challenges, requiring petascale and exascale infrastructures both to address fundamental scientific questions and to anticipate, mitigate, and manage the occurrence of geohazards and their impacts by advancing on, at least, three fronts:
- Capability computing. First of all, on explicitly solving problems that, traditionally, have been parameterised because at present are unaffordable in terms of size of the computational domain, model resolution, complex physical couplings, and huge amounts of data involved (Scientific Grand Challenges,SGC).
- Capacity computing. Second, by introducing a probabilistic framework through a large ensemble of model realisations, which allows high-end data inversion, model data assimilation, and uncertainty quantification (e.g. for physics-based probabilistic hazard assessments). These capacity workloads can range from tightly coupled, as in the case of ensemble-based data assimilation schemes (e.g. Kalman filters), to loosely coupled as in realisations of hazard scenarios.
- Urgent computing. Finally, on solving currently affordable problems but with much less computing time. This aspect is critical in scenarios of early warning and emergency management, where constrains exist in urgency (e.g. faster than real time tsunami simulations).
Overall objectives. ChEESE has 4 general objectives and 15 specific objectives related to scientific, technical, and socio-economic challenges (see Part B for details). First of all, the project is preparing 11 European flagship Community codes for the upcoming Exascale Era in order to address 12 underpinning scientific, technical, and socio-economic Exascale Computational Challenges (ECC) in the areas of computational seismology, magneto-hydrodynamics, physical volcanology, tsunami modelling, geodynamics and glaciar modelling. During the project, codes will be optimised in terms of performance on different types of accelerators, code scalability, deployment, availability on EuroHPC systems following a CI/CD approach, performance portability across current pre-exascale systems and by co-designing with 7 mini-apps, also on hardware architectures emerging from EuroHPC Pilots. Codes and related workflows are combined to implement 9 Pilot Demonstrators (PDs), intended as proofs of concept to address each of the 12 ECCs. The PDs will materialise in 15 Simulation Cases (PCs) representing capability and capacity use cases of particular interest in terms of science, social relevance, or urgency. The capability-oriented use-cases entail also 4 Scientific Grand Challenges (SGC) that will produce relevant EOSC-enabled datasets and prototype services on aspects of geohazards like UC, Early Warning Systems (EWS), forecasts, and hazard assessments. Last, but not least, ChEESE aims at becoming a hub to integrate different actors concerned with different aspects of the SE phenomena. This integration implies domain-specific training, capacity building, dissemination, market analysis for future exploitation of services and, very important, engage with the European Plate Observing System (EPOS) and longer-term mission-like initiatives like the Digital Agenda and Destination Earth.
- Capability computing. First of all, on explicitly solving problems that, traditionally, have been parameterised because at present are unaffordable in terms of size of the computational domain, model resolution, complex physical couplings, and huge amounts of data involved (Scientific Grand Challenges,SGC).
- Capacity computing. Second, by introducing a probabilistic framework through a large ensemble of model realisations, which allows high-end data inversion, model data assimilation, and uncertainty quantification (e.g. for physics-based probabilistic hazard assessments). These capacity workloads can range from tightly coupled, as in the case of ensemble-based data assimilation schemes (e.g. Kalman filters), to loosely coupled as in realisations of hazard scenarios.
- Urgent computing. Finally, on solving currently affordable problems but with much less computing time. This aspect is critical in scenarios of early warning and emergency management, where constrains exist in urgency (e.g. faster than real time tsunami simulations).
Overall objectives. ChEESE has 4 general objectives and 15 specific objectives related to scientific, technical, and socio-economic challenges (see Part B for details). First of all, the project is preparing 11 European flagship Community codes for the upcoming Exascale Era in order to address 12 underpinning scientific, technical, and socio-economic Exascale Computational Challenges (ECC) in the areas of computational seismology, magneto-hydrodynamics, physical volcanology, tsunami modelling, geodynamics and glaciar modelling. During the project, codes will be optimised in terms of performance on different types of accelerators, code scalability, deployment, availability on EuroHPC systems following a CI/CD approach, performance portability across current pre-exascale systems and by co-designing with 7 mini-apps, also on hardware architectures emerging from EuroHPC Pilots. Codes and related workflows are combined to implement 9 Pilot Demonstrators (PDs), intended as proofs of concept to address each of the 12 ECCs. The PDs will materialise in 15 Simulation Cases (PCs) representing capability and capacity use cases of particular interest in terms of science, social relevance, or urgency. The capability-oriented use-cases entail also 4 Scientific Grand Challenges (SGC) that will produce relevant EOSC-enabled datasets and prototype services on aspects of geohazards like UC, Early Warning Systems (EWS), forecasts, and hazard assessments. Last, but not least, ChEESE aims at becoming a hub to integrate different actors concerned with different aspects of the SE phenomena. This integration implies domain-specific training, capacity building, dissemination, market analysis for future exploitation of services and, very important, engage with the European Plate Observing System (EPOS) and longer-term mission-like initiatives like the Digital Agenda and Destination Earth.
During the firsts year, a first code audit campaign has been done in order to: (1) identify code bottlenecks and, (2) set a scalability and performance baseline based on POP3 hybrid metrics to monitor code evolution and quantitatively document progress after the second audit round (year 3). Based on the audit results, WP2 created an optimisation plan tailored to each code (T2.3) and WP4 did an early collection of PD workflow requirements (D4.1 M18). Dedicated code optimisation and tuning actions will be done on single (heterogeneous) node, multi-node scaling, running of kernels on different accelerators (NVIDIA/AMD), and I/O. The 11 codes were made available on accessible git-based repositories and obtained an EOSC-synergy quality assessment certified badge (1 Gold, 7 Silver, 3 Bronze). Repositories were also created for the 7 mini-apps (all accelerated) and for tests and benchmarks, including the necessary metadata for traceability, reproducibility and provenance. Mini-apps will be used, under the lead of BULL-ATOS, for testing on different typologies of emerging hardware (EUPEX and EuPilot). The EPI project has already included the Mini-app and SPECFEM3D (full application) in their public co-design benchmark suite. As a starting point, the first year focussed on benchmarks and development access modes with success: 2 benchmark and 12 development access adding 110,000 node/h (LUMI, Leonardo, and VEGA) plus other 5 National-level projects. In the next phases ChEESE will focus on regular and extreme-scale modes. On the other hand, WP5 started the implementation of novel features in flagship codes and performed some early developments in some of the PDs (PD1, PD3, PD4). A Data Management Plan (DMP; D1.2) has been defined to ensure a FAIR management of data and metadata in all the PDs and is under permanent update. Finally, the definition of scenarios and the roadmap for running the 15 SCs (including 4 Grand Challenges) has been established.
There has been a strong collaboration with CASTIEL2 CSA through: (i) collaboration agreement with the CoEs (COLA, currently v10.1 (ii) contribution to 5 CASTIEL2 deliverables, (iii) contribution to the CI/CD on EuroHPC systems through regular CASTIEL2 CI/CD meetings, pairing activities between CoEs and EuroHPC systems (e.g. T-HySEA code pipelines on the VEGA system) and code sharing (mirroring) to the centralised repository, (iv) participation in the "code-of-the-month" initiative (Elmer/ICE code), (v) participation in the Competence Centre and CoE Advisory Board (CAB), (vi) participation in the monthly Training Coffee break meetings, and (vii) participation in the hyper-connectivity study of the EuroHyPerCon project. In terms of capacity building, dissemination, and outreach, ChEESE has organised the 11th Galileo Conference (May 2023) with the corresponding consensual document (roadmap), 5 webinars and the first training geosciences series (of 7 foreseen). The composition of the IUB has been redefined retaining 25 members only (of which 10 signed NDAs). The Key Performance Indicators (KPIs) for scientific and social impacts of the project are evolving very satisfactorily.
There has been a strong collaboration with CASTIEL2 CSA through: (i) collaboration agreement with the CoEs (COLA, currently v10.1 (ii) contribution to 5 CASTIEL2 deliverables, (iii) contribution to the CI/CD on EuroHPC systems through regular CASTIEL2 CI/CD meetings, pairing activities between CoEs and EuroHPC systems (e.g. T-HySEA code pipelines on the VEGA system) and code sharing (mirroring) to the centralised repository, (iv) participation in the "code-of-the-month" initiative (Elmer/ICE code), (v) participation in the Competence Centre and CoE Advisory Board (CAB), (vi) participation in the monthly Training Coffee break meetings, and (vii) participation in the hyper-connectivity study of the EuroHyPerCon project. In terms of capacity building, dissemination, and outreach, ChEESE has organised the 11th Galileo Conference (May 2023) with the corresponding consensual document (roadmap), 5 webinars and the first training geosciences series (of 7 foreseen). The composition of the IUB has been redefined retaining 25 members only (of which 10 signed NDAs). The Key Performance Indicators (KPIs) for scientific and social impacts of the project are evolving very satisfactorily.
ChEESE is progressing on approaching the SE Community towards Exascale, tackling societal georisks with an unprecedented approach and acting as a pole of attraction to gather actors with very different backgrounds around HPC/HPDA. ChEESE-1P was very successful in fostering and leveraging other projects and initiatives that have benefited from and will take over and sustain part of the project developments (eFlows4HPC, Geo-INQUIRE, DT-GEO). The flourishing of this “ecosystem” is a good indicator of the success that the ChEESE-1P had in federating the SE community around HPC, and also guarantees the sustainability of key aspects of the project (notably, the access to codes, workflow components and services). For ChEESE-2P, it is too early yet to quantify results, but we expect progress in terms of innovative solutions and social impact and in terms of new products and service enabling on EuroHPC machines.