Periodic Reporting for period 2 - EoCoE-II (Energy Oriented Center of Excellence : toward exascale for energy)
Okres sprawozdawczy: 2020-07-01 do 2022-06-30
Europe is undergoing a major transition in its energy generation and supply infrastructure. The recent war in Ukraine has dramatically stressed the critical need for Europe to become much less dependent on fossil energy. The large potential of renewable energies has been largely demonstrated and can no longer be questioned; building a complete, competitive and reliable energy supply chain relying mostly on renewables is a challenge.
Supercomputing is experiencing a major technology shift. Exascale technologies are opening opportunities to tackle complex physical problem but require a collaboration between HPC experts and application specialists.
The conjunction of this two major transitions sparked the idea for the EoCoE consortium gathering institutions involved in both the energy and the HPC domains, combining world-leading research teams from five strategic low-carbon energy domains and leading experts in high-performance computing, applied math and computer sciences.
Modelling capabilities in energy sectors are created at unprecedented scale, demonstrating the benefits to the energy industry, such as accelerated design of storage devices, high-resolution wind and solar forecasting for the power grid and quantitative understanding of plasma core-edge interactions in ITER-scale tokamaks.
Main outcomes of EoCoE-II:
• Empower the scientific challenges with applications, targeting European supercomputers
• Design, develop and test new algorithms, new numerical methods, new software tools on exascale architectures;
• Provide realistic, large scale, test-bed to promote the use of HPC/simulation in the energy field;
• Build a sustainable structure to carry the long-term vision of EoCoE in close collaboration with the European Energy Research Alliance (EERA).
Supercomputing is experiencing a major technology shift. Exascale technologies are opening opportunities to tackle complex physical problem but require a collaboration between HPC experts and application specialists.
The conjunction of this two major transitions sparked the idea for the EoCoE consortium gathering institutions involved in both the energy and the HPC domains, combining world-leading research teams from five strategic low-carbon energy domains and leading experts in high-performance computing, applied math and computer sciences.
Modelling capabilities in energy sectors are created at unprecedented scale, demonstrating the benefits to the energy industry, such as accelerated design of storage devices, high-resolution wind and solar forecasting for the power grid and quantitative understanding of plasma core-edge interactions in ITER-scale tokamaks.
Main outcomes of EoCoE-II:
• Empower the scientific challenges with applications, targeting European supercomputers
• Design, develop and test new algorithms, new numerical methods, new software tools on exascale architectures;
• Provide realistic, large scale, test-bed to promote the use of HPC/simulation in the energy field;
• Build a sustainable structure to carry the long-term vision of EoCoE in close collaboration with the European Energy Research Alliance (EERA).
WP1 – Scientific challenges
Wind:
- Coupling Alya with mesoscale tendencies from WRF
- Two community benchmarks, Hornamossen and Alaiz
- Analisis of extreme weather incidents in Iberdrola wind farms
- Adapt shell elements in Alya to turbine blades
- Fluid Structure simulation of a wind turbine blade
Meteorology:
- Construct probability distribution functions from large ensemble; improving the predictability of cloud and wind
- Development of statistical non-parametric calibration: Asynchronous I/O and compressibility; integration of MELISSA-DA
- Final calibration for ESIAS-met
- Two-step optimization of solar prediction system to calibrate COT input
- Ensemble scoring via cloud-motion and flow structure identification
Materials:
- Interfacing between Wannier90 and the libNEGF code
- Improvement of the scalability of libNEGF towards exascale
- Benchmarking with QMC Reference Calculations
- O(N) kinetic Monte Carlo code KMC-FMM to model hopping type electron conduction in α-NPD, modeling system of over 500k molecules
Water:
- Parflow model performance evaluation for multi-scale processes : lateral groundwater flow, streamflow and water level fluctuations
- Application of HYPERstreamHS, a hydrological model refactored including dual-layer parallelization to improve model computational efficiency
- A mixture of experts surrogate model used to carry out a global sensitivity analysis to classify sources of uncertainty to explain water level variance. The ParFlow-Telemac chained hydrology-hydraulic model was implemented.
- Improved Geothermal Modeling
Fusion:
- Implementation and benchmark of ITER-relevant non-circular magnetic geometries. Benchmark of the electromagnetic model (tearing instability in collisionless regime)
- Study of the plasma-wall interaction physics with kinetic electrons and ions with VOICE, a reduced version of GYSELAX
- Comparative study of the efficiencies of 2-dimensional Poisson solvers
WP2 – Programming models:
- Network of experts: EoCoE, POP, tool providers, PRACE trainers
- Performance evaluation workshops
- Optimization of Alya on CPU and GPU porting using CUDA and OpenACC. Performance comparison with satelite codes WaLBerla, SOWFA and MesoNH.
- Meteo: correction of algorithmic errors in the advection scheme. Optimization of hotspots in the EURAD-IM code. MPI scalability improvement in EURAD-IM
- Performance analysis and bottlenecks identification on the libNEGF code through the JUBE automatic worflow. Optimization of the inelastic self-energies computation based on the work done in PVnegf. Kernel algorithms of libNEGF ported to NVIDIA GPUs (CUDA)
- Adaptive Mesh Refinement implemented in ParFlow using P4est. GPU porting of ParFlow using CUDA and Kokkos. PDI integrated in SHEMAT-Suite.
- Study and programming model exploration to rewrite Gysela using C++. Optimization and adaptation for ARM-based processors (A64FX) in collaboration with ATOS, ARM, Fujitsu & RCCS. Development of the modern C++ Gysela. 2D prototype in modern C++ to prepare GYSELAX rewriting. Strong reduction (~70%) of the computing time with GYSELAX has enabled highly resolved physical studies, leading to publications in high impact factor journals
WP3 – Scalable solvers:
- Integration of PSBLAS/AMG4PSBLAS Krylov solvers and preconditioners to improve ParFlow solver capability
- Extensions of PSBLAS and AMG4PSBLAS to run on hybrid architectures at scale
- Interfacing AGMG to SHEMAT-Suite for improving linear solver capability
- Extension of AGMG to exploit Nvidia GPUs
- Development of multigrid solvers for the gyrokinetic Poisson equation in GyselaX
- Integration of different sparse linear solvers to improve Alya solver capability, to face hybrid (MPI-CUDA) programming models
Integrated solvers are: MUMPS, PaStiX, MaPhyS, AGMG, PSBLAS/AMG4PSBLAS
WP4 – IO & Data Flow:
- PDI improvements: API standardization and unification process, adaption of HDF5 support to new API routines and addition of more HDF5 features, for Gysela and SHEMAT
- Rework of FTI plugin
- New PDI plugin to wrap the NetCDF4 library
- New “User-code” and “pycall” plugin to support individual function calls and in-situ approaches (FlowVR)
- Melissa PDI plugin development
WP5 – Ensemble runs:
- Code release of Melissa for Data Assimilation
- Support of PDAF parallel data assimilation engine into Melissa, and a specific optimized strategy for Particle Filters
- Support of Parflow and WRF for the Weather and Hydro SCs.
- Large scale runs (20K cores) for data assimilation with Parflow and WRF.
- Melissa tested and validated on X86 and Arm-based Fugaku supercomputers.
- Identification of use cases with code and datasets for the Weather and Hydrology applications
- ESIAS/EURAD-IM: WRF integration into Melissa-DA
WP6 – Dissemination & Networking:
- Exploitation strategy
- Training & capacity building
- EoCoE website
- EoCoE Software as a Service portal
- joint Position paper with EERA, launch of a transversal Joint Programme
Wind:
- Coupling Alya with mesoscale tendencies from WRF
- Two community benchmarks, Hornamossen and Alaiz
- Analisis of extreme weather incidents in Iberdrola wind farms
- Adapt shell elements in Alya to turbine blades
- Fluid Structure simulation of a wind turbine blade
Meteorology:
- Construct probability distribution functions from large ensemble; improving the predictability of cloud and wind
- Development of statistical non-parametric calibration: Asynchronous I/O and compressibility; integration of MELISSA-DA
- Final calibration for ESIAS-met
- Two-step optimization of solar prediction system to calibrate COT input
- Ensemble scoring via cloud-motion and flow structure identification
Materials:
- Interfacing between Wannier90 and the libNEGF code
- Improvement of the scalability of libNEGF towards exascale
- Benchmarking with QMC Reference Calculations
- O(N) kinetic Monte Carlo code KMC-FMM to model hopping type electron conduction in α-NPD, modeling system of over 500k molecules
Water:
- Parflow model performance evaluation for multi-scale processes : lateral groundwater flow, streamflow and water level fluctuations
- Application of HYPERstreamHS, a hydrological model refactored including dual-layer parallelization to improve model computational efficiency
- A mixture of experts surrogate model used to carry out a global sensitivity analysis to classify sources of uncertainty to explain water level variance. The ParFlow-Telemac chained hydrology-hydraulic model was implemented.
- Improved Geothermal Modeling
Fusion:
- Implementation and benchmark of ITER-relevant non-circular magnetic geometries. Benchmark of the electromagnetic model (tearing instability in collisionless regime)
- Study of the plasma-wall interaction physics with kinetic electrons and ions with VOICE, a reduced version of GYSELAX
- Comparative study of the efficiencies of 2-dimensional Poisson solvers
WP2 – Programming models:
- Network of experts: EoCoE, POP, tool providers, PRACE trainers
- Performance evaluation workshops
- Optimization of Alya on CPU and GPU porting using CUDA and OpenACC. Performance comparison with satelite codes WaLBerla, SOWFA and MesoNH.
- Meteo: correction of algorithmic errors in the advection scheme. Optimization of hotspots in the EURAD-IM code. MPI scalability improvement in EURAD-IM
- Performance analysis and bottlenecks identification on the libNEGF code through the JUBE automatic worflow. Optimization of the inelastic self-energies computation based on the work done in PVnegf. Kernel algorithms of libNEGF ported to NVIDIA GPUs (CUDA)
- Adaptive Mesh Refinement implemented in ParFlow using P4est. GPU porting of ParFlow using CUDA and Kokkos. PDI integrated in SHEMAT-Suite.
- Study and programming model exploration to rewrite Gysela using C++. Optimization and adaptation for ARM-based processors (A64FX) in collaboration with ATOS, ARM, Fujitsu & RCCS. Development of the modern C++ Gysela. 2D prototype in modern C++ to prepare GYSELAX rewriting. Strong reduction (~70%) of the computing time with GYSELAX has enabled highly resolved physical studies, leading to publications in high impact factor journals
WP3 – Scalable solvers:
- Integration of PSBLAS/AMG4PSBLAS Krylov solvers and preconditioners to improve ParFlow solver capability
- Extensions of PSBLAS and AMG4PSBLAS to run on hybrid architectures at scale
- Interfacing AGMG to SHEMAT-Suite for improving linear solver capability
- Extension of AGMG to exploit Nvidia GPUs
- Development of multigrid solvers for the gyrokinetic Poisson equation in GyselaX
- Integration of different sparse linear solvers to improve Alya solver capability, to face hybrid (MPI-CUDA) programming models
Integrated solvers are: MUMPS, PaStiX, MaPhyS, AGMG, PSBLAS/AMG4PSBLAS
WP4 – IO & Data Flow:
- PDI improvements: API standardization and unification process, adaption of HDF5 support to new API routines and addition of more HDF5 features, for Gysela and SHEMAT
- Rework of FTI plugin
- New PDI plugin to wrap the NetCDF4 library
- New “User-code” and “pycall” plugin to support individual function calls and in-situ approaches (FlowVR)
- Melissa PDI plugin development
WP5 – Ensemble runs:
- Code release of Melissa for Data Assimilation
- Support of PDAF parallel data assimilation engine into Melissa, and a specific optimized strategy for Particle Filters
- Support of Parflow and WRF for the Weather and Hydro SCs.
- Large scale runs (20K cores) for data assimilation with Parflow and WRF.
- Melissa tested and validated on X86 and Arm-based Fugaku supercomputers.
- Identification of use cases with code and datasets for the Weather and Hydrology applications
- ESIAS/EURAD-IM: WRF integration into Melissa-DA
WP6 – Dissemination & Networking:
- Exploitation strategy
- Training & capacity building
- EoCoE website
- EoCoE Software as a Service portal
- joint Position paper with EERA, launch of a transversal Joint Programme
The nature of the project means all EoCoE scientific work goes beyond the state of the art.
The main breakthrough the project is on track to achieve is getting flagship code to the exascale.
This strategy worked, and the results are positive. We can highlight:
- Alya scales up to 100 000 cores
- ESIAS/EURAD-IM scales up to 262 144 cores
- GyselaX scales up to 98 304 cores
- Gysela being ported on IRENE-AMD and Fugaku-ARM
- ParFlow runs on JUWELS AMD with great results, part of GPU Booster programme
- Melissa tested, validated at scale on X86 and Arm Fugaku supercomputers.
Linear Algebra solvers extended and improved to face exascale challenges and exploit capability of GPU accelerators
The main breakthrough the project is on track to achieve is getting flagship code to the exascale.
This strategy worked, and the results are positive. We can highlight:
- Alya scales up to 100 000 cores
- ESIAS/EURAD-IM scales up to 262 144 cores
- GyselaX scales up to 98 304 cores
- Gysela being ported on IRENE-AMD and Fugaku-ARM
- ParFlow runs on JUWELS AMD with great results, part of GPU Booster programme
- Melissa tested, validated at scale on X86 and Arm Fugaku supercomputers.
Linear Algebra solvers extended and improved to face exascale challenges and exploit capability of GPU accelerators