Periodic Reporting for period 3 - ExaHyPE (An Exascale Hyperbolic PDE Engine)
Reporting period: 2018-04-01 to 2019-09-30
To address the challenges of exascale, ExaHyPE developed a novel, communication-avoiding high-order discontinuous-Galerkin scheme with high-order time stepping. At shocks and critical regions, the solver falls back to a robust Finite-Volume limiter. We feature tree-structured dynamical adaptive mesh refinement and express all numerical steps via task-based paradigms. We developed agile load balancing via lightweight work stealing across compute nodes and explored replication-based resiliency and the impact of a-posteriori limiting on detection and mitigation of soft faults.
As one of two demonstrators, we provide ExaSeis, for seismology, which includes (1) a linear model on curvilinear meshes, where the mesh is adapted to topography, material layers or fault planes, and (2) a novel non-linear diffused-interface model that expresses arbitrarily complex domains on Cartesian grids via a color function. As a second demonstrator, we provide solvers for the Einstein field equations of general relativity in first-order CCZ4 formulation, coupled to the general relativistic MHD equations. We thus enable relativistic simulation of black holes and neutron stars, which are being further developed towards mergers of binary systems.
In 2016 and 2017, the engine prototype was extended to a fully functional engine for solving hyperbolic PDE systems. An Application Specific Programming Interface (ASPI) was developed that allows users to easily express specific systems of PDEs via “user functions”. The engine implements linear and non-linear ADER-DG schemes on dynamically adaptive Cartesian-structured meshes. All schemes are parallelised using MPI and Intel Threading Building Blocks as programming model.
For the astrophysics demonstrator, we developed a novel, provably strongly hyperbolic first-order reduction of the conformal and covariant Z4 formulation of the Einstein field equations (FO-CCZ4). For seismology, we defined and prototypically implemented three grand-challenge use-cases based on curvilinear and diffused interface models, both avoiding major bottleneck in many applications of computational seismology: the need to resolve topography and sharp material interfaces via feature-preserving meshes.
Results of 2018 and 2019 include the further development of ExaHyPE's Toolkit and Kernel Generator into a role-oriented code generator with separation of concerns for application, algorithms and optimisation experts. We realised lightweight task-offloading for reactive load balancing across compute nodes. This is complemented by team-based replication of MPI ranks (based on a tailored infrastructure, teaMPI) to improve resilience of capability simulations.
We developed a new numerical approach to discretise curl-type involutions to treat instabilities arising in the coupled FO-CCZ4+GRMHD models. The added curl-cleaning terms increase the size of the PDE system to more than 100 quantities. Work on the Godunov-Peshkow-Romenski model allows to model linear and nonlinear elasto-plastic solids with crack formation, as well as viscous and ideal fluids at the same time within one unified governing PDE system. Alongside the diffused interface model for seismology, we developed a diffuse interface method for compressible flows over fixed and moving solids, which substantially widens the range of ExaHyPE applications towards gas dynamics in complex geometries and fluid-structure-interaction problems.
The recent detection of gravitational waves from the merger of binary black holes and neutron stars (the latter with an electromagnetic counterpart) motivates highly accurate simulations. Simulation tools that cover the full range of physics, i.e. fully relativistic modeling, magnetic fields and viscous hydrodynamics, are not yet available. Existing state-of-the-art simulation software only offers a subset of models and only provides limited adaptive mesh refinement.
The new seismic wave propagation models allow to simulate scenarios with highly complex geometries such as numerical scenarios for the AlpArray, Europe’s currently largest seismology experiment. The ExaSeis code, developed on the basis of ExaHyPE will be further developed as part of the ChEESE Centre of Excellence funded via Horizon 2020.