Periodic Reporting for period 3 - ESiWACE (Excellence in SImulation of Weather and Climate in Europe)
Período documentado: 2018-09-01 hasta 2019-08-31
Numerical weather prediction and climate modelling are highly dependent on the available computing power and the ability to produce, store and analyse large amounts of simulated data. All of these points directly affect the achievable spatial resolution and the completeness and accuracy of physical processes that can be calculated and predicted by weather and climate models. The higher the resolution – i.e. the smaller the distance between the points of the computational grid - the more physical processes can be directly resolved and the higher is the fidelity of the achieved predictions. The steady increase in computational resources has pushed the resolution of global weather and climate simulations, but for operational use it is still limited to O(10km) for weather forecasts and to an even wider grid spacing for climate simulations, i.e. for practical purposes Earth is covered by a grid of 10-100km wide meshes in the simulations. The goal of ESiWACE was to investigate and leverage the opportunities for weather and climate modelling offered by upcoming generations of supercomputers. In particular, global 1km uncoupled ocean and atmosphere simulations as well as coupled 10km ocean-atmosphere simulations were developed to investigate the feasibility of such high-resolution models on exascale supercomputers. The target resolution of 1km is expected to - for the first time - resolve deep moist convection at global scale, which is an essential driver for vertical energy transport in the tropics and governs clouds and precipitation. The development and deployment of these high-resolution modells entails a variety of related challenges. Improved workflow solutions are required, the enormous amount of simulation data (which may easily exceed petabytes) needs to be handled, and staff needs to be trained to program and maintain the models, keeping in mind the increasing hardware complexity at exascale. All this makes global high-resolution modelling a community effort, invoking weather and climate scientists, computer scientists, HPC vendors and related industry.
ESiWACE has demonstrated and improved scalability of some of Europe’s leading weather and climate models on world class HPC systems and is prototyping new tools and methods for management of climate and weather data in the exascale era. However, we are still far from being able to run a 1 km model in operational mode even if some of the fastest supercomputers that are currently available are used. The necessary steps to achieve this goal have been documented in the ESiWACE sustainability plan and a roadmap. The second phase of the project, ESiWACE2, will continue the work and follow up on the roadmap.
ESiWACE has demonstrated and improved scalability of some of Europe’s leading weather and climate models on world class HPC systems and is prototyping new tools and methods for management of climate and weather data in the exascale era. However, we are still far from being able to run a 1 km model in operational mode even if some of the fastest supercomputers that are currently available are used. The necessary steps to achieve this goal have been documented in the ESiWACE sustainability plan and a roadmap. The second phase of the project, ESiWACE2, will continue the work and follow up on the roadmap.
ESiWACE dealt with three central aspects of very high-resolution climate and weather simulations: Scalability, Usability and Exploitability.
1) Scalability: Global High-Resolution Demonstrators
Very high-resolution simulations were successfully developed based on the atmosphere model IFS, ICON and the ocean model NEMO. The target resolution of 1km was reached. Analogously coupled 10km-resolving simulations based on ICON and EC-Earth were established and successfully run on some of the largest supercomputers in Europe; IFS could even be run on Summit, the fastest machine in the world. The results suggest that simulations at 1km are feasible; yet, there is still an enormous gap between achieved throughput in terms of simulated years per day and the target throughput rate of 1 simulated year per day, which is the minimal requirement for most of the relevant science cases. This calls for significant continued efforts in terms of mathematical, algorithmic, and software development.
Yet, the ESiWACE efforts have already created valuable synergies with the science case. In late 2017, the project DYAMOND was launched. DYAMOND stands for DYnamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains and targets the inter-comparison of international global models, running at highest affordable resolution. Both atmosphere-only models IFS and ICON, participated in DYAMOND, and ESiWACE organised two hackathons to bring together international leading experts in the field and to leverage synergies on DYAMOND data evaluation.
2) Usability: Handbooks, Trainings, Workflows, Software Management
To improve the overall workflows required by the weather and climate models, several developments were carried out in ESiWACE. The workflow engine Cylc was supported and improved to provide a robust and scalable solution meeting the community needs. Furthermore, data analytics workflows were advanced based on the analytics workflow manager Ophidia, including the extension of interfaces and capabilities for analytics workflows, HPC scheduling and parallel file system integration.
To prepare the community and supercomputer administration for weather and climate prediction at exascale, handbooks on the software stacks required by advanced weather and climate simulations were established. The program package manager SPACK was extended to support various software packages relevant to weather and climate modelling, which facilitates software installation procedures on varying supercomputing platforms and, thus, reduces system administration efforts.
Further, various workshops, trainings and user support were organised and supported through ESiWACE, including training sessions for the coupling software OASIS and the XIOS I/O software, user support for NEMO, EC-Earth and Cylc, and two HPC workshops.
3) Exploitability: Coping with the Data Avalanche
To understand the requirement for exascale storage systems and determine beneficial variants, various business models for storage systems and corresponding infrastructures were investigated. A major focus was put on the development of the Earth System Data Middleware (ESDM). This software is an interface between storage hardware and the data descriptions used by Earth system scientists. In particular ESDM enables the use of multiple storage backends in parallel – a promising achievement for handling future big data in weather and climate predictions.
The evaluation of the ESDM performance was conducted in large benchmark runs and demonstrated the capability to handle high-resolution data in future production runs.
1) Scalability: Global High-Resolution Demonstrators
Very high-resolution simulations were successfully developed based on the atmosphere model IFS, ICON and the ocean model NEMO. The target resolution of 1km was reached. Analogously coupled 10km-resolving simulations based on ICON and EC-Earth were established and successfully run on some of the largest supercomputers in Europe; IFS could even be run on Summit, the fastest machine in the world. The results suggest that simulations at 1km are feasible; yet, there is still an enormous gap between achieved throughput in terms of simulated years per day and the target throughput rate of 1 simulated year per day, which is the minimal requirement for most of the relevant science cases. This calls for significant continued efforts in terms of mathematical, algorithmic, and software development.
Yet, the ESiWACE efforts have already created valuable synergies with the science case. In late 2017, the project DYAMOND was launched. DYAMOND stands for DYnamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains and targets the inter-comparison of international global models, running at highest affordable resolution. Both atmosphere-only models IFS and ICON, participated in DYAMOND, and ESiWACE organised two hackathons to bring together international leading experts in the field and to leverage synergies on DYAMOND data evaluation.
2) Usability: Handbooks, Trainings, Workflows, Software Management
To improve the overall workflows required by the weather and climate models, several developments were carried out in ESiWACE. The workflow engine Cylc was supported and improved to provide a robust and scalable solution meeting the community needs. Furthermore, data analytics workflows were advanced based on the analytics workflow manager Ophidia, including the extension of interfaces and capabilities for analytics workflows, HPC scheduling and parallel file system integration.
To prepare the community and supercomputer administration for weather and climate prediction at exascale, handbooks on the software stacks required by advanced weather and climate simulations were established. The program package manager SPACK was extended to support various software packages relevant to weather and climate modelling, which facilitates software installation procedures on varying supercomputing platforms and, thus, reduces system administration efforts.
Further, various workshops, trainings and user support were organised and supported through ESiWACE, including training sessions for the coupling software OASIS and the XIOS I/O software, user support for NEMO, EC-Earth and Cylc, and two HPC workshops.
3) Exploitability: Coping with the Data Avalanche
To understand the requirement for exascale storage systems and determine beneficial variants, various business models for storage systems and corresponding infrastructures were investigated. A major focus was put on the development of the Earth System Data Middleware (ESDM). This software is an interface between storage hardware and the data descriptions used by Earth system scientists. In particular ESDM enables the use of multiple storage backends in parallel – a promising achievement for handling future big data in weather and climate predictions.
The evaluation of the ESDM performance was conducted in large benchmark runs and demonstrated the capability to handle high-resolution data in future production runs.
ESiWACE contributed significantly to a step change in the capability of European climate and weather models by scaling them up to a spatial resolution of 1 km. In an international inter-comparison of extremely high resolution atmosphere models initiated and supported via ESiWACE, the breakthrough potential and high socio-economic value of this type of models in general, and the high quality and competitiveness of the European models in particular was established.
A particular impact and success of the European centre of excellence ESiWACE is that it could gather and heavily foster synergies throughout Europe on the joint way towards exascale. This has constituted itself in the form of various meetings, joint code developments and trainings, We carry-on the idea in the project ESiWACE2 that has started in January 2019 and will continue driving weather and climate modelling towards exascale computing.
A particular impact and success of the European centre of excellence ESiWACE is that it could gather and heavily foster synergies throughout Europe on the joint way towards exascale. This has constituted itself in the form of various meetings, joint code developments and trainings, We carry-on the idea in the project ESiWACE2 that has started in January 2019 and will continue driving weather and climate modelling towards exascale computing.