Periodic Reporting for period 2 - EXCELLERAT (The European Centre of Excellence for Engineering Applications)
Berichtszeitraum: 2020-06-01 bis 2022-05-31
Engineering applications are in the phase of exploiting Exascale capabilities and challenges, not only in academia but also within industry. Looking at the engineering sector, we see that the domain of mechanical and plant engineering alone is the largest industrial sector in the EU, in terms of the economic output generated. The field of mechanical engineering has around 80,000 companies with three million employees where this community is strong in its innovation capabilities. It is important to understand that innovation within this field can be accelerated to a maximum by applying HPC, HPDA and AI. And even further, these innovations will lead to new products and knowledge, which will help society to tackle the global challenges arising, from a currently fluctuating market up to the climate crisis and the transfer to a zero-carbon society.
Thus, the EXCELLERAT activity combined the premium European expertise to establish a Centre of Excellence (CoE) in Engineering Applications on HPC with a broad service portfolio, paving the way for the evolution towards Exascale, which has already solved highly complex and costly engineering problems, and created enhanced technological solutions even at the development stage. This has been perfectly aligned with the European HPC Strategy, as implemented through the EuroHPC Joint Undertaking.
EXCELLERAT addressed the setup of a Centre of Excellence by not only looking at the simulation aspects per se, but on the overall lifecycle of engineering in HPC. Thus, the objective was to establish an entity, acting as a single competence and community hub, covering a wide range of issues, from «non-pure-technical» services, such as access to knowledge or networking up to technical services as e.g. co-design, scalability enhancement or code porting to new (Exa-) hardware. This hub is currently represented by the EXCELLERAT service portal (https://services.excellerat.eu/) where information and services are accessible for everyone.
Thus, the EXCELLERAT activity combined the premium European expertise to establish a Centre of Excellence (CoE) in Engineering Applications on HPC with a broad service portfolio, paving the way for the evolution towards Exascale, which has already solved highly complex and costly engineering problems, and created enhanced technological solutions even at the development stage. This has been perfectly aligned with the European HPC Strategy, as implemented through the EuroHPC Joint Undertaking.
EXCELLERAT addressed the setup of a Centre of Excellence by not only looking at the simulation aspects per se, but on the overall lifecycle of engineering in HPC. Thus, the objective was to establish an entity, acting as a single competence and community hub, covering a wide range of issues, from «non-pure-technical» services, such as access to knowledge or networking up to technical services as e.g. co-design, scalability enhancement or code porting to new (Exa-) hardware. This hub is currently represented by the EXCELLERAT service portal (https://services.excellerat.eu/) where information and services are accessible for everyone.
The driving factor of the EXCELLERAT developments were six reference applications (Nek5000, Alya, AVBP, TPLS, FEniCS, CODA), which were analyzed with respect to their potential to support the aim of achieving Exascale performance in HPC for Engineering. Thus, they acted as showcases for the evolution of applications towards their (potential) execution at large scale on Exascale demonstrators, pre-Exascale systems and Exascale machines.
During the runtime of the project, the services provided and successes communicated were published stepwise and contained essential achievements for the codes, for the handling of the application lifecycle and for the community as a whole.
In parallel, a service portfolio for external stakeholders targeting the needs of different actor groups was developed and as a representation layer, the EXCELLERAT service portal was implemented and published for the community and other interested parties. Feedback on this was periodically collected from partners and external (to the consortium) users to ensure a continuous enhancement of usability and the expansion of the service offer. The service portal was integrated with the website to ensure a better usability.
Furthermore, a wide spectrum of training material (tutorials, best practices and guidelines) was made available on the service portal for the EXCELLERAT applications, tools and methodologies and a high number of training activities, mostly in the form of web-seminars due to COVID restrictions, have taken place within the consortium or in collaboration with other H2020 projects.
Finally, to strive towards sustainability, all the market aspects of EXCELLERAT as a future self-sustainable entity have been elaborated (with market assessments, business modelling, business scenario analytics) and provided a clearer picture, leading to further steps. All that was accompanied by a clear and target group-oriented dissemination and awareness creation strategy, leading not only to white papers and academic papers, but also to a plethora of material, which can be found on the EXCELLERAT web page.
During the runtime of the project, the services provided and successes communicated were published stepwise and contained essential achievements for the codes, for the handling of the application lifecycle and for the community as a whole.
In parallel, a service portfolio for external stakeholders targeting the needs of different actor groups was developed and as a representation layer, the EXCELLERAT service portal was implemented and published for the community and other interested parties. Feedback on this was periodically collected from partners and external (to the consortium) users to ensure a continuous enhancement of usability and the expansion of the service offer. The service portal was integrated with the website to ensure a better usability.
Furthermore, a wide spectrum of training material (tutorials, best practices and guidelines) was made available on the service portal for the EXCELLERAT applications, tools and methodologies and a high number of training activities, mostly in the form of web-seminars due to COVID restrictions, have taken place within the consortium or in collaboration with other H2020 projects.
Finally, to strive towards sustainability, all the market aspects of EXCELLERAT as a future self-sustainable entity have been elaborated (with market assessments, business modelling, business scenario analytics) and provided a clearer picture, leading to further steps. All that was accompanied by a clear and target group-oriented dissemination and awareness creation strategy, leading not only to white papers and academic papers, but also to a plethora of material, which can be found on the EXCELLERAT web page.
EXCELLERAT combined Europe’s leading HPC Centres, application specialists and supporting partners, who have worked with and offered their expertise and knowledge to engineering companies and researchers from across industry and academia for the past three decades. Furthermore, EXCELLERAT continuously extended its expertise and capabilities by integrating other stakeholders via interest groups, which ensured constant evolution.
Some examples for the achieved progress beyond the state of the art are listed below. These examples are on codes, which, for example will help to reduce CO2 emissions of combustion engines and thus will clearly have an impact on societal challenges:
• Improved applicability to engineering problems with complex geometry of NEK5000 by improved high order meshing methods and adaptive mesh refinement.
• Demonstrated strong scalability of the Navier-Stokes of Alya on 96,000 cores on 2 billion cells for aerodynamics simulations.
• Demonstrated strong scalability of the Eulerian-Lagrangian solver of Alya on 48,000 cores on 1 billion cells and 200k,000 particles in simulations of spray flames.
• Improved load balancing of reacting flow simulations using the DLB library for chemical integration with implicit ODE solvers.
• Improved efficiency of AVBP by highly scalable mesh partitioning and redistribution methodologies.
• Improvement of strong scalability in FEniCS.
• Development of a unified Central Processing Unit (CPU)/GPU vectorization strategy with system-level dynamic load balancing for heterogeneous architectures in Alya as an extension of the HPC codes to accelerators.
• Strong scaling benchmarking and profiling on 131,000 AMD Epyc2 cores (PRACE Irene Joliot Curie Rome system) with almost 90 % ideal scaling. Ported and tested AVBP up to 32 P100 GPUs on Piz Daint (PRACE CSCS) using the OpenACC development with almost 80% scalability.
• Ported Ax kernel of Nekbone over to FGPAs, currently demonstrating around 60 GFLOPS with a power-draw of 35 Watts with one kernel, and estimate can increase kernel performance further to around 100 GFLOPS and scale to around 4 or 5 kernels. So approx. 400-500 GFLOPS is the likely performance target, which will be competitive against GPUs with significantly reduced power-consumption. CODA’s linear algebra solver library (Spliss) is running on GPUs.
• Ported and tested Nek5000 on up to 256 A100 GPUs on JUWELS Booster (JSC) and Berzelius (NSC) using OpenACC, achieving up to 5 times the performance of the CPU version.
• Development of a Data Management and Workflow Handling platform to best possible support users in a simplified and secured environment.
Some examples for the achieved progress beyond the state of the art are listed below. These examples are on codes, which, for example will help to reduce CO2 emissions of combustion engines and thus will clearly have an impact on societal challenges:
• Improved applicability to engineering problems with complex geometry of NEK5000 by improved high order meshing methods and adaptive mesh refinement.
• Demonstrated strong scalability of the Navier-Stokes of Alya on 96,000 cores on 2 billion cells for aerodynamics simulations.
• Demonstrated strong scalability of the Eulerian-Lagrangian solver of Alya on 48,000 cores on 1 billion cells and 200k,000 particles in simulations of spray flames.
• Improved load balancing of reacting flow simulations using the DLB library for chemical integration with implicit ODE solvers.
• Improved efficiency of AVBP by highly scalable mesh partitioning and redistribution methodologies.
• Improvement of strong scalability in FEniCS.
• Development of a unified Central Processing Unit (CPU)/GPU vectorization strategy with system-level dynamic load balancing for heterogeneous architectures in Alya as an extension of the HPC codes to accelerators.
• Strong scaling benchmarking and profiling on 131,000 AMD Epyc2 cores (PRACE Irene Joliot Curie Rome system) with almost 90 % ideal scaling. Ported and tested AVBP up to 32 P100 GPUs on Piz Daint (PRACE CSCS) using the OpenACC development with almost 80% scalability.
• Ported Ax kernel of Nekbone over to FGPAs, currently demonstrating around 60 GFLOPS with a power-draw of 35 Watts with one kernel, and estimate can increase kernel performance further to around 100 GFLOPS and scale to around 4 or 5 kernels. So approx. 400-500 GFLOPS is the likely performance target, which will be competitive against GPUs with significantly reduced power-consumption. CODA’s linear algebra solver library (Spliss) is running on GPUs.
• Ported and tested Nek5000 on up to 256 A100 GPUs on JUWELS Booster (JSC) and Berzelius (NSC) using OpenACC, achieving up to 5 times the performance of the CPU version.
• Development of a Data Management and Workflow Handling platform to best possible support users in a simplified and secured environment.