Exascale framework for supporting high-fidelity simulations of multiphase reacting flows in complex geometries

Project Information

Exa-FireFlows

Grant agreement ID: 846139

DOI

10.3030/846139

Project closed

EC signature date 10 April 2019

Start date 1 June 2019

End date 31 May 2021

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 172 932,48

EU contribution

€ 172 932,48

Coordinated by

BARCELONA SUPERCOMPUTING CENTER CENTRO NACIONAL DE SUPERCOMPUTACION
Spain

Periodic Reporting for period 1 - Exa-FireFlows (Exascale framework for supporting high-fidelity simulations of multiphase reacting flows in complex geometries)

Reporting period: 2019-06-01 to 2021-05-31

Summary of the context and overall objectives of the project

High-performance computing (HPC) has transformed scientific research across numerous disciplines by supporting theory and experiments with numerical simulations. Exascale computing is the next milestone in HPC and is called to play an essential role in economic competitiveness, societal challenges, and science leadership. Combustion is one of the fields with high strategic importance and potential to fully exploit future exascale systems. Further understanding of the physics and chemistry of the combustion process is fundamental to achieve improvements in fuel efficiency, reducing greenhouse gas emissions and pollutants while transitioning to alternative fuels and greener technologies. Advanced numerical simulations have enabled to make significant contributions for increasing cycle efficiency, reduction of pollutant emissions, and use of alternative fuels in practical applications. However, implementing the new and future supercomputers requires the evolution of multiple and different technologies in a coherent and complementary way, including hardware, software, and application algorithms. Hence, scientific codes and formulations need to be re-designed and adapted to exploit the different levels of parallelism and complex memory hierarchies of the new and future heterogeneous systems. The project aims to explore and develop novel HPC strategies into a software stack that allows the simulation of advanced high-fidelity multiphase reacting flows in complex geometries using unstructured grids.
The main objectives of this work are:
O1: Development of a computational framework supporting a wide range of flow simulations in complex geometries and unstructured grids.
O2: Attaining maximum performance at an intra-node level.
O3: Advancing in inter-node scalability.
O4: Validation and Integration: A demonstrator will be created for predicting emissions using high fidelity simulations in both high resolution and large domains.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

A refactorization of the flamelet model with the objective to optimize the CPU and GPU execution took place in the early phases of the project. New data structures and preprocessing functions were developed for that purpose. The idea was to facilitate the compiler work enabling the generation of vector operations. The same data structures expose the massive parallelism of the operations involved in the assembly of the equations. As a result, the code ended up running five times faster than the previous version, and the data structures exhibited the potential utilization of GPUs using loop-based directives.
Having a CPU and GPU version of the code motivated finding a way to utilize all the resources of the computing nodes concurrently. The proposed execution strategy was name co-execution. Two binaries of the same code interact in a multiple-program multiple-data execution model.
One binary was optimized to CPU execution while the other to GPU execution. The modules regarding communication between processes are the same in both binaries; therefore, the programs can communicate their information transparently. A dynamic load balancing algorithm redistributes the workload for finding the sweet spot of the execution.
The co-execution obtained an acceleration of 20% with respect to executing only using the GPUs.
The benefits of the refactorization were applied to the sectional soot model. Understanding soot phenomena is essential for reducing the soot emissions on the combustion procedures. The sectional soot model demands many computing calculations, so it quickly becomes the bottleneck of high-fidelity simulations. Transferring the learning from the flamelet model allows us to reorganize the code similarly to expose parallelism and vector operations. As a result, the overall acceleration on high-fidelity simulations with soot was accelerated more than five times. The calculation of the sources terms will be released as a library to spread the efficient utilization of this model.

Regarding the emerging technologies, a sparse matrix vector-multiplication was implemented for using the FPGAs efficiently when working with sparsity patterns arisen from unstructured CFD codes. The resulting scheme runs up to 12 times faster than the current state-of-art algorithms for general matrices.


The dissemination of the achievements and results of the proposed project was a priority to engage as many users as possible. For that purpose, the researcher participated in four international conferences. Moreover, four articles have been published in international journals or conference proceedings. Two more papers are expected to be sent for publication by the end of this year. The work was also presented in the Center of Excellence in Combustion (CoEC) monthly meetings that join the leading institution of combustion research in Europe. The researcher participated in two outreach activities to show the importance of HPC to high-school students.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The European Commission and other governments in the world have acknowledged the tremendous importance of HPC for future leadership. Advances in combustion simulations are also of high strategic importance due to their impact on industry developments. The project results show that high-fidelity simulations can run up to five times faster than before when using CPU systems. In other words, simulations that used to take months now only take weeks. This implies a better and efficient usage of the current HPC systems and a reduction in the numerical experiments pipeline. Numerical results are obtained faster, or more simulations can be performed in the same amount of time. The European pre-Exascale systems will be heterogeneous with nodes consisting of CPUs and GPUs from different vendors. The co-execution model will play an important role in efficiently utilizing all the resources of the heterogeneous nodes.
The results are currently being exploited by other European institutions through the Center of Excellence in Combustion and EXCELLERAT.

Overall acceleration of Alya for different VECTOR_SIZE using three compilers

Co-execution times compared with CPU-only and GPU-only executions for different mesh sizes

Acceleration on the sectional soot model for different workloads and sections