The project, based on the definition of the requirements, both from the pillar applications and from the different components of the software stack, designed the project architecture. To achieve the project goals, abstractions to support the software integration of HPC, AI and BD components were developed. Other abstractions to support data transformations between workflow steps were proposed. Project activities also include the optimization of different components, including new proposal based on machine learning that are leveraged by the project pillars.
The project has designed and developed the HPCWaaS methodology. Some important components of the HPCWaaS methodology are provided as services, such as the data catalogue service which links to project data sets, the Data Logistic service, able to move data from external repositories to the HPC systems based on data pipelines, or the Container Image Creation service, able to generate container images for specific hardware.
The project identified from the pillar applications computational and artificial intelligence kernels with performance bottlenecks. A selection of these kernels was optimized for GPUs, FPGAs and in particular for the European Processor ecosystem. The project has also researched the use of new storage devices such as NVRAMs as an alternative for persistent data storage instead of traditional disks.
Pillar I, aimed at developing digital twins for manufacturing, with a workflow developed on top of the eFlows4HPC software stack that allows to run an end-to-end full reduction model of the cooling system of a SIEMENS electrical engine. The final version integrates real physics from heat transfer in electric motors, validated by comparing the results with commercial codes from SIEMENS. The Pillar has achieved tremendous success with speedups of the order of 50x-100x and a large impact on memory efficiency. The partner CIMNE is following an innovation path marketed as SimTwins to commercialize the pillar results and won an innovation award.
Pillar II focused in the development of two workflows: the Dynamic (AI-assisted) Earth System Model (ESM) workflow that aims at pruning ensembles runs based on runtime analytics; and the Statistical analysis and feature extraction workflow aiming at the prediction of tropical cyclones. The ESM workflow is developed on top of the software stack, including PyCOMPSs for workflow orchestration and Hecuba for data management. It has been deployed in multiple supercomputers, such as MareNostrum 4 (BSC) and Levante (DKRZ). The feature extraction workflow integrates advanced features from PyCOMPSs to manage streamed data. The application of this workflow to real data demonstrated its ability to track tropical cyclones in the North Pacific.
Pillar III aimed at developing workflows for earthquakes (UCIS4EQ) and for is subsequent tsunamis (FTRT/PTF). The two workflows have been designed and developed on top of the project software. The Pillar has also developed MLESmap, a novel procedure that exploits the predictive power of ML algorithms to estimate ground acceleration values a few seconds after a large earthquake occurs. Thanks to these workflows, the partners have been able to perform very significant studies such as predicting high-resolution inundation maps from tsunami simulations, or new validation studies of the Puebla and Kos-Bodrun earthquakes. In addition, the pillar has generated a white paper with recommendations for urgent computing for natural hazards.
All project software (software stack, HPCWaaS methodology and developed workflows) is available as open source in public repositories and with online documentation.
The project has done large efforts on dissemination, training and technology transfer. The project partners have published 32 articles in journals and conferences. Up to 8 training courses have been delivered, all of them but one targeting external participants. The project has organized four workshops focusing on user communities, with 10 scientific communities/initiatives engaged.
The project has selected 14 Key Exploitable Results from a total of 47 project results initially listed, based on the degree of innovation, on the exploitability and on impact of each result. Most of the KERs are under exploitation in follow-up projects or by the partners themselves. Most of the workflows are sustained by the user community and one of them (Pillar I) is following an innovation path towards commercialization
