Periodic Reporting for period 2 - LEXIS (Large-scale EXecution for Industry & Society)
Berichtszeitraum: 2020-07-01 bis 2021-12-31
The ever-increasing quantity of data generated by modern industrial and business processes poses an enormous challenges for organisations seeking to glean knowledge and understanding from the data. Combinations of HPC, Cloud and Big Data technologies are key to meeting the increasingly diverse needs of large and small organisations alike, however, access to powerful compute platforms for SMEs which has been difficult due to both technical and financial considerations may now be possible.
The LEXIS (Large-scale EXecution for Industry & Society) project has built an advanced engineering platform at the confluence of HPC, Cloud and Big Data, which leverages existing, geographically distributed large-scale resources in a federation of established EU supercomputing centres. It employs Big Data Analytics solutions, and augments them with Cloud services. Driven by the requirements of several pilot test cases, the LEXIS platform relies on best-in-class data management solutions (EUDAT) and advanced, distributed orchestration solutions (TOSCA), augmenting them with new, efficient hardware and platform capabilities (e.g. in the form of Data Nodes and federation, usage monitoring and accounting/billing support). Thus, LEXIS has realised an innovative solution aimed at stimulating the interest of European industry and at creating an ecosystem of organisations that benefit from the LEXIS platform and its well-integrated HPC, HPDA and Data Management solutions.
The consortium has developed a demonstrator with a significant Open Source dimension, including validation, testing and documentation. It has been validated in large-scale pilots in industrial and scientific sectors. LEXIS has also organised a specific Open Call for platform usage, which has stimulated adoption of the project framework and increased stakeholders’ engagement with the targeted pilots.
The collaboration has promoted, and will keep promoting this solution to the HPC, Cloud and Big Data sectors, maximising impact through targeted and qualified communication. LEXIS has brought together a consortium with the skills and experience to deliver a complex multi-faceted project, spanning a range of complex technologies across seven European countries, including large industry, flagship HPC centres, industrial and scientific compute pilot users, technology providers and SMEs.
The LEXIS (Large-scale EXecution for Industry & Society) project has built an advanced engineering platform at the confluence of HPC, Cloud and Big Data, which leverages existing, geographically distributed large-scale resources in a federation of established EU supercomputing centres. It employs Big Data Analytics solutions, and augments them with Cloud services. Driven by the requirements of several pilot test cases, the LEXIS platform relies on best-in-class data management solutions (EUDAT) and advanced, distributed orchestration solutions (TOSCA), augmenting them with new, efficient hardware and platform capabilities (e.g. in the form of Data Nodes and federation, usage monitoring and accounting/billing support). Thus, LEXIS has realised an innovative solution aimed at stimulating the interest of European industry and at creating an ecosystem of organisations that benefit from the LEXIS platform and its well-integrated HPC, HPDA and Data Management solutions.
The consortium has developed a demonstrator with a significant Open Source dimension, including validation, testing and documentation. It has been validated in large-scale pilots in industrial and scientific sectors. LEXIS has also organised a specific Open Call for platform usage, which has stimulated adoption of the project framework and increased stakeholders’ engagement with the targeted pilots.
The collaboration has promoted, and will keep promoting this solution to the HPC, Cloud and Big Data sectors, maximising impact through targeted and qualified communication. LEXIS has brought together a consortium with the skills and experience to deliver a complex multi-faceted project, spanning a range of complex technologies across seven European countries, including large industry, flagship HPC centres, industrial and scientific compute pilot users, technology providers and SMEs.
The LEXIS Platform allows the orchestration of complex scientific and industrial workflows which can be easily run on distributed HPC and Cloud resources. Thanks to its API and modern web based UI, powerful HPC resources and complex Cloud deployments can be easily accessed and challenging workflows can be executed. It uses YORC (Ystia Orchestrator) to handle the workflows, an iRODS-based data management solution with asynchronous staging APIs, and integration with EUDAT services to handle extensive datasets in a secure and convenient way.
LEXIS Platform consists of several modules covering all the necessary functionalities, communicating by REST APIs, and using a zero-trust security policy based on modern OpenID-Connect/JWT technologies. The LEXIS Platform implements its own identities and remains compatible with local HPC centre identity and project management through the HEAppE middleware.
The individual LEXIS software modules have been implemented, deployed and integrated in the resulting LEXIS Platform, which has been successfully validated, tested and exposed on a public endpoint. The LEXIS Portal is the web based UI through which users can use all the features of the platform, including an overview of consumed resources and its cost. The LEXIS Platform implements multi-tenancy by grouping users in different roles to organisations and projects through which the compute resources are consumed.
An indispensable part of the LEXIS design process was the three pilots with different requirements.
The Aeronautics Pilot comprised two different use cases. The first one focused on Turbomachinery, emphasising accelerating simulations of designed components. The code was rewritten to leverage modern hardware GPU accelerators which led to a 21.4x speed up of the computation in comparison to the original computation time. Additionally, a check-pointing mechanism was implemented which saves the state during the computation and restarts the computation at that point if there is a system failure. Another feature made available for this use case was a simple user-friendly visualisation of the progress through the LEXIS Portal during the computation. The second use case (Rotating Parts) concentrated on the implementation of a new methodology for the simulation of advanced hydrodynamics in complex geometries under turbulent motion. This allows for the definition of the proper accuracy of the simulation and minimisation of the required computing time. Also, the computationally intensive preprocessing phase was included in the LEXIS workflow, making it possible to run the entire computation from the LEXIS Portal.
The formalisation and implementation of an urgent computing workflow was the main focus of the Earthquake and Tsunami pilot. To this end, the partners working in this pilot had to adjust their software to be able to seamlessly communicate between each other. The most computationally expensive computation of the tsunami inundation made by TsunAWI was optimised and parallelised with MPI, which made it significantly faster. The system for managing a building database, central for damage assessment in the workflow, was completely remade from the ground up and made at least 50 times faster than before.
In the Weather and Climate Large-Scale Pilot, well-proven HPC models - as well as preprocessing and application/postprocessing codes, were virtualised and prepared as ready-to-run containers and virtual machines compatible with the LEXIS Platform. The procedures for handling workflows and data from different sources have been completely revised from the ground up and made ready the for next-generation model. Runs of the respective workflows on the platform were presented at SC20; additionally, the model management has reached production quality and utilises a wealth of HPC and Cloud resources integrated by the LEXIS Platform, as well as the newly-designed Weather and Climate Data API by ECMWF.
LEXIS Platform consists of several modules covering all the necessary functionalities, communicating by REST APIs, and using a zero-trust security policy based on modern OpenID-Connect/JWT technologies. The LEXIS Platform implements its own identities and remains compatible with local HPC centre identity and project management through the HEAppE middleware.
The individual LEXIS software modules have been implemented, deployed and integrated in the resulting LEXIS Platform, which has been successfully validated, tested and exposed on a public endpoint. The LEXIS Portal is the web based UI through which users can use all the features of the platform, including an overview of consumed resources and its cost. The LEXIS Platform implements multi-tenancy by grouping users in different roles to organisations and projects through which the compute resources are consumed.
An indispensable part of the LEXIS design process was the three pilots with different requirements.
The Aeronautics Pilot comprised two different use cases. The first one focused on Turbomachinery, emphasising accelerating simulations of designed components. The code was rewritten to leverage modern hardware GPU accelerators which led to a 21.4x speed up of the computation in comparison to the original computation time. Additionally, a check-pointing mechanism was implemented which saves the state during the computation and restarts the computation at that point if there is a system failure. Another feature made available for this use case was a simple user-friendly visualisation of the progress through the LEXIS Portal during the computation. The second use case (Rotating Parts) concentrated on the implementation of a new methodology for the simulation of advanced hydrodynamics in complex geometries under turbulent motion. This allows for the definition of the proper accuracy of the simulation and minimisation of the required computing time. Also, the computationally intensive preprocessing phase was included in the LEXIS workflow, making it possible to run the entire computation from the LEXIS Portal.
The formalisation and implementation of an urgent computing workflow was the main focus of the Earthquake and Tsunami pilot. To this end, the partners working in this pilot had to adjust their software to be able to seamlessly communicate between each other. The most computationally expensive computation of the tsunami inundation made by TsunAWI was optimised and parallelised with MPI, which made it significantly faster. The system for managing a building database, central for damage assessment in the workflow, was completely remade from the ground up and made at least 50 times faster than before.
In the Weather and Climate Large-Scale Pilot, well-proven HPC models - as well as preprocessing and application/postprocessing codes, were virtualised and prepared as ready-to-run containers and virtual machines compatible with the LEXIS Platform. The procedures for handling workflows and data from different sources have been completely revised from the ground up and made ready the for next-generation model. Runs of the respective workflows on the platform were presented at SC20; additionally, the model management has reached production quality and utilises a wealth of HPC and Cloud resources integrated by the LEXIS Platform, as well as the newly-designed Weather and Climate Data API by ECMWF.
During the LEXIS Project, a large-scale, federated European orchestration and data management platform was created and deployed. It is now ready to be further exploited by various use-cases, including those from both academia and industry. The main impact of the LEXIS Platform is lowering the entry barrier for accessing powerful supercomputers and versatile Cloud environments. The most important features of the platform are:
· dynamic data-aware orchestration of complex workflows;
· data sharing between Cloud and HPC resources and distributed data management with an appropriate data backend to the orchestration solutions;
· access to HPC/BigData/Cloud resources for SMEs and Industry;
· easy control of workflows and data through a user-friendly web interface, with seamless integration of remote visualisation services.
Users from various fields can easily gain access to the most appropriate compute resource they need, without the burden of complicated access policies and without steep learning curves related to technical knowledge requirements.
The expected results of the project have been fulfilled as described above.
· dynamic data-aware orchestration of complex workflows;
· data sharing between Cloud and HPC resources and distributed data management with an appropriate data backend to the orchestration solutions;
· access to HPC/BigData/Cloud resources for SMEs and Industry;
· easy control of workflows and data through a user-friendly web interface, with seamless integration of remote visualisation services.
Users from various fields can easily gain access to the most appropriate compute resource they need, without the burden of complicated access policies and without steep learning curves related to technical knowledge requirements.
The expected results of the project have been fulfilled as described above.