Making Europe’s supercomputers user-friendly
Supercomputers are a vital tool for European science and industry. These powerful machines can simulate the Earth’s climate, model the human brain, and design innovative products and materials for industry. A typical high-performance computing (HPC) system combines the central processing units found in all computers with specialised accelerators and a combination of memory and storage options. While such a ‘mix and match’ approach boosts performance, the inherent complexity makes supercomputers tricky to use. The DEEP-SEA project set out to build and integrate bespoke open-source software to help scientists and businesses get the most out of these powerful machines. “It is crucial that Europe invests in improving supercomputer software stacks to better support its heterogeneous HPC systems,” says Estela Suarez, joint lead of novel system architecture design at the Jülich Research Centre in Germany, and DEEP-SEA project coordinator. “So this was our goal : to deliver an advanced software stack for high-end (pre- and exascale) European HPC systems.”
The DEEP-SEA approach
Imagine a computer with a huge number of separate brains and different types of memory. DEEP-SEA wanted to ensure all these parts worked together seamlessly, always solving complex computation tasks in the most resource-efficient way, and always placing data in the best spot to achieve this. The project consortium included 14 European partners, each bringing their own expertise. By integrating existing tools and combining their functionalities, they were able to conduct a comprehensive development process driven by continuous co-design, with the application developers working on real-world use cases from the European HPC community. The DEEP-SEA software stack was built around the concept of ‘Optimisation Cycles’. These were specific flows in the stack that addressed particular objectives. For instance, the ‘modular supercomputing optimisation cycle’ assists users in selecting the most appropriate compute elements and mapping their applications onto a modular HPC system accordingly. By the end of the project, DEEP-SEA had delivered a comprehensive, integrated HPC software stack that included low-level libraries, system software, runtime, programming environments, and tools to improve the performance and usability of supercomputers with mixed components. On top of that, the project also created documentation and training materials to familiarise end users with the features and benefits of using the DEEP-SEA stack and tools, and to help them to navigate and decide which ones to use depending on their individual applications and objectives.
Surpassing expectations
Collaboration was key, and working with similar projects such as IO-SEA and RED-SEA turned out to be even better than expected, with all three projects working together seamlessly, achieving the integration of their respective results and helping to push each line of development further. Suarez explains the impact of this collaboration: “The large amount of technical work that came out of this experience was not something we expected. It was an enriching experience for all projects involved.” As a result, DEEP-SEA published over 65 high-quality scientific papers in just three years. This accomplishment is testament to the quality of the research and development delivered by the project.
Software for the future
The project was carried out with support from the European High Performance Computing Joint Undertaking (EuroHPC JU), an initiative set up to develop a world-class supercomputing and quantum computing ecosystem in Europe. Though DEEP-SEA concluded in March 2024, the work is far from over. “A software stack is never truly finished, and hardware advancements will need ongoing modifications and improvements to keep up with the pace of changes generated by HPC research ,” concludes Suarez. The real success of the DEEP-SEA project lies in its long-term, participatory approach. In combining existing tools and expertise across the EU, DEEP-SEA unlocked new functionalities for HPC that were previously unavailable, and paved the way for straightforward, efficient use of Europe’s supercomputers.
Keywords
DEEP-SEA, EuroHPC JU, programming environment, co-design, exascale, software stack, resource, efficient, supercomputer, HPC
