Software for Efficient and Energy-Aware Supercomputers

Informazioni relative al progetto

SEANERGYS

ID dell’accordo di sovvenzione: 101177590

DOI

10.3030/101177590

Data della firma CE 18 Giugno 2025

Data di avvio 1 Giugno 2025

Data di completamento 31 Maggio 2029

Finanziato da

• Digital, Industry and Space

Costo totale

€ 32 947 132,50

Contributo UE

€ 16 473 566,30

Coordinato da

FORSCHUNGSZENTRUM JULICH GMBH
Germany

Periodic Reporting for period 1 - SEANERGYS (Software for Efficient and Energy-Aware Supercomputers)

Periodo di rendicontazione: 2025-06-01 al 2025-11-30

Sintesi del contesto e degli obiettivi generali del progetto

Europe’s high-performance computing (HPC) and artificial intelligence (AI) centres are powerful yet power-hungry. As demand for computing grows, so do the energy bill and ecological impact. SEANERGYS tackles this challenge by creating an integrated software (SW) suite to make operation of supercomputers more energy-efficient without sacrificing performance, and to help end-users to employ these resources in an energy-efficient way. The SW suite monitors energy use, predicts workload behaviour, and optimises job scheduling, allowing HPC/AI systems to do more with less. SEANERGYS aims to strike a balance between delivering compute results within a set energy budget and reducing energy use for the same output. The system is designed for production-level deployment up to Exascale, ensuring European HPC and AI centres run smarter, greener, and faster.

SEANERGYS creates an integrated European SW suite that optimises the operation of supercomputers. In doing so, it addresses four objectives: reducing the amount of energy used for real-world workload mixes as the primary objective, optimising resource utilisation, enhancing system throughput and reducing response time as secondary objectives. Since these can conflict with each other, site-specific policies will define the weights attached to each, and the SEANERGYS SW suite will tailor system operation towards the combined optimum. Scenarios include improving the throughput of HPC systems, generating more R&D results for a given energy budget, and producing a fixed set of R&D results with less energy, while striving to keep response times constant. The SEANERGYS SW solution consists of a comprehensive monitoring infrastructure (CMI), an Artificial Intelligence data analytics system (AIDAS), and a dynamic scheduling and resource management system (DSRM).

The CMI gathers data from hardware and SW sensors, and correlates it with scheduler information to identify jobs that do not fully utilise allocated resources. Users receive automatic feedback on energy and resource use for each run, plus actionable information on how to optimise these. The AIDAS leverages AI models trained with a vast set of operational data of the participating HPC/AI centres. It fingerprints resource usage patterns, predicts future job behaviour, and identifies complementary job profiles for potential co-scheduling. Finally, the DSRM utilises these insights to develop scheduling policies that maximise resource utilisation and energy efficiency, and supports workloads with dynamic and adaptable resource profiles.

The SEANERGYS suite will be ready for use at Exascale level. It builds on proven European projects, established expertise, and widely used open-source software to develop a SW suite that achieves the functionality, performance and stability needed by European HPC/AI centres, defined by KPIs and acceptance criteria and processes established at the project start. Development follows an agile, DevOps-based approach to provide full traceability by linking and tracking requirements, interface, functional and performance specifications, code design and development steps, and validation/verification throughout the development life cycle. Quality measures will include code reviews, automated SW quality analysis, unit and integration tests and a verification suite. The project will employ progressive testing and validation from single-node environments to mid-scale systems. Finally, there will be acceptance tests on production supercomputers..

Lavoro eseguito dall’inizio del progetto fino alla fine del periodo coperto dalla relazione e principali risultati finora ottenuti

In its first six-months, SEANERGYS has collected a comprehensive set of use cases reflecting the actual operation of its participating HPC/AI centres, their user communities and workloads, needs and concepts for improving energy efficiency, and pain points in doing so. The SEANERGYS SW suite will support these use cases, and the design of its SW architecture, selection of pre-existing components to be adapted/extended/integrated, and definition of data formats and interfaces are driven by them.

The use cases were complemented by a recorded set of requirements which the completed SEANERGYS SW suite will need to fulfil; such requirements reflect the operational needs of the participating HPC/AI centres (and in extension of the larger European centre community), and essential end-user requirements.

To enable effective management of the ambitious, large-scale and distributed SW development and assure end-to-end traceability of all steps in the SW development life cycle, the project has deployed an advanced Development Operations platform based on Gitlab in conjunction with strict processes that govern the development of the SW architecture, code and ML models, as well as unit, integration and system tests. The overriding objective here is to assure a high level of quality commensurate with in-production use of the completed SW suite, and enable straightforward identification and resolution of root causes in case of errors.

Architecture definition has made good progress, with initial focus on the central “data plane” that enables efficient communication between the many SW components, the existing approaches for measuring and recording operational monitoring data of systems, support infrastructure and facilities, and a comprehensive analysis of HPC scheduling and resource management systems suitable as a starting point for the DSRM development. In addition, the use of ML models for implementing the required AIDAS functionality has been studied.

To support early ML model development and to prepare the design of the monitoring data research repository, the participating HPC/AI centres have investigated which privacy-relevant or confidential data is collected by their current monitoring systems, and how such data can be anonymised to allow sharing (first within the project, later by request and under specific conditions to outside researchers). This analysis has also jump-started the discussion on common data and metadata formats.

Progressi oltre lo stato dell’arte e potenziale impatto previsto (incluso l’impatto socioeconomico e le implicazioni sociali più ampie del progetto fino ad ora)

The overarching approach of SEANERGYS is to combine existing, best-of-breed SW tools and components with emerging, ML and AI-based data analytics and advanced, truly dynamic scheduling and resource management to yield a comprehensive SW suite. This suite shall then handle the whole breadth of monitoring data (at the job, node, system and facility levels), creates predictions and draws conclusions about the actual system and facility operation, the characteristics and efficiency of executing and incoming workloads, and then uses these to ensure near-optimal execution of the workloads in terms of balancing energy efficiency with fulfilment of critical user needs. The SW suite will also interface with system operators and end-users, providing actionable feedback on how to further improve system operation and use. Thus, SEANERGYS is firmly based on the current state of the art and extends it by the use of novel ML based data analysis, truly dynamic HPC-compatible scheduling and resource management, and coverage of the full spectrum of HPC/AI centre-related monitoring data.

The use case collection, definition of requirements, and the early architecture and design work in the first project period all support these objectives to surpass the state of the art in HPC/AI operational middleware.