Software for Efficient and Energy-Aware Supercomputers

Projektinformationen

SEANERGYS

ID Finanzhilfevereinbarung: 101177590

DOI

10.3030/101177590

EK-Unterschriftsdatum 18 Juni 2025

Startdatum 1 Juni 2025

Enddatum 31 Mai 2029

Finanziert unter

• Digital, Industry and Space

Gesamtkosten

€ 32 947 132,50

EU-Beitrag

€ 16 473 566,30

Koordiniert durch

FORSCHUNGSZENTRUM JULICH GMBH
Germany

Periodic Reporting for period 1 - SEANERGYS (Software for Efficient and Energy-Aware Supercomputers)

Berichtszeitraum: 2025-06-01 bis 2025-11-30

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

Europe’s high-performance computing (HPC) and artificial intelligence (AI) centres are powerful yet power-hungry. As demand for computing grows, so do the energy bill and ecological impact. SEANERGYS tackles this challenge by creating an integrated software (SW) suite to make operation of supercomputers more energy-efficient without sacrificing performance, and to help end-users to employ these resources in an energy-efficient way. The SW suite monitors energy use, predicts workload behaviour, and optimises job scheduling, allowing HPC/AI systems to do more with less. SEANERGYS aims to strike a balance between delivering compute results within a set energy budget and reducing energy use for the same output. The system is designed for production-level deployment up to Exascale, ensuring European HPC and AI centres run smarter, greener, and faster.

SEANERGYS creates an integrated European SW suite that optimises the operation of supercomputers. In doing so, it addresses four objectives: reducing the amount of energy used for real-world workload mixes as the primary objective, optimising resource utilisation, enhancing system throughput and reducing response time as secondary objectives. Since these can conflict with each other, site-specific policies will define the weights attached to each, and the SEANERGYS SW suite will tailor system operation towards the combined optimum. Scenarios include improving the throughput of HPC systems, generating more R&D results for a given energy budget, and producing a fixed set of R&D results with less energy, while striving to keep response times constant. The SEANERGYS SW solution consists of a comprehensive monitoring infrastructure (CMI), an Artificial Intelligence data analytics system (AIDAS), and a dynamic scheduling and resource management system (DSRM).

The CMI gathers data from hardware and SW sensors, and correlates it with scheduler information to identify jobs that do not fully utilise allocated resources. Users receive automatic feedback on energy and resource use for each run, plus actionable information on how to optimise these. The AIDAS leverages AI models trained with a vast set of operational data of the participating HPC/AI centres. It fingerprints resource usage patterns, predicts future job behaviour, and identifies complementary job profiles for potential co-scheduling. Finally, the DSRM utilises these insights to develop scheduling policies that maximise resource utilisation and energy efficiency, and supports workloads with dynamic and adaptable resource profiles.

The SEANERGYS suite will be ready for use at Exascale level. It builds on proven European projects, established expertise, and widely used open-source software to develop a SW suite that achieves the functionality, performance and stability needed by European HPC/AI centres, defined by KPIs and acceptance criteria and processes established at the project start. Development follows an agile, DevOps-based approach to provide full traceability by linking and tracking requirements, interface, functional and performance specifications, code design and development steps, and validation/verification throughout the development life cycle. Quality measures will include code reviews, automated SW quality analysis, unit and integration tests and a verification suite. The project will employ progressive testing and validation from single-node environments to mid-scale systems. Finally, there will be acceptance tests on production supercomputers..

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

In its first six-months, SEANERGYS has collected a comprehensive set of use cases reflecting the actual operation of its participating HPC/AI centres, their user communities and workloads, needs and concepts for improving energy efficiency, and pain points in doing so. The SEANERGYS SW suite will support these use cases, and the design of its SW architecture, selection of pre-existing components to be adapted/extended/integrated, and definition of data formats and interfaces are driven by them.

The use cases were complemented by a recorded set of requirements which the completed SEANERGYS SW suite will need to fulfil; such requirements reflect the operational needs of the participating HPC/AI centres (and in extension of the larger European centre community), and essential end-user requirements.

To enable effective management of the ambitious, large-scale and distributed SW development and assure end-to-end traceability of all steps in the SW development life cycle, the project has deployed an advanced Development Operations platform based on Gitlab in conjunction with strict processes that govern the development of the SW architecture, code and ML models, as well as unit, integration and system tests. The overriding objective here is to assure a high level of quality commensurate with in-production use of the completed SW suite, and enable straightforward identification and resolution of root causes in case of errors.

Architecture definition has made good progress, with initial focus on the central “data plane” that enables efficient communication between the many SW components, the existing approaches for measuring and recording operational monitoring data of systems, support infrastructure and facilities, and a comprehensive analysis of HPC scheduling and resource management systems suitable as a starting point for the DSRM development. In addition, the use of ML models for implementing the required AIDAS functionality has been studied.

To support early ML model development and to prepare the design of the monitoring data research repository, the participating HPC/AI centres have investigated which privacy-relevant or confidential data is collected by their current monitoring systems, and how such data can be anonymised to allow sharing (first within the project, later by request and under specific conditions to outside researchers). This analysis has also jump-started the discussion on common data and metadata formats.

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

The overarching approach of SEANERGYS is to combine existing, best-of-breed SW tools and components with emerging, ML and AI-based data analytics and advanced, truly dynamic scheduling and resource management to yield a comprehensive SW suite. This suite shall then handle the whole breadth of monitoring data (at the job, node, system and facility levels), creates predictions and draws conclusions about the actual system and facility operation, the characteristics and efficiency of executing and incoming workloads, and then uses these to ensure near-optimal execution of the workloads in terms of balancing energy efficiency with fulfilment of critical user needs. The SW suite will also interface with system operators and end-users, providing actionable feedback on how to further improve system operation and use. Thus, SEANERGYS is firmly based on the current state of the art and extends it by the use of novel ML based data analysis, truly dynamic HPC-compatible scheduling and resource management, and coverage of the full spectrum of HPC/AI centre-related monitoring data.

The use case collection, definition of requirements, and the early architecture and design work in the first project period all support these objectives to surpass the state of the art in HPC/AI operational middleware.