Skip to main content

Adaptive multi-tier intelligent data manager for Exascale

Objective

The growing need to process extremely large data sets is one of the main drivers for building exascale HPC systems today. However, the flat storage hierarchies found in classic HPC architectures no longer satisfy the performance requirements of data-processing applications. Uncoordinated file access in combination with limited bandwidth make the centralised back-end parallel file system a serious bottleneck. At the same time, emerging multi-tier storage hierarchies come with the potential to remove this barrier. But maximising performance still requires careful control to avoid congestion and balance computational with storage performance. Unfortunately, appropriate interfaces and policies for managing such an enhanced I/O stack are still lacking.

The main objective of the ADMIRE project is to establish this control by creating an active I/O stack that dynamically adjusts computation and storage requirements through intelligent global coordination, malleability of computation and I/O, and the scheduling of storage resources along all levels of the storage hierarchy. To achieve this, we will develop a software-defined framework based on the principles of scalable monitoring and control, separated control and data paths, and the orchestration of key system components and applications through embedded control points.

Our software-only solution will allow the throughput of HPC systems and the performance of individual applications to be substantially increased – and consequently energy consumption to be decreased – by taking advantage of fast and power-efficient node-local storage tiers using novel, European ad-hoc storage systems and in-transit/in-situ processing facilities. Furthermore, our enhanced I/O stack will offer quality-of-service (QoS) and resilience. An integrated and operational prototype will be validated with several use cases from various domains, including climate/weather, life sciences, physics, remote sensing, and deep learning.

Call for proposal

H2020-JTI-EuroHPC-2019-1
See other projects for this call

Coordinator

UNIVERSIDAD CARLOS III DE MADRID
Address
Calle Madrid 126
28903 Getafe (Madrid)
Spain
Activity type
Higher or Secondary Education Establishments
EU contribution
€ 383 937,50

Participants (13)

JOHANNES GUTENBERG-UNIVERSITAT MAINZ
Germany
EU contribution
€ 340 562,50
Address
Saarstrasse 21
55122 Mainz
Activity type
Higher or Secondary Education Establishments
BARCELONA SUPERCOMPUTING CENTER-CENTRO NACIONAL DE SUPERCOMPUTACION
Spain
EU contribution
€ 304 375
Address
Calle Jordi Girona 31
08034 Barcelona
Activity type
Research Organisations
TECHNISCHE UNIVERSITAT DARMSTADT
Germany
EU contribution
€ 432 905
Address
Karolinenplatz 5
64289 Darmstadt
Activity type
Higher or Secondary Education Establishments
DATADIRECT NETWORKS FRANCE
France
EU contribution
€ 425 000
Address
10-12 Rue Andras Beck
92360 Meudon La Foret
Activity type
Private for-profit entities (excluding Higher or Secondary Education Establishments)
INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET AUTOMATIQUE
France
EU contribution
€ 345 000
Address
Domaine De Voluceau Rocquencourt
78153 Le Chesnay Cedex
Activity type
Research Organisations
PARATOOLS SAS
France
EU contribution
€ 318 425
Address
2 Rue De La Piquetterie
91680 Bruyeres Le Chatel
Activity type
Private for-profit entities (excluding Higher or Secondary Education Establishments)
FORSCHUNGSZENTRUM JULICH GMBH
Germany
EU contribution
€ 300 886,88
Address
Wilhelm Johnen Strasse
52428 Julich
Activity type
Research Organisations
CONSORZIO INTERUNIVERSITARIO NAZIONALE PER L'INFORMATICA
Italy
EU contribution
€ 298 750
Address
Via Ariosto 25
00185 Roma
Activity type
Higher or Secondary Education Establishments
CINECA CONSORZIO INTERUNIVERSITARIO
Italy
EU contribution
€ 116 250
Address
Via Magnanelli 6/3
40033 Casalecchio Di Reno Bo
Activity type
Research Organisations
E4 COMPUTER ENGINEERING SPA
Italy
EU contribution
€ 153 863,75
Address
Via Martiri Della Liberta 66
42019 Scandiano
Activity type
Private for-profit entities (excluding Higher or Secondary Education Establishments)
INSTYTUT CHEMII BIOORGANICZNEJ POLSKIEJ AKADEMII NAUK
Poland
EU contribution
€ 275 250
Address
Noskowskiego 12-14
61 704 Poznan
Activity type
Research Organisations
KUNGLIGA TEKNISKA HOEGSKOLAN
Sweden
EU contribution
€ 156 437,50
Address
Brinellvagen 8
100 44 Stockholm
Activity type
Higher or Secondary Education Establishments
MAX-PLANCK-GESELLSCHAFT ZUR FORDERUNG DER WISSENSCHAFTEN EV
Germany
EU contribution
€ 130 000
Address
Hofgartenstrasse 8
80539 Munchen
Activity type
Research Organisations