Periodic Reporting for period 2 - SAGE (SAGE)
Período documentado: 2017-03-01 hasta 2018-08-31
The following are the overall objectives of the SAGE project:
• Provide a next-generation multi-tiered object-based data storage system (hardware and enabling software) supporting current and future-generation persistent storage media, (solid-state and disc) within an I/O hierarchy . We term this “Percipient Storage”.
• The project;
o Redefines the storage subsystem as an integral part of the computational infrastructure.
o Provides integrated computational capability anywhere in the storage system.
• Significantly improves the overall scientific output through advancements in systemic I/O performance and latency, and drastically reduces data movements hence improving energy efficiency by:
o providing the ability to flexibly move appropriate computational workloads to where the data resides
o providing a storage architecture built from the ground up to handle Exascale I/O
o providing a potential to use resources in the computational cluster as part of the storage system
• Provides a roadmap of technologies supporting data access for both Exascale/Exabyte and High Performance Data Analytics (HPDA) requirements:
o Targeting scalability to 500-1000PBytes, with bandwidth in the order of 60TB/sec with a storage system energy footprint of less than approximately 5KW/petabyte;
o With flexible and efficient usage of HPDA application environments regardless of the compute node’s architecture and implementation.
• Investigates and documents the requirements of relevant HPC applications and their storage use cases as part of a co-design approach.
• Provides programming models and access methods for the SAGE architecture and validates their usability, including (but not limited to) legacy applications and ‘Big-Data’ data access and analysis methods.
• Validates the the full system in a relevant environment, for a relevant set of applications and benchmarks on a SAGE prototype integrated into an HPC data centre, validating performance, scalability, energy efficiency and the reduction in data transport requirements.
Once accomplished, these objectives will firmly establish European excellence in the areas of Exascale storage, data centric computing, HPDA, and the emerging field of Big Data Extreme Computing (BDEC), and significantly impact computational scientific research.
The last 18 months of the SAGE project (M19 – M36) helped to provide a full working implementation of the SAGE prototype system (“SAGE system”). Multiple tiers working with Mero on the SAGE prototype are now fully realised. The applications are also deployed on the SAGE prototype and we show that they have exploited the capabilities of the SAGE offered by Mero, the multiple tiers and other software components working on top of Mero. The implementations of Mero components, the Clovis API, and the tools (HSM, OSIC, Performance tools), Access methods (pNFS and HDF5), Programming models & analytics (PGAS, Apache Flink), and, Runtimes (Semi-Persistent Cache Manager and Virtual Memory Manager) are also fully realised and tested with Mero and the Clovis API as needed. NVRAM technologies that were available in the time frame of the project have been fully accessed, used in the prototype when available (3DXPoint PCI-E form factor) and emulated when not (NVDIMMS).
Through SAGE disseminations, collaborations had been realised both within and outside the H2020 programme, primarily with partners in other FETHPC projects. One of the major outcomes of these was that this led to the follow on project, Sage2 (led again by Seagate) that will carry forward all the innovations accomplished in SAGE, that will now include other external collaboration partners outside of SAGE, becoming an integral part of the SAGE ecosystem. Market exploitation of many of the key SAGE components such as Mero have already begun with internal products being developed and evaluated by potential customers.
SAGE to lay the foundation for a European storage platform to be #1 at Extreme Scale
In the course of pursuing this vision, the project has already highlighted progress beyond the existing state of the art by:
(1) Providing a working object storage base platform (Mero) and its API Clovis specifically suited for Extreme scale HPC
(2) Providing prototype system hardware for multi-tier storage system with more than 3 tiers (device types) of storage, with in-built compute capability, already deployed in the evaluation environment with applications ported to work on it.
(3) Providing working ecosystem tools (HSM, data integrity checking, performance analysis and debugging, data analytics, programming models, runtimes and visualization utilities) that will be suitable for Extreme scale HPC for multi-tier storage systems such as SAGE.
(4) The first set of applications have now already adapted to use the SAGE platform.
(5) We have opened up the SAGE platform for external communities and this is already actively work in progress that will continue into Sage2.
We now have early stage products (Seagate) and early stage customers (Seagate) based on SAGE outcomes. With dissemination and follow on collaborations we have a clear path for further exploiting SAGE with Sage2 into the European Exascale plans(now incl. AI & Deep Learning). We also also continue to actively provide inputs into EuroHPC through ETP4HPC, based on the learnings from the SAGE projects (SAGE & Sage2).