Skip to main content

Middleware for memory and data-awareness in workflows

Deliverables

Storage, Data Mapping and Memory Integration Design

This will provide architecture of the storage and memory access backend used for Maestro. This deliverable will also cover the design of data object mapping as per the storage integration design.

Initial Specification of Workflow Management and Optimisation

This report provides an initial specification of the workflow management and optimisation developed in T44

Initial Core Middleware specification, API document

An API and usage document based on the requirements of the applications provided by WP2. This document will serve as a review vehicle for further feedback from applications and workflows.

Innovation Management and Exploitation Plan

This report will document the strategy for innovation management and exploitation and comprise a detailed plan for its implementation.

Middleware Requirements and API design

A report containing an updated description of the existing HPC workloads within the project, and describing the requirements they impose on the middleware and API designs, and proposing a resultant API design.

Telemetry Design

A report describing the design of extensions of SelFIe and Mero

Report on quality control procedure

Definition of the quality control processes and templates for internal verification and document review for all project results and deliverables. This will be a living document that is updated throughout the project when necessary.

Workload Characterisation and Middleware Requirements

An initial overview of the existing HPC workloads for applications within the project, and the requirements they impose on the middleware and API designs.

Applications demonstration description

A report describing the selected applications and how they will be used in the demonstration of the Maestro development

Adaptive Transport Design

A specification, API and usage document describing proposed transport mechanisms

Dissemination Plan

This report will document the strategy for dissemination and provide a detailed plan for its implementation.

Systems software test description

A report describing the tests objectives

Report on dissemination, innovation management and exploitation as well as standardisation

In this deliverable a summary of all activities and results related to dissemination innovation management and exploitation as well as standardisation will be provided

Execution Framework Prototype

Prototype for deployment of workflows and containerised data models on a highend HPC platform

Workflow Management and Optimisation Final Release

Demonstrator for higherlevel toolsets for workflow management and optimisation

Storage, Data Mapping and Memory Integration Implementation for Maestro

This task will provide the executables and source code as appropriate This will also include the executables and source code for the object data mapping

Adaptive Transport Release

Full release of the transport mechanisms.

Telemetry demonstration

A demonstration and a report on the telemetry tools and how they can be used with the Maestro applications

Transformations and Map optimisation release

A report and release of the set of maps layout transformations and other lowlevel operations in collaboration with the Highlevel Middleware work package WP4

Core Middleware reference implementation

A reference implementation of the core middleware with the central CDM infrastructure, MSM, LDO and PDO implementations, with all features functional, though without performance optimisations or consideration to advanced usage.

Workflow Models Prototype

An initial version of the model workflows that will be used in middleware evaluation.

Telemetry Prototype

The source code of SelFIe and Mero monitoring prototypes

Full Core Middleware release

A full and final release of the core data model the Memory system model LDO and PDO implementations as well as a tuned reference implementation designed to facilitate development of systems software demonstrations

Final Workflow Models

An updated version of the model workflows that will be used in middleware evaluation

System software demonstrators

The source code used to validate the framework and a report describing the tests

Project Website

Provisioning of a web presence of the Maestro project as major channel for dissemination.

Searching for OpenAIRE data...

Publications

Dynamic Provisioning of Storage Resources: A Case Study with Burst Buffers

Author(s): Francois Tessier; Maxime Martinasso; Matteo Chesi; Mark Klein; Miguel Gila
Published in: 2020, ISBN 978-1-7281-7457-0
Publisher: 2020 IEEE International Parallel and Distributed Processing Symposium Workshops
DOI: 10.1109/ipdpsw50202.2020.00173

Dynamically Provisioning Cray DataWarp Storage

Author(s): François Tessier, Maxime Martinasso, Matteo Chesi, Mark Klein, Miguel Gila
Published in: 2019
Publisher: Cray User Group

A Middleware Supporting Data Movement in Complex and Software-Defined Storage and Memory Architectures

Author(s): Christopher Haine, Utz-Uwe Haus, Maxime Martinasso, Dirk Pleiter, François Tessier, Domokos Sarmany, Simon Smart, Tiago Quintino, Adrian Tate
Published in: 2021
Publisher: SuperCompCloud: 4th Workshop on Interoperability of Supercomputing and Cloud Technologies

Parallelware Tools: An Experimental Evaluation on POWER Systems

Author(s): Manuel Arenaz, Xavier Martorell
Published in: High Performance Computing - ISC High Performance 2019 International Workshops, Frankfurt, Germany, June 16-20, 2019, Revised Selected Papers, 11887, 2019, Page(s) 352-360, ISBN 978-3-030-34355-2
Publisher: Springer International Publishing
DOI: 10.1007/978-3-030-34356-9_27