BigStorage: Storage-based Convergence between HPC and Cloud to handle Big Data

Información del proyecto

BigStorage

Identificador del acuerdo de subvención: 642963

DOI

10.3030/642963

Proyecto cerrado

Fecha de la firma de la CE 16 Diciembre 2014

Fecha de inicio 1 Enero 2015

Fecha de finalización 31 Diciembre 2018

Financiado con arreglo a

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Coste total

€ 3 803 407,92

Aportación de la UE

€ 3 803 407,92

Coordinado por

UNIVERSIDAD POLITECNICA DE MADRID
Spain

CORDIS proporciona enlaces a los documentos públicos y las publicaciones de los proyectos de los programas marco HORIZONTE.

Los enlaces a los documentos y las publicaciones de los proyectos del Séptimo Programa Marco, así como los enlaces a algunos tipos de resultados específicos, como conjuntos de datos y «software», se obtienen dinámicamente de OpenAIRE .

Resultado final

Documents, reports (10)

WP2. Intermediate Report

Report that describes the state-of-the-art of Data Science.

WP1. Requirement description

Report that describes the requirements of the different use cases in detail.

WP5. Intermediate Report on Achieved Milestones

Report that shows the characterization of storage systems and applications in HPC and clouds concerning energy consumption.

WP1. Proposal evaluation

Report providing details of the viability of the BigStorage architecture.

WP3. Intermediate Report

Report that presents a complete overview of current cloud and HPC storage systems.

WP4. Intermediate Report

Report that describes the state-of-the-art related to storage solutions for WP4.

WP4. Final Report

Summarizes activities and results, introducing new techniques for accelerating storage through I/O optimizations.

WP3. HPC / Cloud Convergence – Final

Report that summarizes the final outcome of the WP, presenting a unified architecture for cloud and HPC storage backends.

WP5. Energy savings – Final Report

Report that presents the implementation of energy saving techniques and analyses their impact for datacentres.

WP2. Data Science – Final Report

Report on the final outcome of the WP tasks, presenting new solutions for data transfer and streaming and energy-efficient task scheduling.

Other (2)

WP6. Coordination Management Final Report

Summarizes the final outcome of the coordination and management activities in the network, and the guidelines for future training networks.

WP1. Benchmarks defined & tested

Report that describes the tools and benchmarks studied to emulate BigStorage use cases.

Websites, patent fillings, videos etc. (1)

WP7 - Final Report

Summarizes the communication, dissemination and training activities in the network.

Publicaciones

Conference proceedings (19)

Could Blobs Fuel Storage-Based Convergence Between HPC and Big Data?

Autores: Pierre Matri, Yevhen Alforov, Alvaro Brandon, Michael Kuhn, Philip Carns, Thomas Ludwig
Publicado en: 2017 IEEE International Conference on Cluster Computing (CLUSTER), 2017, Página(s) 81-86, ISBN 978-1-5386-2326-8
Editor: IEEE
DOI: 10.1109/CLUSTER.2017.63

HetFS: A Heterogeneous File System for Everyone

Autores: Georgios Koloventzos, Ramon Nou, Alberto Miranda, Toni Cortes
Publicado en: ISC High Performance Computing, 2017, Página(s) 691-700, ISBN 978-3-319-67630-2
Editor: Springer International Publishing
DOI: 10.1007/978-3-319-67630-2_49

Towards a unified storage and ingestion architecture for stream processing

Autores: Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, Maria S. Perez-Hernandez, Radu Tudoran, Stefano Bortoli, Bogdan Nicolae
Publicado en: 2017 IEEE International Conference on Big Data (Big Data), 2017, Página(s) 2402-2407, ISBN 978-1-5386-2715-0
Editor: IEEE
DOI: 10.1109/BigData.2017.8258196

An Empirical Evaluation of How the Network Impacts the Performance and Energy Efficiency in RAMCloud

Autores: Yacine Taleb, Shadi Ibrahim, Gabriel Antoniu, Toni Cortes
Publicado en: 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2017, Página(s) 1027-1034, ISBN 978-1-5090-6611-7
Editor: IEEE
DOI: 10.1109/CCGRID.2017.127

Machine Learning-based Query Augmentation for SPARQL Endpoints

Autores: Mariano Rico, Rizkallah Touma, Anna Queralt, María S. Pérez
Publicado en: Proceedings of the 14th International Conference on Web Information Systems and Technologies, 2018, Página(s) 57-67, ISBN 978-989-758-324-7
Editor: SCITEPRESS - Science and Technology Publications
DOI: 10.5220/0006925300570067

Towards Efficient Location and Placement of Dynamic Replicas for Geo-Distributed Data Stores

Autores: Pierre Matri, Alexandru Costan, Gabriel Antoniu, Jesús Montes, María S. Pérez
Publicado en: Proceedings of the ACM 7th Workshop on Scientific Cloud Computing - ScienceCloud '16, 2016, Página(s) 3-9, ISBN 9781-450343534
Editor: ACM Press
DOI: 10.1145/2913712.2913715

Spark Versus Flink: Understanding Performance in Big Data Analytics Frameworks

Autores: Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, Maria S. Perez-Hernandez
Publicado en: 2016 IEEE International Conference on Cluster Computing (CLUSTER), 2016, Página(s) 433-442, ISBN 978-1-5090-3653-0
Editor: IEEE
DOI: 10.1109/cluster.2016.22

Týr: Blob Storage Meets Built-In Transactions

Autores: Pierre Matri, Alexandru Costan, Gabriel Antoniu, Jesus Montes, Maria S. Perez
Publicado en: SC16: International Conference for High Performance Computing, Networking, Storage and Analysis, 2016, Página(s) 573-584, ISBN 978-1-4673-8815-3
Editor: IEEE
DOI: 10.1109/SC.2016.48

KerA: Scalable Data Ingestion for Stream Processing

Autores: Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, Maria Perez-Hernandez, Bogdan Nicolae, Radu Tudoran, Stefano Bortoli
Publicado en: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), 2018, Página(s) 1480-1485, ISBN 978-1-5386-6871-9
Editor: IEEE
DOI: 10.1109/ICDCS.2018.00152

TýrFS: Increasing Small Files Access Performance with Dynamic Metadata Replication

Autores: Pierre Matri, Maria S Perez, Alexandru Costan, Gabriel Antoniu
Publicado en: 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2018, Página(s) 452-461, ISBN 978-1-5386-5815-4
Editor: IEEE
DOI: 10.1109/CCGRID.2018.00072

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Autores: Pierre Matri, Philip Carns, Robert Ross, Alexandru Costan, Maria S. Perez, Gabriel Antoniu
Publicado en: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), 2018, Página(s) 1507-1512, ISBN 978-1-5386-6871-9
Editor: IEEE
DOI: 10.1109/ICDCS.2018.00156

GekkoFS - A Temporary Distributed File System for HPC Applications

Autores: Marc-Andre Vef, Nafiseh Moti, Tim SuB, Tommaso Tocci, Ramon Nou, Alberto Miranda, Toni Cortes, Andre Brinkmann
Publicado en: 2018 IEEE International Conference on Cluster Computing (CLUSTER), 2018, Página(s) 319-324, ISBN 978-1-5386-8319-4
Editor: IEEE
DOI: 10.1109/CLUSTER.2018.00049

Towards a TRansparent I/O Solution

Autores: Fotios Nikolaidis, Nick Kossifidis, Thomas Leibovici, Soraya Zertal
Publicado en: 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2018, Página(s) 1221-1228, ISBN 978-1-5386-5555-9
Editor: IEEE
DOI: 10.1109/IPDPSW.2018.00189

"Next Stop ""NoOps"": Enabling Cross-System Diagnostics Through Graph-Based Composition of Logs and Metrics"

Autores: Michal Zasadzinski, Marc Sole, Alvaro Brandon, Victor Muntes-Mulero, David Carrera
Publicado en: 2018 IEEE International Conference on Cluster Computing (CLUSTER), 2018, Página(s) 212-222, ISBN 978-1-5386-8319-4
Editor: IEEE
DOI: 10.1109/cluster.2018.00039

Virtual Machine Boot Time Model

Autores: Thuy Linh Nguyen, Adrien Lebre
Publicado en: 2017 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP), 2017, Página(s) 430-437, ISBN 978-1-5090-6058-0
Editor: IEEE
DOI: 10.1109/PDP.2017.58

Actor Based Root Cause Analysis in a Distributed Environment

Autores: Michal Zasadzinski, Victor Muntes-Mulero, Marc Soe Simo
Publicado en: 2017 IEEE/ACM 3rd International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS), 2017, Página(s) 14-17, ISBN 978-1-5386-4043-2
Editor: IEEE
DOI: 10.1109/SEsCPS.2017.3

Characterizing Performance and Energy-Efficiency of the RAMCloud Storage System

Autores: Yacine Taleb, Shadi Ibrahim, Gabriel Antoniu, Toni Cortes
Publicado en: 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), 2017, Página(s) 1488-1498, ISBN 978-1-5386-1792-2
Editor: IEEE
DOI: 10.1109/ICDCS.2017.51

Exploring Shared State in Key-Value Store for Window-Based Multi-pattern Streaming Analytics

Autores: Ovidiu-Cristian Marcu, Radu Tudoran, Bogdan Nicolae, Alexandru Costan, Gabriel Antoniu, Maria S. Perez-Hernandez
Publicado en: 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2017, Página(s) 1044-1052, ISBN 978-1-5090-6611-7
Editor: IEEE
DOI: 10.1109/CCGRID.2017.126

Predicting Access to Persistent Objects through Static Code Analysis

Autores: Rizkallah Touma, Anna Queralt, Toni Cortes, María S. Pérez
Publicado en: New Trends in Databases and Information Systems (ADBIS 2017), 2017, Página(s) 54-62, ISBN 978-3-319-67161-1
Editor: Springer International Publishing
DOI: 10.1007/978-3-319-67162-8_7

Peer reviewed articles (6)

A Simulation Optimisation Tool and Its Production/Inventory Control Application

Autores: D. Katsios, A. S. Xanthopoulos, D. E. Koulouriotis, A. Kiatipis
Publicado en: International Journal of Simulation Modelling, Edición 17/2, 2018, Página(s) 257-270, ISSN 1726-4529
Editor: DAAAM International Vienna
DOI: 10.2507/ijsimm17(2)425

Using machine learning to optimize parallelism in big data applications

Autores: Álvaro Brandón Hernández, María S. Perez, Smrati Gupta, Victor Muntés-Mulero
Publicado en: Future Generation Computer Systems, Edición 86, 2018, Página(s) 1076-1092, ISSN 0167-739X
Editor: Elsevier BV
DOI: 10.1016/j.future.2017.07.003

Survey of Storage Systems for High-Performance Computing

Autores: Jakob Lüttgau, Michael Kuhn, Kira Duwe, Yevhen Alforov, Eugen Betke, Julian Kunkel, Thomas Ludwig
Publicado en: Supercomputing Frontiers and Innovations, Edición 5/1, 2018, Página(s) 31–58, ISSN 2313-8734
Editor: Publishing Center of South Ural State University
DOI: 10.14529/jsfi180103

Keeping up with storage: Decentralized, write-enabled dynamic geo-replication

Autores: Pierre Matri, María S. Pérez, Alexandru Costan, Luc Bougé, Gabriel Antoniu
Publicado en: Future Generation Computer Systems, Edición 86, 2018, Página(s) 1093-1105, ISSN 0167-739X
Editor: Elsevier BV
DOI: 10.1016/j.future.2017.06.009

FMonE: A Flexible Monitoring Solution at the Edge

Autores: Álvaro Brandón, María S. Pérez, Jesus Montes, Alberto Sanchez
Publicado en: Wireless Communications and Mobile Computing, Edición 2018, 2018, Página(s) 1-15, ISSN 1530-8669
Editor: John Wiley & Sons Inc.
DOI: 10.1155/2018/2068278

Reinforcement Learning-Based and Parametric Production-Maintenance Control Policies for a Deteriorating Manufacturing System

Autores: A. S. Xanthopoulos, Athanasios Kiatipis, D. E. Koulouriotis, Sepp Stieger
Publicado en: IEEE Access, Edición 6, 2018, Página(s) 576-588, ISSN 2169-3536
Editor: Institute of Electrical and Electronics Engineers Inc.
DOI: 10.1109/ACCESS.2017.2771827

Book chapters (1)

Early Termination of Failed HPC Jobs Through Machine and Deep Learning

Autores: Michał Zasadziński, Victor Muntés-Mulero, Marc Solé, David Carrera, Thomas Ludwig
Publicado en: Euro-Par 2018: Parallel Processing - 24th International Conference on Parallel and Distributed Computing, Turin, Italy, August 27 - 31, 2018, Proceedings, Edición 11014, 2018, Página(s) 163-177, ISBN 978-3-319-96982-4
Editor: Springer International Publishing
DOI: 10.1007/978-3-319-96983-1_12

Derechos de propiedad intelectual

Patent (1)

MODEL BASED ROOT CAUSE ANALYSIS

Número de solicitud/publicación: US 20170372212A1
Fecha: 2016-06-28
Solicitante(s): CA TECHNOLOGIES DEVELOPMENT SPAIN SA