Skip to main content

BigStorage: Storage-based Convergence between HPC and Cloud to handle Big Data

Deliverables

WP2. Intermediate Report

Report that describes the state-of-the-art of Data Science.

WP1. Requirement description

Report that describes the requirements of the different use cases in detail.

WP5. Intermediate Report on Achieved Milestones

Report that shows the characterization of storage systems and applications in HPC and clouds concerning energy consumption.

WP1. Proposal evaluation

Report providing details of the viability of the BigStorage architecture.

WP3. Intermediate Report

Report that presents a complete overview of current cloud and HPC storage systems.

WP4. Intermediate Report

Report that describes the state-of-the-art related to storage solutions for WP4.

WP4. Final Report

Summarizes activities and results, introducing new techniques for accelerating storage through I/O optimizations.

WP3. HPC / Cloud Convergence – Final

Report that summarizes the final outcome of the WP, presenting a unified architecture for cloud and HPC storage backends.

WP5. Energy savings – Final Report

Report that presents the implementation of energy saving techniques and analyses their impact for datacentres.

WP2. Data Science – Final Report

Report on the final outcome of the WP tasks, presenting new solutions for data transfer and streaming and energy-efficient task scheduling.

WP6. Coordination Management Final Report

Summarizes the final outcome of the coordination and management activities in the network, and the guidelines for future training networks.

WP1. Benchmarks defined & tested

Report that describes the tools and benchmarks studied to emulate BigStorage use cases.

WP7 - Final Report

Summarizes the communication, dissemination and training activities in the network.

Searching for OpenAIRE data...

Publications

Could Blobs Fuel Storage-Based Convergence Between HPC and Big Data?

Author(s): Pierre Matri, Yevhen Alforov, Alvaro Brandon, Michael Kuhn, Philip Carns, Thomas Ludwig
Published in: 2017 IEEE International Conference on Cluster Computing (CLUSTER), 2017, Page(s) 81-86, ISBN 978-1-5386-2326-8
Publisher: IEEE
DOI: 10.1109/CLUSTER.2017.63

HetFS: A Heterogeneous File System for Everyone

Author(s): Georgios Koloventzos, Ramon Nou, Alberto Miranda, Toni Cortes
Published in: ISC High Performance Computing, 2017, Page(s) 691-700, ISBN 978-3-319-67630-2
Publisher: Springer International Publishing
DOI: 10.1007/978-3-319-67630-2_49

Towards a unified storage and ingestion architecture for stream processing

Author(s): Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, Maria S. Perez-Hernandez, Radu Tudoran, Stefano Bortoli, Bogdan Nicolae
Published in: 2017 IEEE International Conference on Big Data (Big Data), 2017, Page(s) 2402-2407, ISBN 978-1-5386-2715-0
Publisher: IEEE
DOI: 10.1109/BigData.2017.8258196

An Empirical Evaluation of How the Network Impacts the Performance and Energy Efficiency in RAMCloud

Author(s): Yacine Taleb, Shadi Ibrahim, Gabriel Antoniu, Toni Cortes
Published in: 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2017, Page(s) 1027-1034, ISBN 978-1-5090-6611-7
Publisher: IEEE
DOI: 10.1109/CCGRID.2017.127

Machine Learning-based Query Augmentation for SPARQL Endpoints

Author(s): Mariano Rico, Rizkallah Touma, Anna Queralt, María S. Pérez
Published in: Proceedings of the 14th International Conference on Web Information Systems and Technologies, 2018, Page(s) 57-67, ISBN 978-989-758-324-7
Publisher: SCITEPRESS - Science and Technology Publications
DOI: 10.5220/0006925300570067

Towards Efficient Location and Placement of Dynamic Replicas for Geo-Distributed Data Stores

Author(s): Pierre Matri, Alexandru Costan, Gabriel Antoniu, Jesús Montes, María S. Pérez
Published in: Proceedings of the ACM 7th Workshop on Scientific Cloud Computing - ScienceCloud '16, 2016, Page(s) 3-9, ISBN 9781-450343534
Publisher: ACM Press
DOI: 10.1145/2913712.2913715

Spark Versus Flink: Understanding Performance in Big Data Analytics Frameworks

Author(s): Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, Maria S. Perez-Hernandez
Published in: 2016 IEEE International Conference on Cluster Computing (CLUSTER), 2016, Page(s) 433-442, ISBN 978-1-5090-3653-0
Publisher: IEEE
DOI: 10.1109/cluster.2016.22

Týr: Blob Storage Meets Built-In Transactions

Author(s): Pierre Matri, Alexandru Costan, Gabriel Antoniu, Jesus Montes, Maria S. Perez
Published in: SC16: International Conference for High Performance Computing, Networking, Storage and Analysis, 2016, Page(s) 573-584, ISBN 978-1-4673-8815-3
Publisher: IEEE
DOI: 10.1109/SC.2016.48

KerA: Scalable Data Ingestion for Stream Processing

Author(s): Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, Maria Perez-Hernandez, Bogdan Nicolae, Radu Tudoran, Stefano Bortoli
Published in: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), 2018, Page(s) 1480-1485, ISBN 978-1-5386-6871-9
Publisher: IEEE
DOI: 10.1109/ICDCS.2018.00152

TýrFS: Increasing Small Files Access Performance with Dynamic Metadata Replication

Author(s): Pierre Matri, Maria S Perez, Alexandru Costan, Gabriel Antoniu
Published in: 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2018, Page(s) 452-461, ISBN 978-1-5386-5815-4
Publisher: IEEE
DOI: 10.1109/CCGRID.2018.00072

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Author(s): Pierre Matri, Philip Carns, Robert Ross, Alexandru Costan, Maria S. Perez, Gabriel Antoniu
Published in: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), 2018, Page(s) 1507-1512, ISBN 978-1-5386-6871-9
Publisher: IEEE
DOI: 10.1109/ICDCS.2018.00156

GekkoFS - A Temporary Distributed File System for HPC Applications

Author(s): Marc-Andre Vef, Nafiseh Moti, Tim SuB, Tommaso Tocci, Ramon Nou, Alberto Miranda, Toni Cortes, Andre Brinkmann
Published in: 2018 IEEE International Conference on Cluster Computing (CLUSTER), 2018, Page(s) 319-324, ISBN 978-1-5386-8319-4
Publisher: IEEE
DOI: 10.1109/CLUSTER.2018.00049

Towards a TRansparent I/O Solution

Author(s): Fotios Nikolaidis, Nick Kossifidis, Thomas Leibovici, Soraya Zertal
Published in: 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2018, Page(s) 1221-1228, ISBN 978-1-5386-5555-9
Publisher: IEEE
DOI: 10.1109/IPDPSW.2018.00189

"Next Stop ""NoOps"": Enabling Cross-System Diagnostics Through Graph-Based Composition of Logs and Metrics"

Author(s): Michal Zasadzinski, Marc Sole, Alvaro Brandon, Victor Muntes-Mulero, David Carrera
Published in: 2018 IEEE International Conference on Cluster Computing (CLUSTER), 2018, Page(s) 212-222, ISBN 978-1-5386-8319-4
Publisher: IEEE
DOI: 10.1109/cluster.2018.00039

Virtual Machine Boot Time Model

Author(s): Thuy Linh Nguyen, Adrien Lebre
Published in: 2017 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP), 2017, Page(s) 430-437, ISBN 978-1-5090-6058-0
Publisher: IEEE
DOI: 10.1109/PDP.2017.58

Actor Based Root Cause Analysis in a Distributed Environment

Author(s): Michal Zasadzinski, Victor Muntes-Mulero, Marc Soe Simo
Published in: 2017 IEEE/ACM 3rd International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS), 2017, Page(s) 14-17, ISBN 978-1-5386-4043-2
Publisher: IEEE
DOI: 10.1109/SEsCPS.2017.3

Characterizing Performance and Energy-Efficiency of the RAMCloud Storage System

Author(s): Yacine Taleb, Shadi Ibrahim, Gabriel Antoniu, Toni Cortes
Published in: 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), 2017, Page(s) 1488-1498, ISBN 978-1-5386-1792-2
Publisher: IEEE
DOI: 10.1109/ICDCS.2017.51

Exploring Shared State in Key-Value Store for Window-Based Multi-pattern Streaming Analytics

Author(s): Ovidiu-Cristian Marcu, Radu Tudoran, Bogdan Nicolae, Alexandru Costan, Gabriel Antoniu, Maria S. Perez-Hernandez
Published in: 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2017, Page(s) 1044-1052, ISBN 978-1-5090-6611-7
Publisher: IEEE
DOI: 10.1109/CCGRID.2017.126

Predicting Access to Persistent Objects through Static Code Analysis

Author(s): Rizkallah Touma, Anna Queralt, Toni Cortes, María S. Pérez
Published in: New Trends in Databases and Information Systems (ADBIS 2017), 2017, Page(s) 54-62, ISBN 978-3-319-67161-1
Publisher: Springer International Publishing
DOI: 10.1007/978-3-319-67162-8_7

A Simulation Optimisation Tool and Its Production/Inventory Control Application

Author(s): D. Katsios, A. S. Xanthopoulos, D. E. Koulouriotis, A. Kiatipis
Published in: International Journal of Simulation Modelling, 17/2, 2018, Page(s) 257-270, ISSN 1726-4529
Publisher: DAAAM International Vienna
DOI: 10.2507/ijsimm17(2)425

Using machine learning to optimize parallelism in big data applications

Author(s): Álvaro Brandón Hernández, María S. Perez, Smrati Gupta, Victor Muntés-Mulero
Published in: Future Generation Computer Systems, 86, 2018, Page(s) 1076-1092, ISSN 0167-739X
Publisher: Elsevier BV
DOI: 10.1016/j.future.2017.07.003

Survey of Storage Systems for High-Performance Computing

Author(s): Jakob Lüttgau, Michael Kuhn, Kira Duwe, Yevhen Alforov, Eugen Betke, Julian Kunkel, Thomas Ludwig
Published in: Supercomputing Frontiers and Innovations, 5/1, 2018, Page(s) 31–58, ISSN 2313-8734
Publisher: Publishing Center of South Ural State University
DOI: 10.14529/jsfi180103

Keeping up with storage: Decentralized, write-enabled dynamic geo-replication

Author(s): Pierre Matri, María S. Pérez, Alexandru Costan, Luc Bougé, Gabriel Antoniu
Published in: Future Generation Computer Systems, 86, 2018, Page(s) 1093-1105, ISSN 0167-739X
Publisher: Elsevier BV
DOI: 10.1016/j.future.2017.06.009

FMonE: A Flexible Monitoring Solution at the Edge

Author(s): Álvaro Brandón, María S. Pérez, Jesus Montes, Alberto Sanchez
Published in: Wireless Communications and Mobile Computing, 2018, 2018, Page(s) 1-15, ISSN 1530-8669
Publisher: John Wiley & Sons Inc.
DOI: 10.1155/2018/2068278

Reinforcement Learning-Based and Parametric Production-Maintenance Control Policies for a Deteriorating Manufacturing System

Author(s): A. S. Xanthopoulos, Athanasios Kiatipis, D. E. Koulouriotis, Sepp Stieger
Published in: IEEE Access, 6, 2018, Page(s) 576-588, ISSN 2169-3536
Publisher: Institute of Electrical and Electronics Engineers Inc.
DOI: 10.1109/ACCESS.2017.2771827

Early Termination of Failed HPC Jobs Through Machine and Deep Learning

Author(s): Michał Zasadziński, Victor Muntés-Mulero, Marc Solé, David Carrera, Thomas Ludwig
Published in: Euro-Par 2018: Parallel Processing - 24th International Conference on Parallel and Distributed Computing, Turin, Italy, August 27 - 31, 2018, Proceedings, 11014, 2018, Page(s) 163-177, ISBN 978-3-319-96982-4
Publisher: Springer International Publishing
DOI: 10.1007/978-3-319-96983-1_12