Skip to main content

BigStorage: Storage-based Convergence between HPC and Cloud to handle Big Data

Deliverables

WP2. Intermediate Report

Report that describes the state-of-the-art of Data Science.

WP1. Requirement description

Report that describes the requirements of the different use cases in detail.

WP5. Intermediate Report on Achieved Milestones

Report that shows the characterization of storage systems and applications in HPC and clouds concerning energy consumption.

WP1. Proposal evaluation

Report providing details of the viability of the BigStorage architecture.

WP3. Intermediate Report

Report that presents a complete overview of current cloud and HPC storage systems.

WP4. Intermediate Report

Report that describes the state-of-the-art related to storage solutions for WP4.

WP4. Final Report

Summarizes activities and results, introducing new techniques for accelerating storage through I/O optimizations.

WP3. HPC / Cloud Convergence – Final

Report that summarizes the final outcome of the WP, presenting a unified architecture for cloud and HPC storage backends.

WP5. Energy savings – Final Report

Report that presents the implementation of energy saving techniques and analyses their impact for datacentres.

WP2. Data Science – Final Report

Report on the final outcome of the WP tasks, presenting new solutions for data transfer and streaming and energy-efficient task scheduling.

WP6. Coordination Management Final Report

Summarizes the final outcome of the coordination and management activities in the network, and the guidelines for future training networks.

WP1. Benchmarks defined & tested

Report that describes the tools and benchmarks studied to emulate BigStorage use cases.

WP7 - Final Report

Summarizes the communication, dissemination and training activities in the network.

Publications

Could Blobs Fuel Storage-Based Convergence Between HPC and Big Data?

Author(s): Pierre Matri, Yevhen Alforov, Alvaro Brandon, Michael Kuhn, Philip Carns, Thomas Ludwig
Published in: 2017 IEEE International Conference on Cluster Computing (CLUSTER), 2017, Page(s) 81-86
DOI: 10.1109/CLUSTER.2017.63

HetFS: A Heterogeneous File System for Everyone

Author(s): Georgios Koloventzos, Ramon Nou, Alberto Miranda, Toni Cortes
Published in: ISC High Performance Computing, 2017, Page(s) 691-700
DOI: 10.1007/978-3-319-67630-2_49

Towards a unified storage and ingestion architecture for stream processing

Author(s): Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, Maria S. Perez-Hernandez, Radu Tudoran, Stefano Bortoli, Bogdan Nicolae
Published in: 2017 IEEE International Conference on Big Data (Big Data), 2017, Page(s) 2402-2407
DOI: 10.1109/BigData.2017.8258196

An Empirical Evaluation of How the Network Impacts the Performance and Energy Efficiency in RAMCloud

Author(s): Yacine Taleb, Shadi Ibrahim, Gabriel Antoniu, Toni Cortes
Published in: 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2017, Page(s) 1027-1034
DOI: 10.1109/CCGRID.2017.127

Machine Learning-based Query Augmentation for SPARQL Endpoints

Author(s): Mariano Rico, Rizkallah Touma, Anna Queralt, María S. Pérez
Published in: Proceedings of the 14th International Conference on Web Information Systems and Technologies, 2018, Page(s) 57-67
DOI: 10.5220/0006925300570067

Towards Efficient Location and Placement of Dynamic Replicas for Geo-Distributed Data Stores

Author(s): Pierre Matri, Alexandru Costan, Gabriel Antoniu, Jesús Montes, María S. Pérez
Published in: Proceedings of the ACM 7th Workshop on Scientific Cloud Computing - ScienceCloud '16, 2016, Page(s) 3-9
DOI: 10.1145/2913712.2913715

Spark Versus Flink: Understanding Performance in Big Data Analytics Frameworks

Author(s): Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, Maria S. Perez-Hernandez
Published in: 2016 IEEE International Conference on Cluster Computing (CLUSTER), 2016, Page(s) 433-442
DOI: 10.1109/cluster.2016.22

Týr: Blob Storage Meets Built-In Transactions

Author(s): Pierre Matri, Alexandru Costan, Gabriel Antoniu, Jesus Montes, Maria S. Perez
Published in: SC16: International Conference for High Performance Computing, Networking, Storage and Analysis, 2016, Page(s) 573-584
DOI: 10.1109/SC.2016.48

KerA: Scalable Data Ingestion for Stream Processing

Author(s): Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, Maria Perez-Hernandez, Bogdan Nicolae, Radu Tudoran, Stefano Bortoli
Published in: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), 2018, Page(s) 1480-1485
DOI: 10.1109/ICDCS.2018.00152

TýrFS: Increasing Small Files Access Performance with Dynamic Metadata Replication

Author(s): Pierre Matri, Maria S Perez, Alexandru Costan, Gabriel Antoniu
Published in: 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2018, Page(s) 452-461
DOI: 10.1109/CCGRID.2018.00072

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Author(s): Pierre Matri, Philip Carns, Robert Ross, Alexandru Costan, Maria S. Perez, Gabriel Antoniu
Published in: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), 2018, Page(s) 1507-1512
DOI: 10.1109/ICDCS.2018.00156

GekkoFS - A Temporary Distributed File System for HPC Applications

Author(s): Marc-Andre Vef, Nafiseh Moti, Tim SuB, Tommaso Tocci, Ramon Nou, Alberto Miranda, Toni Cortes, Andre Brinkmann
Published in: 2018 IEEE International Conference on Cluster Computing (CLUSTER), 2018, Page(s) 319-324
DOI: 10.1109/CLUSTER.2018.00049

Towards a TRansparent I/O Solution

Author(s): Fotios Nikolaidis, Nick Kossifidis, Thomas Leibovici, Soraya Zertal
Published in: 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2018, Page(s) 1221-1228
DOI: 10.1109/IPDPSW.2018.00189

"Next Stop ""NoOps"": Enabling Cross-System Diagnostics Through Graph-Based Composition of Logs and Metrics"

Author(s): Michal Zasadzinski, Marc Sole, Alvaro Brandon, Victor Muntes-Mulero, David Carrera
Published in: 2018 IEEE International Conference on Cluster Computing (CLUSTER), 2018, Page(s) 212-222
DOI: 10.1109/cluster.2018.00039

Virtual Machine Boot Time Model

Author(s): Thuy Linh Nguyen, Adrien Lebre
Published in: 2017 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP), 2017, Page(s) 430-437
DOI: 10.1109/PDP.2017.58

Actor Based Root Cause Analysis in a Distributed Environment

Author(s): Michal Zasadzinski, Victor Muntes-Mulero, Marc Soe Simo
Published in: 2017 IEEE/ACM 3rd International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS), 2017, Page(s) 14-17
DOI: 10.1109/SEsCPS.2017.3

Characterizing Performance and Energy-Efficiency of the RAMCloud Storage System

Author(s): Yacine Taleb, Shadi Ibrahim, Gabriel Antoniu, Toni Cortes
Published in: 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), 2017, Page(s) 1488-1498
DOI: 10.1109/ICDCS.2017.51

Exploring Shared State in Key-Value Store for Window-Based Multi-pattern Streaming Analytics

Author(s): Ovidiu-Cristian Marcu, Radu Tudoran, Bogdan Nicolae, Alexandru Costan, Gabriel Antoniu, Maria S. Perez-Hernandez
Published in: 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2017, Page(s) 1044-1052
DOI: 10.1109/CCGRID.2017.126

Predicting Access to Persistent Objects through Static Code Analysis

Author(s): Rizkallah Touma, Anna Queralt, Toni Cortes, María S. Pérez
Published in: New Trends in Databases and Information Systems (ADBIS 2017), 2017, Page(s) 54-62
DOI: 10.1007/978-3-319-67162-8_7

A Simulation Optimisation Tool and Its Production/Inventory Control Application

Author(s): D. Katsios, A. S. Xanthopoulos, D. E. Koulouriotis, A. Kiatipis
Published in: International Journal of Simulation Modelling, Issue 17/2, 2018, Page(s) 257-270, ISSN 1726-4529
DOI: 10.2507/ijsimm17(2)425

Using machine learning to optimize parallelism in big data applications

Author(s): Álvaro Brandón Hernández, María S. Perez, Smrati Gupta, Victor Muntés-Mulero
Published in: Future Generation Computer Systems, Issue 86, 2018, Page(s) 1076-1092, ISSN 0167-739X
DOI: 10.1016/j.future.2017.07.003

Survey of Storage Systems for High-Performance Computing

Author(s): Jakob Lüttgau, Michael Kuhn, Kira Duwe, Yevhen Alforov, Eugen Betke, Julian Kunkel, Thomas Ludwig
Published in: Supercomputing Frontiers and Innovations, Issue 5/1, 2018, Page(s) 31–58, ISSN 2313-8734
DOI: 10.14529/jsfi180103

Keeping up with storage: Decentralized, write-enabled dynamic geo-replication

Author(s): Pierre Matri, María S. Pérez, Alexandru Costan, Luc Bougé, Gabriel Antoniu
Published in: Future Generation Computer Systems, Issue 86, 2018, Page(s) 1093-1105, ISSN 0167-739X
DOI: 10.1016/j.future.2017.06.009

FMonE: A Flexible Monitoring Solution at the Edge


Published in: ISSN 1530-8669
DOI: 10.1155/2018/2068278

Reinforcement Learning-Based and Parametric Production-Maintenance Control Policies for a Deteriorating Manufacturing System

Author(s): A. S. Xanthopoulos, Athanasios Kiatipis, D. E. Koulouriotis, Sepp Stieger
Published in: IEEE Access, Issue 6, 2018, Page(s) 576-588, ISSN 2169-3536
DOI: 10.1109/ACCESS.2017.2771827

Early Termination of Failed HPC Jobs Through Machine and Deep Learning

Author(s): Michał Zasadziński, Victor Muntés-Mulero, Marc Solé, David Carrera, Thomas Ludwig
Published in: Euro-Par 2018: Parallel Processing - 24th International Conference on Parallel and Distributed Computing, Turin, Italy, August 27 - 31, 2018, Proceedings, Issue 11014, 2018, Page(s) 163-177
DOI: 10.1007/978-3-319-96983-1_12