Periodic Reporting for period 1 - NEO-QE (NEtwork-aware Optimization for Query Executions in Large Systems)
Período documentado: 2018-07-01 hasta 2020-06-30
Efficient execution of query operations is crucial for the overall performance of a data system, and one of the main performance challenges in large distributed environments is the network communications. In the past years, significant performance improvements have been achieved by using state-of-the-art methods, designed in the data management and data communication domain. However, all the proposed techniques in both fields just view each other as a black box, and the additional gains in performance from a co-optimization perspective have not yet been explored. This project aims to bridge the gap and further improve the performance of query execution in large distributed systems.
Data warehouses and data centres are the foundation of computing infrastructure for data analytics. The project aims to improve the performance of one of the core tasks in the scenarios, i.e. query execution. The developed system can either make applications faster or make time-restricted applications complete within deadlines, and consequently big data users will benefit from the designs on making decisions in a more efficient way. In the meantime, using energy-efficient data movement strategies, money cost on power consumption (e.g. by switches and cooling systems) and carbon emissions of the underlying hardware in data centers will be also efficiently reduced. Moreover, since the proposed system is designed on the basis of the coflow model, the proposed techniques on data flow-scheduling will be able to be applied to current big data computing platforms such as Hadoop and Spark, and consequently contribute to the big data community.
The project focuses on mathematically modeling the co-optimization problem between application-level data locality assignments and network-level data communications for query executions, and on that basis to design, development and analysis of a novel query execution system to make sure that a best runtime performance can be achieved. Moreover, to meet query latency/deadline/energy requirements in different application scenarios, bandwidth reserving and routing strategies in complex networks will be also explored. The final developed system will either make big data analytics faster or complete within deadlines, with energy consumption being efficiently reduced.
Data warehouses and data centres are the foundation of computing infrastructure for data analytics. The project aims to improve the performance of one of the core tasks in the scenarios, i.e. query execution. The developed system can either make applications faster or make time-restricted applications complete within deadlines, and consequently big data users will benefit from the designs on making decisions in a more efficient way. In the meantime, using energy-efficient data movement strategies, money cost on power consumption (e.g. by switches and cooling systems) and carbon emissions of the underlying hardware in data centers will be also efficiently reduced. Moreover, since the proposed system is designed on the basis of the coflow model, the proposed techniques on data flow-scheduling will be able to be applied to current big data computing platforms such as Hadoop and Spark, and consequently contribute to the big data community.
The project focuses on mathematically modeling the co-optimization problem between application-level data locality assignments and network-level data communications for query executions, and on that basis to design, development and analysis of a novel query execution system to make sure that a best runtime performance can be achieved. Moreover, to meet query latency/deadline/energy requirements in different application scenarios, bandwidth reserving and routing strategies in complex networks will be also explored. The final developed system will either make big data analytics faster or complete within deadlines, with energy consumption being efficiently reduced.
The project is terminated at the beginning of M15 but not M24. The reason is that the Fellow, Long Cheng, received an offer from Dublin City University as an Assistant Professor in Computing, and he decided to terminate his contract with UCD and the MSCA project “NEO-QE”. As a result, this report for M01- M15 is the final report of the project.
To develop the co-optimization framework (D.1 – M1) for query execution, the Fellow has proposed various data-level scheduling methods for data operators in distributed environments, which lead to a conference and a journal paper. IoT networks with edge servers have been used for the studies of data processing in complex networks (D.3 – M3). One conference paper and one workshop paper have been presented on the basis. In terms of prototype development (D.4.1 and D.4.2) one journal paper has been accepted from the exploration of searching approximate optimal solutions. Moreover, to test and apply the proposed techniques in complex workloads, two conference papers have been presented and one journal paper has been accepted. The Fellow has studied the problem of “multi-query conditions” (D.2 – M2), and one paper has been submitted and now is under review.
Follow the project plan, a publicly accessible website is available for NEO-QE, and the Fellow has organized one workshop on network-aware big data computing. During the fellowship, the Fellow has served as a technical program committee member of six international conferences and a reviewer of several international journals. Additionally, the Fellow also has collaborated with another Lab in UCD and contributed to the teaching and supervision at the Performance Engineering Lab (M4).
To summarize, from M01 to M15, the Fellow has implemented various tasks listed in the NEO-QE project. The main outputs have covered most of the planned deliverables D.1 D.3 D4.1 D4.2 and milestones M1, M3, M4. The relevant results have been published in eight technical papers. Moreover, some newly arising optimization problems in big data computing have also been identified during the development. For example, how to use the proposed framework to handle service-level requirements for query executions in complex computing environments such as Cloud. Another interesting question is that the proposed project and techniques follow the general concept that communication networks are used to move data and ignore the fact that emerging programmable chips have created the opportunity for in-network computing. In this case, it is very interesting to think about how to extend the built models to meet the performance requirements of network communications in future datacentre networks.
To develop the co-optimization framework (D.1 – M1) for query execution, the Fellow has proposed various data-level scheduling methods for data operators in distributed environments, which lead to a conference and a journal paper. IoT networks with edge servers have been used for the studies of data processing in complex networks (D.3 – M3). One conference paper and one workshop paper have been presented on the basis. In terms of prototype development (D.4.1 and D.4.2) one journal paper has been accepted from the exploration of searching approximate optimal solutions. Moreover, to test and apply the proposed techniques in complex workloads, two conference papers have been presented and one journal paper has been accepted. The Fellow has studied the problem of “multi-query conditions” (D.2 – M2), and one paper has been submitted and now is under review.
Follow the project plan, a publicly accessible website is available for NEO-QE, and the Fellow has organized one workshop on network-aware big data computing. During the fellowship, the Fellow has served as a technical program committee member of six international conferences and a reviewer of several international journals. Additionally, the Fellow also has collaborated with another Lab in UCD and contributed to the teaching and supervision at the Performance Engineering Lab (M4).
To summarize, from M01 to M15, the Fellow has implemented various tasks listed in the NEO-QE project. The main outputs have covered most of the planned deliverables D.1 D.3 D4.1 D4.2 and milestones M1, M3, M4. The relevant results have been published in eight technical papers. Moreover, some newly arising optimization problems in big data computing have also been identified during the development. For example, how to use the proposed framework to handle service-level requirements for query executions in complex computing environments such as Cloud. Another interesting question is that the proposed project and techniques follow the general concept that communication networks are used to move data and ignore the fact that emerging programmable chips have created the opportunity for in-network computing. In this case, it is very interesting to think about how to extend the built models to meet the performance requirements of network communications in future datacentre networks.
This is the final report. Based on the four objectives as listed in the project, the results achieved in the project have been documented as nine papers, including eight published papers and one is under review, and the details are below.
In the process of developing the framework, an approach called “Locality-Aware Scheduling (LAS)” has been proposed and presented to minimize network traffic for distributed data operators, and a paper from that basis has been presented at Euro-Par 2018. Moreover, a method on data decomposition aiming for improve load-balancing in outer join execution has been published in Journal of Parallel and Distributed Computing 2019.
To meet service level requirements, the performance model for multi-query executions with bandwidth reservation has been done, and a paper has been submitted to IEEE Transactions on Parallel and Distributed Systems.
Along with the work in framework development, the work on complex networks includes the development of performance models for data transferring and load-balancing in IoT networks with edge servers. To characterize the optimization problem, a deep reinforcement learning based approach has been proposed, and a joint paper has been presented in the NEAC workshop in IEEE/ACM CCGrid 2019. Moreover, to describe the workload patterns from IoT devices, another paper has been presented in conference IEEE Services 2019.
To find approximate optimal solutions for the proposed system, a metaheuristic based on the latest WOA algorithm has been studied and a paper on that basis has been accepted in IEEE Systems Journal 2019. Moreover, as a study on applying real-time scheduling for complex workloads, a joint paper has been presented in the conference ICPP 2019. Additionally, a joint paper on using scheduling in machine learning accelerator has been published in the conference IEEE ASAP 2019.
All the papers can either outperform the state-of-the-art or provide efficient optimization for existing solutions, such as runtime, network traffic, or communication time, for query execution in large systems.
In the process of developing the framework, an approach called “Locality-Aware Scheduling (LAS)” has been proposed and presented to minimize network traffic for distributed data operators, and a paper from that basis has been presented at Euro-Par 2018. Moreover, a method on data decomposition aiming for improve load-balancing in outer join execution has been published in Journal of Parallel and Distributed Computing 2019.
To meet service level requirements, the performance model for multi-query executions with bandwidth reservation has been done, and a paper has been submitted to IEEE Transactions on Parallel and Distributed Systems.
Along with the work in framework development, the work on complex networks includes the development of performance models for data transferring and load-balancing in IoT networks with edge servers. To characterize the optimization problem, a deep reinforcement learning based approach has been proposed, and a joint paper has been presented in the NEAC workshop in IEEE/ACM CCGrid 2019. Moreover, to describe the workload patterns from IoT devices, another paper has been presented in conference IEEE Services 2019.
To find approximate optimal solutions for the proposed system, a metaheuristic based on the latest WOA algorithm has been studied and a paper on that basis has been accepted in IEEE Systems Journal 2019. Moreover, as a study on applying real-time scheduling for complex workloads, a joint paper has been presented in the conference ICPP 2019. Additionally, a joint paper on using scheduling in machine learning accelerator has been published in the conference IEEE ASAP 2019.
All the papers can either outperform the state-of-the-art or provide efficient optimization for existing solutions, such as runtime, network traffic, or communication time, for query execution in large systems.