Periodic Reporting for period 2 - ACTiCLOUD (ACTivating resource efficiency and large databases in the CLOUD)
Période du rapport: 2018-07-01 au 2019-12-31
Despite their proliferation as a dominant computing paradigm, cloud computing systems lack effective mechanisms to manage their vast amounts of resources efficiently, leading to severe resource waste and ultimately limiting their applicability to large classes of critical applications that pose non-moderate resource demands. This creates a significant need to lift existing technological barriers of actual fluidity and scalability of cloud resources towards promoting cloud computing as a critical cornerstone for digital economy.
ACTiCLOUD proposes a novel cloud computing architecture for drastically improved management of cloud resources, achieving 1.5x increase in resource efficiency and more than 10x in scalability. ACTiCLOUD successfully developed and integrated architectures and components that: (i) improve the effective utilization of cloud resources, targeting resource efficiency and performance stability, and (ii) enable the deployment of resource demanding applications in the cloud, targeting scalability and elasticity in resource provisioning.
To achieve its goals, ACTiCLOUD took a holistic approach working across all layers of the computing stack. By utilizing modest investments on hardware intelligence that enables true resource disaggregation between multiple servers, we progressed current state-of-the-art in hypervisors, middleware, and cloud management systems, promoting holistic resource management both at the rack scale and across distributed cloud sites. On top of this, we evolved the ecosystem around in-memory databases to provide cost-efficient and highly performant DataBase-as-a-Service (DBaaS) cloud platforms. With the ACTiCLOUD architecture, the project has provided a novel cloud computing architecture that drastically improves the management and utilization of cloud resources while making important steps towards bringing resource-demanding, in-memory databases to the cloud.
ACTiCLOUD proposes a novel cloud computing architecture for drastically improved management of cloud resources, achieving 1.5x increase in resource efficiency and more than 10x in scalability. ACTiCLOUD successfully developed and integrated architectures and components that: (i) improve the effective utilization of cloud resources, targeting resource efficiency and performance stability, and (ii) enable the deployment of resource demanding applications in the cloud, targeting scalability and elasticity in resource provisioning.
To achieve its goals, ACTiCLOUD took a holistic approach working across all layers of the computing stack. By utilizing modest investments on hardware intelligence that enables true resource disaggregation between multiple servers, we progressed current state-of-the-art in hypervisors, middleware, and cloud management systems, promoting holistic resource management both at the rack scale and across distributed cloud sites. On top of this, we evolved the ecosystem around in-memory databases to provide cost-efficient and highly performant DataBase-as-a-Service (DBaaS) cloud platforms. With the ACTiCLOUD architecture, the project has provided a novel cloud computing architecture that drastically improves the management and utilization of cloud resources while making important steps towards bringing resource-demanding, in-memory databases to the cloud.
The project has successfully met its objectives.
In the first period the project focus was to: a) To set the basis for the successful implementation of the ACTiCLOUD system by defining its architecture, b) To perform an initial integration scenario that would bring together all the critical components of the system, i.e. the hardware platforms (Numascale and KMAX systems), the virtualization layer (Microvisor), the cloud manager (Openstack), the language runtime (JVM) and the two database frameworks (MonetDB and Neo4j) and c) To implement initial versions of the system modules, including ACTiManager (the cloud management plug-in that incorporates intelligence to resource management), the extended Microvisor, system libraries, hyperscale JVM and cloudified MonetDB and Neo4j.
The focus of the project in the second period was to: a) to optimize all ACTiCLOUD software components, b) to perform various integration scenarios using multiple ACTiCLOUD components to form the ACTiCLOUD final prototypes, and c) to extensively evaluate the ACTiCLOUD components and the prototypes.
An overview of the results across the entire computing stack include the integration of the embraced hardware platforms, the rack-scale hypervisor, the system libraries, the hyperscale JVM, the ACTiManager distributed cloud resource manager, and the enhancements for MonetDB and Neo4j towards their cloudification.
Our evaluation results show that the final ACTiCLOUD prototype manages to increase the effective utilization of resources by increased resource efficiency and performance stability, and to enable the deployment of resource demanding applications such as in-memory and graph databases in the cloud through support for scalability and elasticity in resource provisioning.
Together with the technical work, the consortium maximized its effort to increase the impact of the project. To this direction, we disseminated our project concepts, objectives, and results in 22 research publications, established our presence in the web and social media (https://twitter.com/ACTiCLOUD https://www.researchgate.net/project/ACTiCLOUD) co-organized two scientific workshop and gave several talks and presented demos and posters in scientific conferences and workshops.
In the first period the project focus was to: a) To set the basis for the successful implementation of the ACTiCLOUD system by defining its architecture, b) To perform an initial integration scenario that would bring together all the critical components of the system, i.e. the hardware platforms (Numascale and KMAX systems), the virtualization layer (Microvisor), the cloud manager (Openstack), the language runtime (JVM) and the two database frameworks (MonetDB and Neo4j) and c) To implement initial versions of the system modules, including ACTiManager (the cloud management plug-in that incorporates intelligence to resource management), the extended Microvisor, system libraries, hyperscale JVM and cloudified MonetDB and Neo4j.
The focus of the project in the second period was to: a) to optimize all ACTiCLOUD software components, b) to perform various integration scenarios using multiple ACTiCLOUD components to form the ACTiCLOUD final prototypes, and c) to extensively evaluate the ACTiCLOUD components and the prototypes.
An overview of the results across the entire computing stack include the integration of the embraced hardware platforms, the rack-scale hypervisor, the system libraries, the hyperscale JVM, the ACTiManager distributed cloud resource manager, and the enhancements for MonetDB and Neo4j towards their cloudification.
Our evaluation results show that the final ACTiCLOUD prototype manages to increase the effective utilization of resources by increased resource efficiency and performance stability, and to enable the deployment of resource demanding applications such as in-memory and graph databases in the cloud through support for scalability and elasticity in resource provisioning.
Together with the technical work, the consortium maximized its effort to increase the impact of the project. To this direction, we disseminated our project concepts, objectives, and results in 22 research publications, established our presence in the web and social media (https://twitter.com/ACTiCLOUD https://www.researchgate.net/project/ACTiCLOUD) co-organized two scientific workshop and gave several talks and presented demos and posters in scientific conferences and workshops.
"The progress beyond the state of the art involves the definition of the ACTiCLOUD architecture, the development and optimization of the ACTiCLOUD components, their integration, and their evaluation.
The main results after the completion of the project, are as follows:
- the integration of the embraced scale-up hardware platforms (Numascale and KMAX) with cloud technologies.
- a rack-scale hypervisor that provides all the necessary mechanisms for virtualizing, managing and monitoring compute, network, and storage resources across the rack, as well as the mechanisms to reconfigure resources on demand.
- system libraries for running resource demanding applications such as Big Data, Analytics and HPC applications.
- compilation, scheduling, and garbage collection optimizations within managed runtime systems, putting significant effort on bringing up a state-of-the-art research VM (MaxineVM) and transitioning it to ACTiCLOUD’s Hyperscale JVM.
- a distributed cloud resource manager (ACTiManager) consisting of functional components, placement algorithms, and models for workload classification, interference detection and mitigation, and performance prediction.
- the ""cloudified"" enhancement of MonetDB to exploit the ACTiCLOUD architecture by being more resource aware, reacting to changing workloads quickly, and provisioning resources with elasticity.
- the ""cloudified"" enhancement of Neo4j to support online graph analytics alongside online transaction processing, on systems that provide an abundance of RAM and cores that typically constitute cloud envirnoments.
Some core impacts thanks to the success of ACTiCLOUD are as follows:
Strengthening the competitiveness and growth of cloud service providers: ACTiCLOUD’s achieves significant improvement in resource efficiency which may translate to a significant reduction of TCO. With a total worldwide spending on cloud IT infrastructure (server, storage, networking) reaching several billion euros, ACTiCLOUD’s impact on cost savings can lead to a significant paradigm shift in cloud architectures.
Lowering the costs of cloud services for end-users: While cloud offerings already provide competitive pricing to their customers compared to hosting in-house facilities, further cost reductions can be delivered by utilizing the cost benefits of ACTiCLOUD-enabled systems. We anticipate that based on the resource efficiency achieved by ACTiCLOUD, cloud service providers can proceed with charge reductions in the order of 20-30% for modestly equipped offerings, and to more than 10x for highly equipped offerings that are currently charged at extremely high monthly rates. This will further increase the uptake of cloud technology to more end-users both with modest and high resource demands.
Facilitating the “cloudification” of critical business applications: Banking, telecommunications e-commerce and numerous other domains rely on large, in-memory databases that pose non-conventional resource demands to cloud infrastructures and are currently either operating outside the cloud or are overpriced. ACTiCLOUD had a special target for resource demanding applications built on top of database frameworks in order to facilitate their smooth integration in cloud environments with reasonable pricing."
The main results after the completion of the project, are as follows:
- the integration of the embraced scale-up hardware platforms (Numascale and KMAX) with cloud technologies.
- a rack-scale hypervisor that provides all the necessary mechanisms for virtualizing, managing and monitoring compute, network, and storage resources across the rack, as well as the mechanisms to reconfigure resources on demand.
- system libraries for running resource demanding applications such as Big Data, Analytics and HPC applications.
- compilation, scheduling, and garbage collection optimizations within managed runtime systems, putting significant effort on bringing up a state-of-the-art research VM (MaxineVM) and transitioning it to ACTiCLOUD’s Hyperscale JVM.
- a distributed cloud resource manager (ACTiManager) consisting of functional components, placement algorithms, and models for workload classification, interference detection and mitigation, and performance prediction.
- the ""cloudified"" enhancement of MonetDB to exploit the ACTiCLOUD architecture by being more resource aware, reacting to changing workloads quickly, and provisioning resources with elasticity.
- the ""cloudified"" enhancement of Neo4j to support online graph analytics alongside online transaction processing, on systems that provide an abundance of RAM and cores that typically constitute cloud envirnoments.
Some core impacts thanks to the success of ACTiCLOUD are as follows:
Strengthening the competitiveness and growth of cloud service providers: ACTiCLOUD’s achieves significant improvement in resource efficiency which may translate to a significant reduction of TCO. With a total worldwide spending on cloud IT infrastructure (server, storage, networking) reaching several billion euros, ACTiCLOUD’s impact on cost savings can lead to a significant paradigm shift in cloud architectures.
Lowering the costs of cloud services for end-users: While cloud offerings already provide competitive pricing to their customers compared to hosting in-house facilities, further cost reductions can be delivered by utilizing the cost benefits of ACTiCLOUD-enabled systems. We anticipate that based on the resource efficiency achieved by ACTiCLOUD, cloud service providers can proceed with charge reductions in the order of 20-30% for modestly equipped offerings, and to more than 10x for highly equipped offerings that are currently charged at extremely high monthly rates. This will further increase the uptake of cloud technology to more end-users both with modest and high resource demands.
Facilitating the “cloudification” of critical business applications: Banking, telecommunications e-commerce and numerous other domains rely on large, in-memory databases that pose non-conventional resource demands to cloud infrastructures and are currently either operating outside the cloud or are overpriced. ACTiCLOUD had a special target for resource demanding applications built on top of database frameworks in order to facilitate their smooth integration in cloud environments with reasonable pricing."