Periodic Reporting for period 2 - MELODIC (Multi-cloud Execution-ware for Large-scale Optimized Data-Intensive Computing)
Okres sprawozdawczy: 2018-06-01 do 2020-01-31
Melodic will overcome the risk of Cloud vendor lock-in by realising the potential of cross-Cloud computing for big data and data-intensive applications. This is done by transparently taking advantage of distinct characteristics of available private and public Clouds, dynamic optimisation of resource utilisation, considering data locality, while conforming to the user’s privacy needs, service requirements, and other constraints.
These benefits are possible owing to the integrating open source platforms available from two major European Cloud projects and extending the result of the integration with the Hadoop and Spark big data frameworks:
• the PaaSage platform used for intelligent and autonomic cross-Cloud deployment and Melodic adds data aware modelling and extended deployment configuration reasoning;
• the Cloudiator platform is extended with the support for Hadoop and Spark in cross-Cloud application deployment and management.
Melodic will integrate with the existing open source development teams for these two platforms, and the contributions will be released back to the used platforms as open source.
Additionally, Melodic will ensure unified data security and cross-Cloud privacy by the security concepts taken from European project PaaSword. The Melodic platform is tested in several demanding real-world applications: scalable Customer Relationship Management, real-time optimised traffic routing, and fast and scalable processing of genomic data.
The integrated Melodic platform will be maintained and exploited by a professional software house and it represents a unique opportunity for European SMEs to save cost and improve the efficiency of their big-data applications while taking advantage of the Cloud computing business model.
These benefits are possible owing to the integrating open source platforms available from two major European Cloud projects and extending the result of the integration with the Hadoop and Spark big data frameworks:
• the PaaSage platform used for intelligent and autonomic cross-Cloud deployment and Melodic adds data aware modelling and extended deployment configuration reasoning;
• the Cloudiator platform is extended with the support for Hadoop and Spark in cross-Cloud application deployment and management.
Melodic will integrate with the existing open source development teams for these two platforms, and the contributions will be released back to the used platforms as open source.
Additionally, Melodic will ensure unified data security and cross-Cloud privacy by the security concepts taken from European project PaaSword. The Melodic platform is tested in several demanding real-world applications: scalable Customer Relationship Management, real-time optimised traffic routing, and fast and scalable processing of genomic data.
The integrated Melodic platform will be maintained and exploited by a professional software house and it represents a unique opportunity for European SMEs to save cost and improve the efficiency of their big-data applications while taking advantage of the Cloud computing business model.
There have been three main highlights during the first 18 months: two releases 1.0 and 1.5 and the evaluation of them by all the use cases. The main outcomes of this period are listed below:
• Implementation of the new integration layer with clearly separated Control Flow and Monitoring flow, using the MuleSoft Enterprise Service Bus (ESB), an enterprise-grade solution, orchestrated by Business Process Models (BPM) process using the Camunda BPM.
• Implementation of the new monitoring system using the ESPER Complex Event Processing engine.
• Evaluation and changes to the underpinning frameworks: some PaaSage and Cloudiator, the execution ware, where some components have been rewritten from scratch, while others have been significantly improved. Features from the next version of Cloudiator have been partially introduced.
• Creation and introduction of a new Utility Generator component separating the solvers’ objective functions from the utility value and thus allowing the generalised application utility to be represented algorithmically in the future.
• Introduction of industry-standard software development process focusing on software quality assurance, including the selection and the introduction of a complete Continuous Integration and delivery platform.
• All Melodic platform components are now separate containers using Docker technology exploiting Docker Swarm to manage the deployment of the Melodic platform itself.
Both releases of Melodic (1.0 and 1.5) have been tested by using established quality process; over 70 test cases have been executed and reported, with extensive usage of the automated test cases. All use cases, selected for the project, have been initially deployed using Melodic platform. Initial optimisation of the deployment has been successfully executed, showing that the complete autonomic computing loop consisting of deployment, metrics collection, and reconfiguration works properly. It is a proof that the Melodic approach is fully implemented and provides significant benefits for the use case applications. The results also indicate that Melodic is currently able to deploy and adapt any application with dedicated support for data intensive applications designed and under way to be implemented in the second main release along with the security mechanisms.
• Implementation of the new integration layer with clearly separated Control Flow and Monitoring flow, using the MuleSoft Enterprise Service Bus (ESB), an enterprise-grade solution, orchestrated by Business Process Models (BPM) process using the Camunda BPM.
• Implementation of the new monitoring system using the ESPER Complex Event Processing engine.
• Evaluation and changes to the underpinning frameworks: some PaaSage and Cloudiator, the execution ware, where some components have been rewritten from scratch, while others have been significantly improved. Features from the next version of Cloudiator have been partially introduced.
• Creation and introduction of a new Utility Generator component separating the solvers’ objective functions from the utility value and thus allowing the generalised application utility to be represented algorithmically in the future.
• Introduction of industry-standard software development process focusing on software quality assurance, including the selection and the introduction of a complete Continuous Integration and delivery platform.
• All Melodic platform components are now separate containers using Docker technology exploiting Docker Swarm to manage the deployment of the Melodic platform itself.
Both releases of Melodic (1.0 and 1.5) have been tested by using established quality process; over 70 test cases have been executed and reported, with extensive usage of the automated test cases. All use cases, selected for the project, have been initially deployed using Melodic platform. Initial optimisation of the deployment has been successfully executed, showing that the complete autonomic computing loop consisting of deployment, metrics collection, and reconfiguration works properly. It is a proof that the Melodic approach is fully implemented and provides significant benefits for the use case applications. The results also indicate that Melodic is currently able to deploy and adapt any application with dedicated support for data intensive applications designed and under way to be implemented in the second main release along with the security mechanisms.
The working releases are encouraging the commercial partners trust and desire of exploiting the Melodic platform both as a support of their own business applications and as a provider of a consultancy and guidance to other users of the platform. All the consortium members, who participate in developing the platform, have expressed commitments with ensuring its availability. The platform is particularly useful for SMEs, which may not have sufficient internal competence and resources to master complex cross-Cloud deployments of their applications. A good user community is essential to sustain the Melodic platform after the end of the project and the committed open source approach instil trust in future users that the Melodic platform will remain maintained and available for many the years to come.