Multi-cloud Execution-ware for Large-scale Optimized Data-Intensive Computing

Project Information

MELODIC

Grant agreement ID: 731664

DOI

10.3030/731664

Project closed

EC signature date 12 October 2016

Start date 1 December 2016

End date 31 January 2020

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 4 890 223,75

EU contribution

€ 4 890 223,75

Coordinated by

UNIVERSITETET I OSLO
Norway

Periodic Reporting for period 2 - MELODIC (Multi-cloud Execution-ware for Large-scale Optimized Data-Intensive Computing)

Reporting period: 2018-06-01 to 2020-01-31

Summary of the context and overall objectives of the project

Melodic will overcome the risk of Cloud vendor lock-in by realising the potential of cross-Cloud computing for big data and data-intensive applications. This is done by transparently taking advantage of distinct characteristics of available private and public Clouds, dynamic optimisation of resource utilisation, considering data locality, while conforming to the user’s privacy needs, service requirements, and other constraints.

These benefits are possible owing to the integrating open source platforms available from two major European Cloud projects and extending the result of the integration with the Hadoop and Spark big data frameworks:
• the PaaSage platform used for intelligent and autonomic cross-Cloud deployment and Melodic adds data aware modelling and extended deployment configuration reasoning;
• the Cloudiator platform is extended with the support for Hadoop and Spark in cross-Cloud application deployment and management.
Melodic will integrate with the existing open source development teams for these two platforms, and the contributions will be released back to the used platforms as open source.
Additionally, Melodic will ensure unified data security and cross-Cloud privacy by the security concepts taken from European project PaaSword. The Melodic platform is tested in several demanding real-world applications: scalable Customer Relationship Management, real-time optimised traffic routing, and fast and scalable processing of genomic data.

The integrated Melodic platform will be maintained and exploited by a professional software house and it represents a unique opportunity for European SMEs to save cost and improve the efficiency of their big-data applications while taking advantage of the Cloud computing business model.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

There have been three main highlights during the first 18 months: two releases 1.0 and 1.5 and the evaluation of them by all the use cases. The main outcomes of this period are listed below:

• Implementation of the new integration layer with clearly separated Control Flow and Monitoring flow, using the MuleSoft Enterprise Service Bus (ESB), an enterprise-grade solution, orchestrated by Business Process Models (BPM) process using the Camunda BPM. 
• Implementation of the new monitoring system using the ESPER Complex Event Processing engine. 
• Evaluation and changes to the underpinning frameworks: some PaaSage and Cloudiator, the execution ware, where some components have been rewritten from scratch, while others have been significantly improved. Features from the next version of Cloudiator have been partially introduced. 
• Creation and introduction of a new Utility Generator component separating the solvers’ objective functions from the utility value and thus allowing the generalised application utility to be represented algorithmically in the future. 
• Introduction of industry-standard software development process focusing on software quality assurance, including the selection and the introduction of a complete Continuous Integration and delivery platform. 
• All Melodic platform components are now separate containers using Docker technology exploiting Docker Swarm to manage the deployment of the Melodic platform itself.

Both releases of Melodic (1.0 and 1.5) have been tested by using established quality process; over 70 test cases have been executed and reported, with extensive usage of the automated test cases. All use cases, selected for the project, have been initially deployed using Melodic platform. Initial optimisation of the deployment has been successfully executed, showing that the complete autonomic computing loop consisting of deployment, metrics collection, and reconfiguration works properly. It is a proof that the Melodic approach is fully implemented and provides significant benefits for the use case applications. The results also indicate that Melodic is currently able to deploy and adapt any application with dedicated support for data intensive applications designed and under way to be implemented in the second main release along with the security mechanisms.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The working releases are encouraging the commercial partners trust and desire of exploiting the Melodic platform both as a support of their own business applications and as a provider of a consultancy and guidance to other users of the platform. All the consortium members, who participate in developing the platform, have expressed commitments with ensuring its availability. The platform is particularly useful for SMEs, which may not have sufficient internal competence and resources to master complex cross-Cloud deployments of their applications. A good user community is essential to sustain the Melodic platform after the end of the project and the committed open source approach instil trust in future users that the Melodic platform will remain maintained and available for many the years to come.

Project Logo