Periodic Reporting for period 2 - SODALITE (SOftware Defined AppLication Infrastructures managemenT and Engineering)
Periodo di rendicontazione: 2020-02-01 al 2022-01-31
SODALITE is solving the deployment problem over the existing infrastructure, with particular focus on application management, performance.
SODALITE provides the tools to declare the application and infrastructure requirements through a simpler descriptive language within an IDE. The describing of the applications and infrastructure is supported through an intelligent semantic suggestion system and extensively checked for errors, with an intelligent, pattern-based system.
This enables to deploy applications faster and in a more informed way. The application is optimised even before deployment. During application execution, an innovative application predictive refactoring system, coupled with monitoring, improves the application execution under certain conditions. The goal is to provide support to the development, deployment, operation and execution of heterogeneous applications while accounting for diverse circumstances on heterogeneous, software-defined, high-performance, cloud infrastructures. A particular focus is on performance, quality, manageability, and reliability of the applications and infrastructure.
SODALITE will produce several high-level tangible results:
A pattern-based abstraction library that includes application, infrastructure, and performance abstractions;
A design and programming model for both full-stack applications and infrastructures based on the abstraction library;
A deployment framework that enables the static optimization of abstracted applications onto specific infrastructure;
Automated static and run-time optimization and management of applications.
The exploitable planned results produced will be:
Smart IDE - Application Deployment Modelling (and IaC Modelling)
PESTO - a combination of Deployment, Orchestration and Provisioning integrated toolset.
FindIaCBug - Verification and Bug Prediction in the IaC code
RIDER - Monitoring and Reconfiguration of the running application, to improve its performance.
POET - Performance optimization of the applications on different levels.
SODALITE provides the tools to declare the application and infrastructure requirements through a simpler descriptive language within an IDE. The describing of the applications and infrastructure is supported through an intelligent semantic suggestion system and extensively checked for errors, with an intelligent, pattern-based system.
This enables to deploy applications faster and in a more informed way. The application is optimised even before deployment. During application execution, an innovative application predictive refactoring system, coupled with monitoring, improves the application execution under certain conditions. The goal is to provide support to the development, deployment, operation and execution of heterogeneous applications while accounting for diverse circumstances on heterogeneous, software-defined, high-performance, cloud infrastructures. A particular focus is on performance, quality, manageability, and reliability of the applications and infrastructure.
SODALITE will produce several high-level tangible results:
A pattern-based abstraction library that includes application, infrastructure, and performance abstractions;
A design and programming model for both full-stack applications and infrastructures based on the abstraction library;
A deployment framework that enables the static optimization of abstracted applications onto specific infrastructure;
Automated static and run-time optimization and management of applications.
The exploitable planned results produced will be:
Smart IDE - Application Deployment Modelling (and IaC Modelling)
PESTO - a combination of Deployment, Orchestration and Provisioning integrated toolset.
FindIaCBug - Verification and Bug Prediction in the IaC code
RIDER - Monitoring and Reconfiguration of the running application, to improve its performance.
POET - Performance optimization of the applications on different levels.
In the first half of the project, the work within the Consortium was mainly focused on ramping up the facilities to successfully implement the project (first 6 months), followed by the research and implementation of the project’s goals. Technical work was mostly around the requirements, understanding of the use-cases and the initial designs of the overall architecture and the architecture of specific components. From the technical aspect we performed two iterations on the project architecture and the component developments. We built on the requirements from the initial phase and produced the first demonstrator, which has been presented at M12 (please see the project’s YouTube channel: https://www.youtube.com/channel/UCrArVp55GaJs78jFt1lUfFg). The use-cases were partially deployed using SODALITE technologies and the main components of the overall system were partially implemented. During this time, a lot of work was put into the use of the semantic technologies, targeting suggestions and improvement of the workflow for the developers.
In the time from M12 to M18 we revised the requirements, which were improved to take into account new discoveries. The architecture was updated and the work on the components continued, to produce the first integrated demonstrator, which can be deployed using SODALITE’s own blueprint, which can be downloaded from the project’s GitHub (https://GitHub.com/SODALITE-EU/).
The second half of the project period (M19 to M36) was mostly focused on the development of advanced features for all components, on the consolidation of the software integration and quality check processes and on making available the testbeds that support our experiments. We paid special attention to incorporating a state of the art Identity and Access Management (IAM) framework and a system to store secret keys, in order to increase the potential utilisability of the SODALITE framework in real contexts. We developed the case studies planned for the project and used them to identify and correct the limitations of the framework. The case studies, in particular, have been an inspiring source of ideas and have allowed us to introduce new useful features, from the Cloud to HPC data management approach, to the Operation Governance View made available through the IDE, to the integration with Kubernetes elements to control edge resources. In addition to the case studies, we defined controlled experiments to support the evaluation of the framework by external users. We established contacts with other initiatives to evaluate the possibility to establish synergies and cooperative sub-projects.
In the time from M12 to M18 we revised the requirements, which were improved to take into account new discoveries. The architecture was updated and the work on the components continued, to produce the first integrated demonstrator, which can be deployed using SODALITE’s own blueprint, which can be downloaded from the project’s GitHub (https://GitHub.com/SODALITE-EU/).
The second half of the project period (M19 to M36) was mostly focused on the development of advanced features for all components, on the consolidation of the software integration and quality check processes and on making available the testbeds that support our experiments. We paid special attention to incorporating a state of the art Identity and Access Management (IAM) framework and a system to store secret keys, in order to increase the potential utilisability of the SODALITE framework in real contexts. We developed the case studies planned for the project and used them to identify and correct the limitations of the framework. The case studies, in particular, have been an inspiring source of ideas and have allowed us to introduce new useful features, from the Cloud to HPC data management approach, to the Operation Governance View made available through the IDE, to the integration with Kubernetes elements to control edge resources. In addition to the case studies, we defined controlled experiments to support the evaluation of the framework by external users. We established contacts with other initiatives to evaluate the possibility to establish synergies and cooperative sub-projects.
Approaches supporting resource provisioning and deployment of applications do exist. They include IaC languages such as Ansible, Chef, Puppet for deployment and configuration and Terraform and TOSCA (a standardization effort that has raised the interest of multiple organisations) for provisioning of resources. However, when exploiting such approaches, a number of challenges must be faced:
- Defining a proper deployment for a complex application is not an easy task.
- Even when an expert is able to master the IaC approach which is available, this expert will still need to have at his/her disposal the specification of the resources to be used for deployment.
- Every new type of resource, even a different traditional cloud IaaS, offers different APIs and different access control mechanisms. As a result, its usage requires the development of new resource-specific code.
SODALITE supports a broad set of application types and resource types. Most importantly, it guides the users in defining new resource specifications and, for those infrastructures that allow for it, is able to automatically discover resources and generate the corresponding descriptions. Moreover, it offers the guided creation of defect- and error-free IaC artefacts and the optimization of application components for specific hardware resources, as well as runtime support for monitoring and reconfiguration.
More in detail, the SODALITE framework features a unique blend of tools that offer the following features:
- Smart creation of deployment models.
- Support design-time optimization of applications, especially in the case they adopt HPC resources.
- Support resource experts in modelling their resources and in automating the process of discovering new resources and deriving suitable models for them.
- Support the identification of bugs and smells in deployment models.
- Offer light-weight execution environments
- Incorporate monitoring configuration within deployment models
- Support runtime optimization of applications
- Secure operation of the stack
- Enable data transfer between heterogeneous data sources and platforms.
- Defining a proper deployment for a complex application is not an easy task.
- Even when an expert is able to master the IaC approach which is available, this expert will still need to have at his/her disposal the specification of the resources to be used for deployment.
- Every new type of resource, even a different traditional cloud IaaS, offers different APIs and different access control mechanisms. As a result, its usage requires the development of new resource-specific code.
SODALITE supports a broad set of application types and resource types. Most importantly, it guides the users in defining new resource specifications and, for those infrastructures that allow for it, is able to automatically discover resources and generate the corresponding descriptions. Moreover, it offers the guided creation of defect- and error-free IaC artefacts and the optimization of application components for specific hardware resources, as well as runtime support for monitoring and reconfiguration.
More in detail, the SODALITE framework features a unique blend of tools that offer the following features:
- Smart creation of deployment models.
- Support design-time optimization of applications, especially in the case they adopt HPC resources.
- Support resource experts in modelling their resources and in automating the process of discovering new resources and deriving suitable models for them.
- Support the identification of bugs and smells in deployment models.
- Offer light-weight execution environments
- Incorporate monitoring configuration within deployment models
- Support runtime optimization of applications
- Secure operation of the stack
- Enable data transfer between heterogeneous data sources and platforms.