Periodic Reporting for period 2 - PIACERE (Programming trustworthy Infrastructure As Code in a sEcuRE framework)
Período documentado: 2022-06-01 hasta 2023-11-30
The virtualization revolution that has taken place in the last years along with the advent of the cloud computing continuum (combination of cloud and edge), has allowed for an increase in the use of software to build, control and configure entire virtual data centers and the entire infrastructure layer. The use of such tools and APIs has only stressed the importance of software in the infrastructure arena.
Infrastructure-as-Code (IaC), enables the automation of several deployment, configuration and management tasks that otherwise would have to be performed manually by an operator. IaC has a lot of potential in a cloud computing context as it results in a significant saving of time when an application needs to be redeployed on a different set of resources or needs to be extended with new components, even possibly running on different cloud infrastructures.As such, IaC has represented a very important progress that has dramatically changed the work organization of many IT-intensive organizations (e.g. Netflix ).
Unfortunately, IaC still suffers from these five main issues: variety of competing tools, focus on a single step or type of infrastrcutural element, not adressed the Fog Comptuing or complex Cloud Continumm environments. security is only applied in the end of the cycle andnot from the beginning.
The main objective of the PIACERE project is thus to provide means (tools, methods and techniques) to enable most organizations to fully embrace the Infrastructure-as-Code approach, through the DevSecOps philosophy, by making the creation of such infrastructural code more accessible to designers, developers and operators (DevSecOps teams), increasing the quality, security, trustworthiness and evolvability of infrastructural code while ensuring its business continuity by providing self-healing mechanisms anticipation of failures and violations, allowing it to self-learn from the conditions that triggered such re-adaptations.
Infrastructure-as-Code (IaC), enables the automation of several deployment, configuration and management tasks that otherwise would have to be performed manually by an operator. IaC has a lot of potential in a cloud computing context as it results in a significant saving of time when an application needs to be redeployed on a different set of resources or needs to be extended with new components, even possibly running on different cloud infrastructures.As such, IaC has represented a very important progress that has dramatically changed the work organization of many IT-intensive organizations (e.g. Netflix ).
Unfortunately, IaC still suffers from these five main issues: variety of competing tools, focus on a single step or type of infrastrcutural element, not adressed the Fog Comptuing or complex Cloud Continumm environments. security is only applied in the end of the cycle andnot from the beginning.
The main objective of the PIACERE project is thus to provide means (tools, methods and techniques) to enable most organizations to fully embrace the Infrastructure-as-Code approach, through the DevSecOps philosophy, by making the creation of such infrastructural code more accessible to designers, developers and operators (DevSecOps teams), increasing the quality, security, trustworthiness and evolvability of infrastructural code while ensuring its business continuity by providing self-healing mechanisms anticipation of failures and violations, allowing it to self-learn from the conditions that triggered such re-adaptations.
Main achievements for this period can be summarized as follows:
• First version of the functional requirements of all Key Results , the overall architecture of the design time and runtime components of PIACERE solutions and the workflow and related sub-workflows described as sequence diagrams, that is, how all PIACERE components work together. These have been also translated into the PIACERE scenarios, describing the workflows from the user perspective.
• First working version of all individual components covering the whole lifecycle of the IaC, from design time to run-time. The DevSecOps modelling language has been designed and specified including the concepts to be supported at different levels of abstraction (. Two versions of DOML have been released with all the needed concepts for the PIACERE tools and the use cases to create their infrastructural descriptions of the applications. The PIACERE IDE (based on Eclipse Desktop) integrates all the design time tools and some of the runtime tools (i.e. Canary Sandbox Environments Provisioner) and serves as the main entry point for the PIACERE tools. The ICG generates Terraform and Ansible code from the DOML specification and the IOP supports the optimization of up to three objectives declared in the DOML specification. The DOML checker(previously Verification Tool) inspects semantics of the DOML and assess the correctness of the DOML instance against DOML v1 and DOML v2. The IaC scanner assess the security vulnerabilities of the Terraform and Ansible files. At run-time the Canary Sandbox environment allows testing of dynamic properties of the generated IaC creating opinionated local environments, such as OpenStack. The IEM executes the generated IaC scripts, and use them to provision, configure, and install the required infrastructure. Currently it supports the creation of OpenStack and Amazon based resources. The run time is handled by the PIACERE runtime tooling, specifically, the PIACERE Runtime Contoller (PRC), based on Camunda which orchestrates the rest of the runtime tooling to achieve coherency in the runtime. The continuous monitoring components (both performance and security) generate the needed Ansible playbooks to deploy agents integrated in the IaC examples provided by the ICG and feed the time series data bases. The information collected is the input for the self-learning component which analyses several metrics (i.e. cpu usage idle, then cpu usage system) and applies algorithmic analysis to predict failures so which are informed to the self-healing component. In the self-healing component four strategies have been defined to address the potential issues in the infrastructure: reboot, redeploy, vertical scalability and horizontal scalability. At M18 And we have focused on the realisation of the redeployment strategy integrated with the PRC.
• First integrated version of the PIACERE DevSecOps framework. This first version includes the IDE and the PRC as the two main components orchestrating all the PIACERE tools. It allows the creation of a PIACERE project in the IDE and is the trigger of the design time tools (DOML, DOML checker, IaC scanner, ICG) as well as the CSE and the catalogue. The PRC incorporates the calls to the CSE and the IEM in the current version. Also, the self-healing component interoperates with the PRC to trigger a new self-healing strategy.
• Definition and implementation of the use cases (v1 and v2) and the validation strategy, accompanied by a set of use cases requirements and KPIs that will serve to measure the benefits provided by the PIACERE KRs.
• Continuous market watch on competitors, value proposition of PAICERE KERs along with the HRB European Commission initiative. This is accompanied by an analysis of the potential exploitation strategy and the individual business models that the piacere partners envision for their offerings. Furthermore, an initial IPR and the accompanying licenses have been defined.
• First version of the functional requirements of all Key Results , the overall architecture of the design time and runtime components of PIACERE solutions and the workflow and related sub-workflows described as sequence diagrams, that is, how all PIACERE components work together. These have been also translated into the PIACERE scenarios, describing the workflows from the user perspective.
• First working version of all individual components covering the whole lifecycle of the IaC, from design time to run-time. The DevSecOps modelling language has been designed and specified including the concepts to be supported at different levels of abstraction (. Two versions of DOML have been released with all the needed concepts for the PIACERE tools and the use cases to create their infrastructural descriptions of the applications. The PIACERE IDE (based on Eclipse Desktop) integrates all the design time tools and some of the runtime tools (i.e. Canary Sandbox Environments Provisioner) and serves as the main entry point for the PIACERE tools. The ICG generates Terraform and Ansible code from the DOML specification and the IOP supports the optimization of up to three objectives declared in the DOML specification. The DOML checker(previously Verification Tool) inspects semantics of the DOML and assess the correctness of the DOML instance against DOML v1 and DOML v2. The IaC scanner assess the security vulnerabilities of the Terraform and Ansible files. At run-time the Canary Sandbox environment allows testing of dynamic properties of the generated IaC creating opinionated local environments, such as OpenStack. The IEM executes the generated IaC scripts, and use them to provision, configure, and install the required infrastructure. Currently it supports the creation of OpenStack and Amazon based resources. The run time is handled by the PIACERE runtime tooling, specifically, the PIACERE Runtime Contoller (PRC), based on Camunda which orchestrates the rest of the runtime tooling to achieve coherency in the runtime. The continuous monitoring components (both performance and security) generate the needed Ansible playbooks to deploy agents integrated in the IaC examples provided by the ICG and feed the time series data bases. The information collected is the input for the self-learning component which analyses several metrics (i.e. cpu usage idle, then cpu usage system) and applies algorithmic analysis to predict failures so which are informed to the self-healing component. In the self-healing component four strategies have been defined to address the potential issues in the infrastructure: reboot, redeploy, vertical scalability and horizontal scalability. At M18 And we have focused on the realisation of the redeployment strategy integrated with the PRC.
• First integrated version of the PIACERE DevSecOps framework. This first version includes the IDE and the PRC as the two main components orchestrating all the PIACERE tools. It allows the creation of a PIACERE project in the IDE and is the trigger of the design time tools (DOML, DOML checker, IaC scanner, ICG) as well as the CSE and the catalogue. The PRC incorporates the calls to the CSE and the IEM in the current version. Also, the self-healing component interoperates with the PRC to trigger a new self-healing strategy.
• Definition and implementation of the use cases (v1 and v2) and the validation strategy, accompanied by a set of use cases requirements and KPIs that will serve to measure the benefits provided by the PIACERE KRs.
• Continuous market watch on competitors, value proposition of PAICERE KERs along with the HRB European Commission initiative. This is accompanied by an analysis of the potential exploitation strategy and the individual business models that the piacere partners envision for their offerings. Furthermore, an initial IPR and the accompanying licenses have been defined.
The main innovations of PIACERE for this period can be summarized as follows:
• First version of the ‘DevSecOps’ concept, that includes the approach and the required tools to apply the DevOps philosophy to the development and operation of the IaC including security aspects in the whole process.
• DOML v1 language as the modeling language to define the complete infrastructural model of an application that allows to seamlessly generate the related IaC to provision the infrastructural elements and deploy the application components on them. DOML has been created in a modular way with several layers of abstractions that ease the understanding and utilization of the model and enables the extension mechanism to incorporate new infrastructural elements.
• An Integrated Development Environment, and the integrated design time tools for the generation of the IaC code that enables the automatic creation of the IaC (Terraform + Ansible) and provision the infrastructure in different environments (Canary or Production).
• SAST and DAST supporting tools to incorporate the security aspects in the IaC SDLC and SOLC.
• Automatic monitoring of the non-functional requirements (performance and security) established for the for the infrastructural elements, and predictive analysis of the non-compliance of the thresholds through ML techniques.
• Self-healing strategies to be applied to the IaC lifecycle.
• First version of the ‘DevSecOps’ concept, that includes the approach and the required tools to apply the DevOps philosophy to the development and operation of the IaC including security aspects in the whole process.
• DOML v1 language as the modeling language to define the complete infrastructural model of an application that allows to seamlessly generate the related IaC to provision the infrastructural elements and deploy the application components on them. DOML has been created in a modular way with several layers of abstractions that ease the understanding and utilization of the model and enables the extension mechanism to incorporate new infrastructural elements.
• An Integrated Development Environment, and the integrated design time tools for the generation of the IaC code that enables the automatic creation of the IaC (Terraform + Ansible) and provision the infrastructure in different environments (Canary or Production).
• SAST and DAST supporting tools to incorporate the security aspects in the IaC SDLC and SOLC.
• Automatic monitoring of the non-functional requirements (performance and security) established for the for the infrastructural elements, and predictive analysis of the non-compliance of the thresholds through ML techniques.
• Self-healing strategies to be applied to the IaC lifecycle.