Periodic Reporting for period 2 - ELEGANT (Secure and Seamless Edge-to-Cloud Analytics)
Okres sprawozdawczy: 2022-07-01 do 2023-12-31
Imagine a Big Data application that processes a large amount of data arriving as streams from IoT devices. Such compute infrastructures are paving the way towards the future Sharing Economy Systems and they typically follow a producer/consumer model in which the analytics platform processes the IoT generated data. Since the processing capabilities of IoT devices are continually increasing, edge-processing has started being utilized as a means to increase both responsiveness and real-time characteristics of every deployed application, achieve energy efficient processing by reducing the computational cycles of the heavyweight cluster or cloud-deployed Big Data frameworks, and increase the security of the deployment in an end-to-end manner by enabling in-situ computations thus minimizing data exchanges through the network.
Although several industrial and academic initiatives are currently working towards unifying the software ecosystems of Big Data and IoT, a number of fundamental technical challenges still exist. These challenges mainly derive from the separate development routes that Big Data and IoT have taken during the last decade. In addition to the challenge of software diversification, current IoT and Big Data deployments are hardly interoperable due to:
1. Lack of dynamic semantic code adaptation
2. Inability of dynamic code orchestration
3. Weakened security features
ELEGANT has addressed these challenges proposing a new software unification paradigm for Big Data and IoT that enables their seamless interoperability through global orchestration of dynamic code motion, efficient usage of available resources, and adaptive security features.
ELEGANT’s vision is summarized by the following objectives:
➢ Unification of programming environments
➢ Dynamic Code Motion
➢ Intelligent resource selection and allocation
➢ Secure, Reliable, and Dependable code deployment
Although several industrial and academic initiatives are currently working towards unifying the software ecosystems of Big Data and IoT, a number of fundamental technical challenges still exist. These challenges mainly derive from the separate development routes that Big Data and IoT have taken during the last decade. In addition to the challenge of software diversification, current IoT and Big Data deployments are hardly interoperable due to:
1. Lack of dynamic semantic code adaptation
2. Inability of dynamic code orchestration
3. Weakened security features
ELEGANT has addressed these challenges proposing a new software unification paradigm for Big Data and IoT that enables their seamless interoperability through global orchestration of dynamic code motion, efficient usage of available resources, and adaptive security features.
ELEGANT’s vision is summarized by the following objectives:
➢ Unification of programming environments
➢ Dynamic Code Motion
➢ Intelligent resource selection and allocation
➢ Secure, Reliable, and Dependable code deployment
The ELEGANT project progressed towards the achievement of all its objectives. More specifically, the following components were developed as prototypes:
- the ELEGANT acceleration as a service,
- the ELEGANT Planner,
- the ELEGANT unified code validation and runtime verification techniques,
- the ELEGANT cybersecurity mechanisms (code and networking),
- the ELEGANT IDE for automatic code deployment and holistic management,
These components were then integrated within the NebulaStream platform, the functionality of which has been augmented to support the ELEGANT use cases.
As regards the acceleration as a service component, a web service was developed that interfaces with the modified TornadoVM and can transparently generate hardware accelerated code that can be seamlessly executed both on the edge and the cloud. A spin-off has already been created for the exploitation of the result.
The ELEGANT Planner, the component which can make intelligent, automated decisions both on how and where to map arbitrary data analytics tasks to the underlying heterogeneous infrastructure, was designed and developed in its prototype form and will be exploited scientifically, within research projects, and as part of PhDs.
The ELEGANT Code Verification Service was developed to support the transparent verification of use cases code segments and to enable the identification of potential security flaws due to the deployment of malicious or erroneous code, and will be further improved in terms of maturity and overall efficacy within research projects.
The ELEGANT cybersecurity mechanism for IoT networking was developed to meet the objectives identified by the Cybersecurity landscape study and use cases’ analysis performed and will be exploited as part of a cybersecurity service.
The ELEGANT IDE plugin and the User Interface were developed to assist developers in using ELEGANT operators as part of their code, and further exploitation is planned as part of the NebulaStream platform.
The above-mentioned components were developed as prototypes and will be integrated in NebulaStream, a novel, general purpose, end-to-end data management system for the IoT. Further exploitation is scheduled with meetings with industrial partners.
- the ELEGANT acceleration as a service,
- the ELEGANT Planner,
- the ELEGANT unified code validation and runtime verification techniques,
- the ELEGANT cybersecurity mechanisms (code and networking),
- the ELEGANT IDE for automatic code deployment and holistic management,
These components were then integrated within the NebulaStream platform, the functionality of which has been augmented to support the ELEGANT use cases.
As regards the acceleration as a service component, a web service was developed that interfaces with the modified TornadoVM and can transparently generate hardware accelerated code that can be seamlessly executed both on the edge and the cloud. A spin-off has already been created for the exploitation of the result.
The ELEGANT Planner, the component which can make intelligent, automated decisions both on how and where to map arbitrary data analytics tasks to the underlying heterogeneous infrastructure, was designed and developed in its prototype form and will be exploited scientifically, within research projects, and as part of PhDs.
The ELEGANT Code Verification Service was developed to support the transparent verification of use cases code segments and to enable the identification of potential security flaws due to the deployment of malicious or erroneous code, and will be further improved in terms of maturity and overall efficacy within research projects.
The ELEGANT cybersecurity mechanism for IoT networking was developed to meet the objectives identified by the Cybersecurity landscape study and use cases’ analysis performed and will be exploited as part of a cybersecurity service.
The ELEGANT IDE plugin and the User Interface were developed to assist developers in using ELEGANT operators as part of their code, and further exploitation is planned as part of the NebulaStream platform.
The above-mentioned components were developed as prototypes and will be integrated in NebulaStream, a novel, general purpose, end-to-end data management system for the IoT. Further exploitation is scheduled with meetings with industrial partners.
On the front of Embedded JVMs and Heterogeneous Java Execution, ELEGANT advances heterogeneous execution of Java by integrating JVMs targeting embedded devices and heterogeneous hardware into the NebulaStream runtime. These attributes is direct result of the integration of the TornadoVM framework with NebulaStream, an approach that will lead to significant performance paired with energy-efficiency.
To further improve cooperation and ensure proper code motion between all participating entities regardless of their position, ELEGANT develops the Intelligent Orchestrator, a dedicated module for (i) gathering profiling information for all devices, (ii) making scheduling decisions regarding job execution and (iii) instructing code motion through data operator submission to a dedicated thread running on ELEGANT Elastic Runtime.
On the front of IoT programming frameworks and management, ELEGANT is innovating upon the currently available tools and platforms by providing a unified programming framework between Big Data and IoT applications. Unlike any existing solution, the proposed features of ELEGANT will lead to Big Data/IoT deployments where: (i) orchestration and functional decisions can be made at real-time, (ii) extreme code reuse will become a norm, (iii) elasticity will be achieved during execution since data stream rate transmitted by IoT nodes will be intelligently controlled on the receiving end, and (iv) better scheduling will consequently induce better power/performance ratio.
To further improve cooperation and ensure proper code motion between all participating entities regardless of their position, ELEGANT develops the Intelligent Orchestrator, a dedicated module for (i) gathering profiling information for all devices, (ii) making scheduling decisions regarding job execution and (iii) instructing code motion through data operator submission to a dedicated thread running on ELEGANT Elastic Runtime.
On the front of IoT programming frameworks and management, ELEGANT is innovating upon the currently available tools and platforms by providing a unified programming framework between Big Data and IoT applications. Unlike any existing solution, the proposed features of ELEGANT will lead to Big Data/IoT deployments where: (i) orchestration and functional decisions can be made at real-time, (ii) extreme code reuse will become a norm, (iii) elasticity will be achieved during execution since data stream rate transmitted by IoT nodes will be intelligently controlled on the receiving end, and (iv) better scheduling will consequently induce better power/performance ratio.