Periodic Reporting for period 1 - ELEGANT (Secure and Seamless Edge-to-Cloud Analytics)
Periodo di rendicontazione: 2021-01-01 al 2022-06-30
Imagine a Big Data application that processes a large amount of data arriving as streams from IoT devices. Such compute infrastructures are paving the way towards the future Sharing Economy Systems and they typically follow a producer/consumer model in which the analytics platform processes the IoT generated data. Since the processing capabilities of IoT devices are continually increasing, edge-processing has started being utilized as a means to increase both responsiveness and real-time characteristics of every deployed application, achieve energy efficient processing by reducing the computational cycles of the heavyweight cluster or cloud-deployed Big Data frameworks, and increase the security of the deployment in an end-to-end manner by enabling in-situ computations thus minimizing data exchanges through the network.
Although several industrial and academic initiatives are currently working towards unifying the software ecosystems of Big Data and IoT, a number of fundamental technical challenges still exist. These challenges mainly derive from the separate development routes that Big Data and IoT have taken during the last decade. In addition to the challenge of software diversification, current IoT and Big Data deployments are hardly interoperable due to:
1. Lack of dynamic semantic code adaptation
2. Inability of dynamic code orchestration
3. Weakened security features
ELEGANT addresses these challenges proposing a new software unification paradigm for Big Data and IoT that enables their seamless interoperability through global orchestration of dynamic code motion, efficient usage of available resources, and adaptive security features.
ELEGANT’s vision is summarized by the following objectives:
➢ Unification of programming environments
➢ Dynamic Code Motion
➢ Intelligent resource selection and allocation
➢ Secure, Reliable, and Dependable code deployment
Although several industrial and academic initiatives are currently working towards unifying the software ecosystems of Big Data and IoT, a number of fundamental technical challenges still exist. These challenges mainly derive from the separate development routes that Big Data and IoT have taken during the last decade. In addition to the challenge of software diversification, current IoT and Big Data deployments are hardly interoperable due to:
1. Lack of dynamic semantic code adaptation
2. Inability of dynamic code orchestration
3. Weakened security features
ELEGANT addresses these challenges proposing a new software unification paradigm for Big Data and IoT that enables their seamless interoperability through global orchestration of dynamic code motion, efficient usage of available resources, and adaptive security features.
ELEGANT’s vision is summarized by the following objectives:
➢ Unification of programming environments
➢ Dynamic Code Motion
➢ Intelligent resource selection and allocation
➢ Secure, Reliable, and Dependable code deployment
During the 1st reporting period, the ELEGANT project progressed towards the achievement of all its objectives. More specifically,
- the user requirements were extracted early in the project, allowing
- the development and initial evaluation of the prototype versions of all ELEGANT platform components. This, in turn, enabled
- the definition of the four diverse use cases in such a way so as to be directly related to current operations and needs, while also looking at the future and leveraging the ELEGANT platform features.
With regard to the extraction of the user requirements, the use case partners were able to identify the ELEGANT features that are relevant to their operations and subsequently adapt their use cases so that they fully leverage them, and technical providers were able to assess the requirements that the ELEGANT platform shall meet in order to support these use cases. Four diverse use cases, in the areas of medical wearables, video analytics, smart riding, and smart water metering, were initially defined and then implemented by the use case partners.
The following components were initially designed and subsequently developed as prototypes:
- the ELEGANT acceleration as a service,
- the ELEGANT Planner,
- the ELEGANT unified code validation and runtime verification techniques,
- the ELEGANT cybersecurity mechanisms (code and networking),
- the ELEGANT IDE for automatic code deployment and holistic management,
These components will be integrated within the NebulaStream platform, the functionality of which has been augmented to support the ELEGANT use cases.
As regards the acceleration as a service component, a web service was developed that interfaces with the modified TornadoVM and can transparently generate hardware accelerated code that can be seamlessly executed both on the edge and the cloud.
The ELEGANT Planner, the component which can make intelligent, automated decisions both on how and where to map arbitrary data analytics tasks to the underlying heterogeneous infrastructure, was designed and developed in its prototype form during the reporting period under consideration.
The ELEGANT Code Verification Service was developed to support the transparent verification of use cases code segments and to enable the identification of potential security flaws due to the deployment of malicious or erroneous code.
The ELEGANT cybersecurity mechanism for IoT networking was developed to meet the objectives identified by the Cybersecurity landscape study and use cases’ analysis performed.
The ELEGANT IDE plugin was developed to assist developers in using ELEGANT operators as part of their code.
The NES UI was extended with the addition of a user management mechanism that improves the overall security and allows ELEGANT to support more complex multi-user scenarios.
The above-mentioned components were developed as prototypes and will be integrated in NebulaStream, a novel, general purpose, end-to-end data management system for the IoT. During the first reporting period, an API was developed to let users specify data processing pipelines on an IoT network. Furthermore, an elastic runtime was developed to seamlessly scale between IoT devices and high-performance servers.
As regards the integration work performed during the reporting period under consideration, it focused on two directions:
- Deployment of the use cases on top of the ELEGANT platform
- Integration of ELEGANT components with NebulaStream.
The integration plan was defined in such a way to allow these two processes to proceed independently. The preliminary integrated ELEGANT prototype was delivered, extending NebulaStream with additional components to implement ELEGANT functionality.
As regards horizontal activities related to impact creation, the consortium produced 12 presentations and 16 publications, including 7 in scientific journals. As regards communication activities, eight such blogposts were produced by consortium partners and can be found in the project website.
- the user requirements were extracted early in the project, allowing
- the development and initial evaluation of the prototype versions of all ELEGANT platform components. This, in turn, enabled
- the definition of the four diverse use cases in such a way so as to be directly related to current operations and needs, while also looking at the future and leveraging the ELEGANT platform features.
With regard to the extraction of the user requirements, the use case partners were able to identify the ELEGANT features that are relevant to their operations and subsequently adapt their use cases so that they fully leverage them, and technical providers were able to assess the requirements that the ELEGANT platform shall meet in order to support these use cases. Four diverse use cases, in the areas of medical wearables, video analytics, smart riding, and smart water metering, were initially defined and then implemented by the use case partners.
The following components were initially designed and subsequently developed as prototypes:
- the ELEGANT acceleration as a service,
- the ELEGANT Planner,
- the ELEGANT unified code validation and runtime verification techniques,
- the ELEGANT cybersecurity mechanisms (code and networking),
- the ELEGANT IDE for automatic code deployment and holistic management,
These components will be integrated within the NebulaStream platform, the functionality of which has been augmented to support the ELEGANT use cases.
As regards the acceleration as a service component, a web service was developed that interfaces with the modified TornadoVM and can transparently generate hardware accelerated code that can be seamlessly executed both on the edge and the cloud.
The ELEGANT Planner, the component which can make intelligent, automated decisions both on how and where to map arbitrary data analytics tasks to the underlying heterogeneous infrastructure, was designed and developed in its prototype form during the reporting period under consideration.
The ELEGANT Code Verification Service was developed to support the transparent verification of use cases code segments and to enable the identification of potential security flaws due to the deployment of malicious or erroneous code.
The ELEGANT cybersecurity mechanism for IoT networking was developed to meet the objectives identified by the Cybersecurity landscape study and use cases’ analysis performed.
The ELEGANT IDE plugin was developed to assist developers in using ELEGANT operators as part of their code.
The NES UI was extended with the addition of a user management mechanism that improves the overall security and allows ELEGANT to support more complex multi-user scenarios.
The above-mentioned components were developed as prototypes and will be integrated in NebulaStream, a novel, general purpose, end-to-end data management system for the IoT. During the first reporting period, an API was developed to let users specify data processing pipelines on an IoT network. Furthermore, an elastic runtime was developed to seamlessly scale between IoT devices and high-performance servers.
As regards the integration work performed during the reporting period under consideration, it focused on two directions:
- Deployment of the use cases on top of the ELEGANT platform
- Integration of ELEGANT components with NebulaStream.
The integration plan was defined in such a way to allow these two processes to proceed independently. The preliminary integrated ELEGANT prototype was delivered, extending NebulaStream with additional components to implement ELEGANT functionality.
As regards horizontal activities related to impact creation, the consortium produced 12 presentations and 16 publications, including 7 in scientific journals. As regards communication activities, eight such blogposts were produced by consortium partners and can be found in the project website.
On the front of Embedded JVMs and Heterogeneous Java Execution, ELEGANT advances heterogeneous execution of Java by integrating JVMs targeting embedded devices and heterogeneous hardware into the NebulaStream runtime. These attributes is direct result of the integration of the TornadoVM framework with NebulaStream, an approach that will lead to significant performance paired with energy-efficiency.
To further improve cooperation and ensure proper code motion between all participating entities regardless of their position, ELEGANT develops the Intelligent Orchestrator, a dedicated module for (i) gathering profiling information for all devices, (ii) making scheduling decisions regarding job execution and (iii) instructing code motion through data operator submission to a dedicated thread running on ELEGANT Elastic Runtime.
On the front of IoT programming frameworks and management, ELEGANT is innovating upon the currently available tools and platforms by providing a unified programming framework between Big Data and IoT applications. Unlike any existing solution, the proposed features of ELEGANT will lead to Big Data/IoT deployments where: (i) orchestration and functional decisions can be made at real-time, (ii) extreme code reuse will become a norm, (iii) elasticity will be achieved during execution since data stream rate transmitted by IoT nodes will be intelligently controlled on the receiving end, and (iv) better scheduling will consequently induce better power/performance ratio.
To further improve cooperation and ensure proper code motion between all participating entities regardless of their position, ELEGANT develops the Intelligent Orchestrator, a dedicated module for (i) gathering profiling information for all devices, (ii) making scheduling decisions regarding job execution and (iii) instructing code motion through data operator submission to a dedicated thread running on ELEGANT Elastic Runtime.
On the front of IoT programming frameworks and management, ELEGANT is innovating upon the currently available tools and platforms by providing a unified programming framework between Big Data and IoT applications. Unlike any existing solution, the proposed features of ELEGANT will lead to Big Data/IoT deployments where: (i) orchestration and functional decisions can be made at real-time, (ii) extreme code reuse will become a norm, (iii) elasticity will be achieved during execution since data stream rate transmitted by IoT nodes will be intelligently controlled on the receiving end, and (iv) better scheduling will consequently induce better power/performance ratio.