Periodic Reporting for period 2 - PrEstoCloud (PrEstoCloud - Proactive Cloud Resources Management at the Edge for Efficient Real-Time Big Data Processing)
Okres sprawozdawczy: 2018-07-01 do 2019-12-31
Recent trends in cloud computing go towards the development of new paradigms, e.g. heterogeneous, federated, distributed multi-clouds and clouds which interact with local and edge resources. The new paradigms alleviate the tight interactions between the computing and networking infrastructures, with the purpose of addressing the ever-increasing dynamic processing and storage requirements of data-intensive applications. The dynamicity of emerging data-intensive applications requires optimized use of cloud and edge resources with respect to cost, flexibility and scalability.
The PrEstoCloud project made substantial research contributions in the cloud computing and real-time data intensive applications domains, by providing a dynamic, distributed, self-adaptive and proactively configurable architecture for deploying highly dynamic, data-intensive applications. In particular, PrEstoCloud combines real-time Big Data, mobile processing and cloud computing research in an innovative way that facilitates flexible deployment of data-intensive applications and extension of the fog computing paradigm to the extreme edge of the network. The PrEstoCloud solution is driven by the micro services paradigm and has been structured across five different conceptual layers: i) Meta-management; ii) Control; iii) Cloud infrastructure; iv) Cloud/Edge communication and v) Devices, layers. The solution supports the flexible deployment of data-intensive applications on multi-cloud, fog and edge resources. The validation of PrEstoCloud in three pilots from the logistics, mobile journalism and security surveillance application domains proved that it can improve the average response time, is domain agnostic and can make optimized use of available resources. Thus, it demonstrated added-value for attracting early adopters, initializing the exploitation process by showing concrete examples and benefits.
The PrEstoCloud project made substantial research contributions in the cloud computing and real-time data intensive applications domains, by providing a dynamic, distributed, self-adaptive and proactively configurable architecture for deploying highly dynamic, data-intensive applications. In particular, PrEstoCloud combines real-time Big Data, mobile processing and cloud computing research in an innovative way that facilitates flexible deployment of data-intensive applications and extension of the fog computing paradigm to the extreme edge of the network. The PrEstoCloud solution is driven by the micro services paradigm and has been structured across five different conceptual layers: i) Meta-management; ii) Control; iii) Cloud infrastructure; iv) Cloud/Edge communication and v) Devices, layers. The solution supports the flexible deployment of data-intensive applications on multi-cloud, fog and edge resources. The validation of PrEstoCloud in three pilots from the logistics, mobile journalism and security surveillance application domains proved that it can improve the average response time, is domain agnostic and can make optimized use of available resources. Thus, it demonstrated added-value for attracting early adopters, initializing the exploitation process by showing concrete examples and benefits.
Following the regular project processes and guidelines set up a requirement and State of the Art analysis was performed. A conceptual architecture was developed and concepts for each of the PrestoCloud components based on identified requirements were developed. The infrastructure has been set up, first prototypical implementations of components has been developed, deployed and integrated to a first demonstrator for the surveillance use. The platform architecture and components are shown in the figure. Beside the technical actions also the communication, dissemination and a first iteration of exploitation concept has been finished and communication successfully performed. Prestocloud and its progress is presented in all relevant social media, generates awareness on fares and publications out of PrestoCloud has been successfully accepted by scientific journals and conferences.
In second and final project phase all components has been completely developed and integrated to a generic platform demonstrator. All components have been tested, integrated and validated as well as the complete platform has been assessed and successfully checked against the requirements based on the three different use cases with their different requirements.
All 3 pilots has been implemented, using different combinations of PrEstoClouds platform components. So as a main result the project offers three demonstrators using PrEstoCloud platform for three different domains – Logistic, Surveillance and Media, showing the generic approach of architecture, platform and components
The results, especially the successful findings of the pilot validation and assessment has been reported in three different deliverables showing the benefits, improvements and possible cost reductions reached by using PrEstoCloud integrated platform. For exploitation, not only the architecture and platform, but also new offers of different PrEstoCloud bundled components has been identified and defined which will be offered by involved partners.
The dissemination and exploitation activities has been intensified in second phase. This is reflected in the large number of accepted papers and presentations not only within Europe but worldwide (e.g. Japan) as well as successful participation at worlds largest industry fair, where PrEstoCloud logistics use case was presented on stage of a project partner.
PrEstoCloud has been accepted and added to the Cyberwatching Technologie Radar https://www.cyberwatching.eu/technology-radar and in January 2020 was invited to join the marketplace https://cyberwatching.eu/market-products-list , which will happen as soon as possible
In second and final project phase all components has been completely developed and integrated to a generic platform demonstrator. All components have been tested, integrated and validated as well as the complete platform has been assessed and successfully checked against the requirements based on the three different use cases with their different requirements.
All 3 pilots has been implemented, using different combinations of PrEstoClouds platform components. So as a main result the project offers three demonstrators using PrEstoCloud platform for three different domains – Logistic, Surveillance and Media, showing the generic approach of architecture, platform and components
The results, especially the successful findings of the pilot validation and assessment has been reported in three different deliverables showing the benefits, improvements and possible cost reductions reached by using PrEstoCloud integrated platform. For exploitation, not only the architecture and platform, but also new offers of different PrEstoCloud bundled components has been identified and defined which will be offered by involved partners.
The dissemination and exploitation activities has been intensified in second phase. This is reflected in the large number of accepted papers and presentations not only within Europe but worldwide (e.g. Japan) as well as successful participation at worlds largest industry fair, where PrEstoCloud logistics use case was presented on stage of a project partner.
PrEstoCloud has been accepted and added to the Cyberwatching Technologie Radar https://www.cyberwatching.eu/technology-radar and in January 2020 was invited to join the marketplace https://cyberwatching.eu/market-products-list , which will happen as soon as possible
We have a generic PrestoCloud platform supporting at least the 3 main use case scenarios and ready to run by one of the Project partners or been sold Big Data Solution and Service providers and/or cloud service and infrastructure providers. Specifically, the project has gone beyond the state of the art regarding the main objectives:
- Dynamic monitoring in real-time Big Data processing architectures – a detection of anomalous situations on the fly (not predefined anomalies): combined predictive capabilities with the ability to recommend cloud resource adaptations, a new semantic-based model for distributed real-time processing architectures, monitoring of the real-time processing architecture, a support of the meta-modelling of the adaptation process and we enable real-time changes of the processing pipeline.
- Situation-aware and context-driven adaptation recommender systems – we combine predictive capabilities with the ability to recommend cloud resource adaptations by developing a big data situation meta model that can model situations relevant to cloud and edge resources topology, status and generic capabilities and propose algorithms for devising proactive adaptation actions.
- Real-time mobile stream processing – we use conventional big data analytics and integrate promising concepts of edge computing to investigate the concept of bandwidth and medium utilization and decide on edge, cloud or hybrid computing, to investigate on acceleration concepts for local analysis and global analytics control local analytics and business rules.
- Pro-active cloud computing - autonomous cloud management platform („proactive cloud automation“) that: provides a workflow catalogue system for provisioning and deployment workflows, is using a scalable scheduler and has the ability to connect to a variety of cloud providers.
- Network virtualization – we go for a deployment of a network overlay seamlessly interconnecting virtual networks on multiple sites, with reinforced security at the network level, formulated firewalls, access control, and monitoring data to provide meaningful input to the resource manager.
The project improves the utilization in terms of cost / time efficiency of paid cloud resources as well as the resource consumption of own cloud and premises. By improving flexibility in resource usage through an on demand resources management it also have impact on direct cost of operation, reducing costs for paid services/traffic, processing volume. The resource exploitation at extreme edge of network also is improved. Regarding services it improves infrastructure to enable new/better services as well as improvements regarding speed / volume. Also deployment of new services on the cloud, service performance, runtime and maintenance as well as the improvement of service monitoring can be taken into measurable account.
- Dynamic monitoring in real-time Big Data processing architectures – a detection of anomalous situations on the fly (not predefined anomalies): combined predictive capabilities with the ability to recommend cloud resource adaptations, a new semantic-based model for distributed real-time processing architectures, monitoring of the real-time processing architecture, a support of the meta-modelling of the adaptation process and we enable real-time changes of the processing pipeline.
- Situation-aware and context-driven adaptation recommender systems – we combine predictive capabilities with the ability to recommend cloud resource adaptations by developing a big data situation meta model that can model situations relevant to cloud and edge resources topology, status and generic capabilities and propose algorithms for devising proactive adaptation actions.
- Real-time mobile stream processing – we use conventional big data analytics and integrate promising concepts of edge computing to investigate the concept of bandwidth and medium utilization and decide on edge, cloud or hybrid computing, to investigate on acceleration concepts for local analysis and global analytics control local analytics and business rules.
- Pro-active cloud computing - autonomous cloud management platform („proactive cloud automation“) that: provides a workflow catalogue system for provisioning and deployment workflows, is using a scalable scheduler and has the ability to connect to a variety of cloud providers.
- Network virtualization – we go for a deployment of a network overlay seamlessly interconnecting virtual networks on multiple sites, with reinforced security at the network level, formulated firewalls, access control, and monitoring data to provide meaningful input to the resource manager.
The project improves the utilization in terms of cost / time efficiency of paid cloud resources as well as the resource consumption of own cloud and premises. By improving flexibility in resource usage through an on demand resources management it also have impact on direct cost of operation, reducing costs for paid services/traffic, processing volume. The resource exploitation at extreme edge of network also is improved. Regarding services it improves infrastructure to enable new/better services as well as improvements regarding speed / volume. Also deployment of new services on the cloud, service performance, runtime and maintenance as well as the improvement of service monitoring can be taken into measurable account.