Periodic Reporting for period 1 - MARVEL (Multimodal Extreme Scale Data Analytics for Smart Cities Environments)
Período documentado: 2021-01-01 hasta 2022-06-30
MARVEL aspires the convergence of AI, analytics, multimodal perception, software engineering, and HPC in the context of an Edge-Fog-Cloud (E2F2C) Continuum. MARVEL goes beyond traditional Big Data conventional architectures, capitalising on distributed resources and heterogeneous data sources and implementing privacy preservation at all data modalities and all architecture levels. This will enable support of data-driven real-time application workflows and decision-making in modern cities, showcasing the potential to address societal challenges, from increasing public safety and security to analysing traffic flows and traffic behaviour in the cities of Trento and Malta, and will be accomplished through the following objectives:
• Leverage innovative technologies for data acquisition, management and distribution to develop a privacy-aware engineering solution for revealing valuable and hidden societal knowledge in a smart city environment.
• Deliver AI-based multimodal perception and intelligence for audio-visual scene recognition, event detection and situational awareness in a smart city environment.
• Break technological silos, converge very diverse and novel engineering paradigms and establish a distributed and secure E2F2C ubiquitous computing framework in the big data value chain.
• Realise societal opportunities in a smart city environment by validating tools and techniques in real-world settings.
• Foster the European Data Economy vision and create new scientific and business opportunities by offering the MARVEL Data Corpus as a free service and contributing to BDVA standards.
• Leverage innovative technologies for data acquisition, management and distribution to develop a privacy-aware engineering solution for revealing valuable and hidden societal knowledge in a smart city environment.
• Deliver AI-based multimodal perception and intelligence for audio-visual scene recognition, event detection and situational awareness in a smart city environment.
• Break technological silos, converge very diverse and novel engineering paradigms and establish a distributed and secure E2F2C ubiquitous computing framework in the big data value chain.
• Realise societal opportunities in a smart city environment by validating tools and techniques in real-world settings.
• Foster the European Data Economy vision and create new scientific and business opportunities by offering the MARVEL Data Corpus as a free service and contributing to BDVA standards.
Project highlights per WP include:
WP1 analysed the benefits of multimodal analysis in smart cities, analysed SOTA and MARVEL’s ambitions beyond SOTA, and defined the experimental protocol, the pilot use cases and the conceptual MARVEL architecture.
WP2 resulted in (i) definition of the use cases’ datasets, (ii) development, calibration and connection of the Data Management Platform toolkit, (iii) common flexible augmentation strategy, iv) development of the MARVEL Data Corpus and iv) ethics, privacy, and data protection compliance.
WP3 resulted in (i) development of face and voice-swapping techniques and anonymisation components at the edge, (ii) novel methodologies and components for AV ML/DL training and inference, personalised federated training, anonymisation, model compression and optimised model deployment, (iii) E2F2C Kubernetes-based continuum computing, and iv) the MARVEL AI model repository, a querable database of MARVEL ML/DL model.
WP4 resulted in (i) enhanced audio acquisition device by increasing the number of microphones and device capabilities, (ii) a new robust and light-weighted Voice Activity Detection model, (iii) deployment and evaluation of GPURegex component, (iv) encryption and protection of data using EdgeSec VPN and EdgeSec TEE, and (v) implementation of interactive visualisations and AV analytics tool.
WP5 resulted in (i) delivery of an infrastructure tailored for the application requirements, (ii) efficient resource management processes, (iii) release of the MARVEL Minimum Viable Product (M12) and the MARVEL 1st Integrated framework (M18) with 31 out of 35 components integrated in 5 use cases, (iv) initial benchmarking strategy and component-level benchmarking after the delivery of the MVP.
WP6 led to the realisation of 5 real-life use cases from 3 smart cities selected for the M18 prototype. Also, the experimental protocol was revisited.
WP7 performed initial market analysis and business modelling. The dissemination KPIs have been overachieved (participation in more than 60 events, publication of 30 scientific papers). Individual exploitation plans were collected, and an internal exploitation workshop was organised toward a joint exploitation plan based on the dynamic business model canvas. The current and emerging standards were reviewed and a two-way interaction between MARVEL and standardisation bodies has been stimulated.
WP8 led to the definition of all MARVEL general management procedures, including quality assurance and risk analysis, and the definition of the Data Management Plan and MARVEL Innovations. MARVEL Advisory Board was established including 4 external members; their first feedback was received.
WP9 addressed all ethical requirements; the respective risk mitigation and compliance activities of the consortium were reported. The Ethics Advisory Board has been established with 3 external members; their first feedback was received.
WP1 analysed the benefits of multimodal analysis in smart cities, analysed SOTA and MARVEL’s ambitions beyond SOTA, and defined the experimental protocol, the pilot use cases and the conceptual MARVEL architecture.
WP2 resulted in (i) definition of the use cases’ datasets, (ii) development, calibration and connection of the Data Management Platform toolkit, (iii) common flexible augmentation strategy, iv) development of the MARVEL Data Corpus and iv) ethics, privacy, and data protection compliance.
WP3 resulted in (i) development of face and voice-swapping techniques and anonymisation components at the edge, (ii) novel methodologies and components for AV ML/DL training and inference, personalised federated training, anonymisation, model compression and optimised model deployment, (iii) E2F2C Kubernetes-based continuum computing, and iv) the MARVEL AI model repository, a querable database of MARVEL ML/DL model.
WP4 resulted in (i) enhanced audio acquisition device by increasing the number of microphones and device capabilities, (ii) a new robust and light-weighted Voice Activity Detection model, (iii) deployment and evaluation of GPURegex component, (iv) encryption and protection of data using EdgeSec VPN and EdgeSec TEE, and (v) implementation of interactive visualisations and AV analytics tool.
WP5 resulted in (i) delivery of an infrastructure tailored for the application requirements, (ii) efficient resource management processes, (iii) release of the MARVEL Minimum Viable Product (M12) and the MARVEL 1st Integrated framework (M18) with 31 out of 35 components integrated in 5 use cases, (iv) initial benchmarking strategy and component-level benchmarking after the delivery of the MVP.
WP6 led to the realisation of 5 real-life use cases from 3 smart cities selected for the M18 prototype. Also, the experimental protocol was revisited.
WP7 performed initial market analysis and business modelling. The dissemination KPIs have been overachieved (participation in more than 60 events, publication of 30 scientific papers). Individual exploitation plans were collected, and an internal exploitation workshop was organised toward a joint exploitation plan based on the dynamic business model canvas. The current and emerging standards were reviewed and a two-way interaction between MARVEL and standardisation bodies has been stimulated.
WP8 led to the definition of all MARVEL general management procedures, including quality assurance and risk analysis, and the definition of the Data Management Plan and MARVEL Innovations. MARVEL Advisory Board was established including 4 external members; their first feedback was received.
WP9 addressed all ethical requirements; the respective risk mitigation and compliance activities of the consortium were reported. The Ethics Advisory Board has been established with 3 external members; their first feedback was received.
The major MARVEL SOTA advances and innovations with significant potential impact on the relevant scientific communities but also socio-economic impact in the smart cities domain are:
WP1 provides a clear specification of the MARVEL E2F2C ubiquitous computing framework, the pilot use cases against which the framework will be validated and a detailed experimental protocol for the components and the overall framework.
WP2 focuses on the efficient and secure data management and distribution aspects of the MARVEL platform. Main goal is the development of the Data Corpus-as-a-Service, where Big Data datasets from the piloting smart city environments will become publicly available to scientific and industrial communities.
In WP3, the project provided significant advances: 1) method for audio and video anonymisation (face/voice swapping, anonymisation at the edge); 2) novel methodologies and algorithms for personalised FL; 3) multi-modal audio-visual AI (AV crowd counting, AV anomaly detection); 4) optimised E2F2C processing and deployment; and 5) novel compression methodologies for edge AI.
In WP4, the project provided significant advances: 1) MEMS microphones plus anonymisation pipeline able to detect speech segments, anonymising the audio stream by removing the speakers attributes while preserving the acoustic environment features, 2) Dual Attention in both time and frequency domain, 3) GPU-accelerated pattern matching tailored for captioning data, 3) SmartViz, an innovative data visualisation toolkit that consists of a set of visualisation tools developed to allow making faster data-driven decisions.
In WP5 tools and methodologies used for 1) cloud infrastructure and service deployment, 2) integration and delivery and 3) benchmarking are based on SOTA methods. In the future, WP5 aims to impact the SOTA of E2F2C benchmarking continuum with the definition of appropriate methods.
In WP6, MARVEL pilots planned ten use cases with spanning societal challenges in the areas of transport, personal safety and security, and crowd monitoring along with quantitative and qualitative (business-related) metrics which will allow to carefully and systematically assess the impact of the innovations. The MARVEL iterative experimental protocol takes into consideration both technical benchmarks and business requirements, aiming to evaluate the performance of the MARVEL solution and validate its alignment with the needs of the industrial users.
The main goal of WP7 is to raise awareness and to make sure that the project results and the technical innovations of MARVEL will be communicated to stakeholders in such a way that the European Data economy will be promoted. More than half of the WP7 progress KPIs were achieved, indicating the major impact of MARVEL early on, the interest of the various stakeholders and the zeal of the partners to spread the outcomes.
WP1 provides a clear specification of the MARVEL E2F2C ubiquitous computing framework, the pilot use cases against which the framework will be validated and a detailed experimental protocol for the components and the overall framework.
WP2 focuses on the efficient and secure data management and distribution aspects of the MARVEL platform. Main goal is the development of the Data Corpus-as-a-Service, where Big Data datasets from the piloting smart city environments will become publicly available to scientific and industrial communities.
In WP3, the project provided significant advances: 1) method for audio and video anonymisation (face/voice swapping, anonymisation at the edge); 2) novel methodologies and algorithms for personalised FL; 3) multi-modal audio-visual AI (AV crowd counting, AV anomaly detection); 4) optimised E2F2C processing and deployment; and 5) novel compression methodologies for edge AI.
In WP4, the project provided significant advances: 1) MEMS microphones plus anonymisation pipeline able to detect speech segments, anonymising the audio stream by removing the speakers attributes while preserving the acoustic environment features, 2) Dual Attention in both time and frequency domain, 3) GPU-accelerated pattern matching tailored for captioning data, 3) SmartViz, an innovative data visualisation toolkit that consists of a set of visualisation tools developed to allow making faster data-driven decisions.
In WP5 tools and methodologies used for 1) cloud infrastructure and service deployment, 2) integration and delivery and 3) benchmarking are based on SOTA methods. In the future, WP5 aims to impact the SOTA of E2F2C benchmarking continuum with the definition of appropriate methods.
In WP6, MARVEL pilots planned ten use cases with spanning societal challenges in the areas of transport, personal safety and security, and crowd monitoring along with quantitative and qualitative (business-related) metrics which will allow to carefully and systematically assess the impact of the innovations. The MARVEL iterative experimental protocol takes into consideration both technical benchmarks and business requirements, aiming to evaluate the performance of the MARVEL solution and validate its alignment with the needs of the industrial users.
The main goal of WP7 is to raise awareness and to make sure that the project results and the technical innovations of MARVEL will be communicated to stakeholders in such a way that the European Data economy will be promoted. More than half of the WP7 progress KPIs were achieved, indicating the major impact of MARVEL early on, the interest of the various stakeholders and the zeal of the partners to spread the outcomes.