Multimodal Extreme Scale Data Analytics for Smart Cities Environments

Project Information

MARVEL

Grant agreement ID: 957337

DOI

10.3030/957337

Project closed

EC signature date 22 July 2020

Start date 1 January 2021

End date 31 December 2023

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 5 998 086,25

EU contribution

€ 5 998 086,25

Coordinated by

IDRYMA TECHNOLOGIAS KAI EREVNAS
Greece

Periodic Reporting for period 2 - MARVEL (Multimodal Extreme Scale Data Analytics for Smart Cities Environments)

Reporting period: 2022-07-01 to 2023-12-31

Summary of the context and overall objectives of the project

MARVEL aspires the convergence of AI, analytics, multimodal perception, software engineering, and HPC in the context of an Edge-Fog-Cloud (E2F2C) Continuum. MARVEL goes beyond traditional Big Data conventional architectures, capitalising on distributed resources and heterogeneous data sources and implementing privacy preservation at all data modalities and all architecture levels. This will enable support of data-driven real-time application workflows and decision-making in modern cities, showcasing the potential to address societal challenges, from increasing public safety and security to analysing traffic flows and traffic behaviour in the cities of Trento and Malta, and will be accomplished through the following objectives:
• Leverage innovative technologies for data acquisition, management and distribution to develop a privacy-aware engineering solution for revealing valuable and hidden societal knowledge in a smart city environment.
• Deliver AI-based multimodal perception and intelligence for audio-visual scene recognition, event detection and situational awareness in a smart city environment.
• Break technological silos, converge very diverse and novel engineering paradigms and establish a distributed and secure E2F2C ubiquitous computing framework in the big data value chain.
• Realise societal opportunities in a smart city environment by validating tools and techniques in real-world settings.
• Foster the European Data Economy vision and create new scientific and business opportunities by offering the MARVEL Data Corpus as a free service and contributing to BDVA standards.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

Project highlights per WP include:
WP1 analysed the benefits of multimodal analysis in smart cities, analysed SOTA and MARVEL’s ambitions beyond SOTA, and defined the experimental protocol, the pilot use cases and the conceptual MARVEL architecture.
WP2 resulted in (1) definition of the use cases’ datasets, (2) development, calibration and connection of the Data Management Platform toolkit, (3) common flexible augmentation strategy, (4) development of the MARVEL Data Corpus and (5) ethics, privacy, and data protection compliance.
WP3 resulted in (1) development of face and voice-swapping techniques and anonymisation components at the edge, (2) novel methodologies and components for AV ML/DL training and inference, personalised federated training, anonymisation, model compression and optimised model deployment, (3) E2F2C Kubernetes-based continuum computing, and (4) the MARVEL AI model repository, a querable database of MARVEL ML/DL model.
WP4 resulted in (1) enhanced audio acquisition device by increasing the number of microphones and device capabilities, (2) a new robust and light-weighted Voice Activity Detection model, (3) deployment and evaluation of GPURegex component, (4) encryption and protection of data using EdgeSec VPN and EdgeSec TEE, and (5) implementation of interactive visualisations and AV analytics tool.
WP5 achieved: (1) infrastructure delivery tailored to application requirements, (2) efficient resource management processes, (3) release of MARVEL Minimum Viable Product (M12), 1st Integrated framework (M18), and final framework with 32 integrated components in 10 use cases, (4) benchmarking strategy implementation, including component-level and end-to-end benchmarking, and (5) conducted scaling and extensibility analysis.
WP6 led to the realisation of 10 real-life use cases from 3 smart cities. The experimental protocol was revisited. All the use cases were evaluated by external end-users.
In WP7, individual exploitation plans and Business Model Canvases were created for selected components, along with market analyses, competitive positioning and financial models tailored for the MARVEL framework and long-term sustainability strategy conducted. Dissemination KPIs were achieved with a highlight of 69 scientific papers.
WP8 led to the definition of all MARVEL general management procedures, including quality assurance and risk analysis, and the definition of the Data Management Plan and MARVEL Innovations. MARVEL Advisory Board was established including 4 external members; continuous feedback was received.
WP9 addressed all ethical requirements; the respective risk mitigation and compliance activities of the consortium were reported. The Ethics Advisory Board has been established with 3 external members; continuous feedback was received.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Key MARVEL advances and innovations with substantial potential impact on scientific and socio-economic realms in smart cities include:
WP1 specified the MARVEL E2F2C ubiquitous computing framework, pilot use cases, and an experimental protocol for components and the overall framework.
WP2 focused on the efficient and secure data management and distribution aspects of the MARVEL platform, aiming to develop the Data Corpus-as-a-Service, a novel Big Data repository ensuring performance, security, and privacy, where datasets from piloting smart city environments along with augmented became publicly available to scientific and industrial communities. Several components provided data management and fusion functionality (StreamHandler, DatAna, DFB and HDD), enabling the efficient parallel processing of several streaming sources.
In WP3, key achievements include: development of the sound event localisation and detection (SELD) component, trained on the UNS data recorded outdoors; integration of voice activity detection (VAD) and audio anonymisation (AudioAnony) methodologies in a single component; New and effective methodologies based on DNNs for VCC, ViAD, AAC, AVAD, AVER; a new methodology for FL in unsupervised settings based on the one-shot clustering; Implementation of new monitoring tools and effective task-scheduling algorithms for the optimisation of execution sites selection; novel methodologies for face swapping for microcontrollers and decentralised learning techniques for anomaly detection on resource constrained devices.
In WP4, key achievements include: 1) MEMS microphones and anonymization pipeline detecting speech segments, preserving acoustic environment features while removing speaker attributes, 2) Dual Attention in time and frequency domains, 3) GPU-accelerated pattern matching for captioning data, 4) EdgeSec TEE and VPN for data protection during processing and exchange, 5) SmartViz, an innovative data visualization toolkit for faster data-driven decisions.
In WP5, tools and methodologies used for 1) integration and delivery, 2) benchmarking on SOTA of E2F2C continuum as well as on individual components and 3) exploitation of large-scale scenarios regarding the scaling-up and extensibility of the MARVEL framework.
In WP6, MARVEL pilots planned ten use cases with spanning societal challenges in the areas of transport, personal safety and security, and crowd monitoring with metrics for assessing innovation impact. The iterative experimental protocol considered technical benchmarks and business requirements for thorough evaluation. Performance was validated for alignment with industrial user needs, and a final impact analysis was conducted with detailed assessments by end-users and stakeholders.
WP7 aimed to raise awareness and promote the technical innovations of MARVEL to stakeholders, fostering the growth of the European Data economy. The majority of WP7 KPIs were met, underscoring the significant impact of the project and the strong commitment of partners to disseminating and exploiting its outcomes.

MARVEL solution for Junction Traffic Trajectory Collection use case in Malta

MARVEL solution for Detecting Criminal and Anti-Social Behaviours use case in Trento

MARVEL conceptual architecture