DISTRIBUTED MULTI-SENSOR SYSTEMS FOR HUMAN SAFETY AND HEALTH

Project Information

DistriMuSe

Grant agreement ID: 101139769

DOI

10.3030/101139769

EC signature date 28 June 2024

Start date 1 May 2024

End date 30 April 2027

Funded under

• Digital, Industry and Space

Total cost

€ 34 764 819,75

EU contribution

€ 11 274 522,51

Coordinated by

TEKNOLOGIAN TUTKIMUSKESKUS VTT OY
Finland

Periodic Reporting for period 2 - DistriMuSe (DISTRIBUTED MULTI-SENSOR SYSTEMS FOR HUMAN SAFETY AND HEALTH)

Reporting period: 2025-05-01 to 2026-04-30

Summary of the context and overall objectives of the project

We are surrounded by a variety of more-or-less intelligent technical devices, designed to serve you us or others. Applications onin your mobile phones, wrist-worn health sensors on your wrists, autonomous vacuum cleaners, robots on the factory floor and increasingly autonomous cars – all pledge to ease your tasks and keep usyou safe and healthy. The seamless interplay with these devices gains more importane as these devices proliferate and grow in t with the increased autonomy and pervasive presence of the devices. We expect continuously available support fromin the services they provide − yet we want them to disappear unobtrusively in the background when not needed. In order to provide support in a collaborative environment with human, physical and digital players, the technology needs to be equipped with senses to grasp human presence, their mental and physical state, their activities and their intentions. This is required to ensure human safety, safeguard their health, and allow for natural interaction. This project intends to improve sensing of human presence, behaviour and health in a collaborative or common environment by means of multi-sensor systems.
Objectives:
O1: Optimised innovative sensors and sensor systems for advanced human observation, measurement, and interaction
O2: ML-based data analytics, fusion and decision-making methodologies for (semi) real-time multi-sensor information processing
O3: Supporting distribution of computation algorithms across available resources from edge to cloud
O4: Building a comprehensive understanding of human physical health, mental state, behaviour, intentions, and safety risks
O5: Validating the technologies in human-centric use cases related to health and wellbeing monitoring and support; ensuring traffic safety especially for VRUs; safe interaction with robots and automated systems

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The main work performed and achievements can be described in relation to the objectives of the project:

O1 status: During the reporting period, partners finalised and tested their sensing technologies and integrated them into demonstrators for validation in the pilots. New sensing technologies developed for the project included radars (FMCW, UWB, polarimetric), LiDARs, IMUs (gyroscope, accelerometer, car based, in wearable), vital sign wearables (ECG, PPG, blood pressure), bed sensors (BCG), cameras (visible light, thermal, in car environment, for gaze tracking), positioning (UWB/FMCW/BLE based). Most reached the current level of the state of the art, and were tested for feasibility in the first cycle pilots. The basis for taking them beyond the state of the art in the next cycle is thus established.
O2 status: Data was collected partially from previous pilots, or from public sources, as well as from early runs of pilots. Initial versions of algorithms and ML-based data analytics and fusion models have been developed for the foreseen needs in the demonstrators and pilots. Updates of these algorithms and models will be developed in the second cycle.
O3 status:WP4 has continued to evaluate the suitability of the reference platform for the particular needs of the various demonstrators and pilots. Metrics for successful performance in the various domains were assessed and evaluated for a variety of applications. Uptake of the platform was facilitated and advocated in WP4. Not all demonstrators will be able to adopt the platform due to a variety of restrictions, often related to the domain (e.g. regulation or established standard architectures). An overview of reasons why (not) to adopt the platform was made.
O4 status: 15 Pilots were planned to evaluate the demonstrators developed for the use cases. The validation of these pilots was reported in the deliverable D5.3 which is publicly available and provides a good overview of the intermediate outcomes of the project. As part of the pilot planning, focus group discussions were held to review the ethical implications of the intended applications. Ethical permission was also applied for the pilots when necessary. The nature of most of the pilot studies emphasised the study of the technical feasibility of the approach as well as allowed for the collection of realistic data and user feedback. Most pilots were executed according to the original schedule, but the Finnish pilots were delayed too much for reporting in the deliverables for the first cycle and will be reported later. Some pilots will also continue to run in between of the two pilot periods (e.g. the sleep monitoring pilots) in order to maximise the data collection.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The project advances beyond the state of the art through technology development, demonstrator integration and real-world piloting. In its first of two cycles, it completed development and piloting across 15 use cases in health, mobility and robotics, establishing the technical baseline for the second cycle.
Sensing work included dedicated sensors, such as a polarised imaging FMCW radar, and extensive sensor fusion for indoor positioning and 3D environment mapping using IMUs, BLE, cameras, radar and lidar. Fusion methods ranged from early in-sensor fusion to late and cooperative fusion.
Machine learning was mainly applied in mobility and robotics demonstrators, with some solutions adding explainability to improve user understanding and trust.
A Zenoh-based reference platform for distributed sensing was defined and partly adopted by about two thirds of the pilots, supporting orchestration, distributed computing and component communication.

The pilots cover the following main topics:

Use case 1: Continuous hybrid health monitoring, covering elderly care, sleep and exercise pilots.
Elderly pilots in Lleida, Geel/Ghent and Kuopio/Helsinki studied home activity, frailty, Parkinson’s disease and mild cognitive decline using wearables, bed sensors and radar in both long-term monitoring and controlled tests.
Sleep pilots in Kempenhaeghe and Kuopio developed unobtrusive home screening methods for sleep disorders such as apnea, using bed sensors, radar and continuous blood-pressure sensing with healthy volunteers and sleep-lab patients.
Exercise pilots in Brno and Torino ranged from exertion testing in healthy volunteers to gait analysis in frail individuals with Parkinson’s disease.

Use case 2: Situational awareness for VRU and driver safety.
This use case addresses collaborative situational awareness and driver distraction.
Collaborative situational awareness pilots in Brussels, Hamburg and Kortrijk used roadside radar, lidar and cameras to develop sensor fusion, world mapping and traffic-participant digital twins.
Driver distraction was studied in Reggio Emilia using a driving simulator for adaptive HMI development, and in Tampere by matching driver gaze with traffic objects detected by lidar and camera.

Use case 3: Safe interaction and cooperation with robots.
The robotics use case includes two human–robot collaboration pilots and one simulation pilot.
In Best, a palletizing robot worked alongside operators while IMU and camera monitoring supported safety. Son and Bremen pilots combined radar with lidar, RGB, thermal and DVS cameras to detect humans and maintain safe robot operation.
The simulation environment supported preparation for the Best pilot and evaluation of user-related KPIs such as mental stress, situational awareness and emotional response.