Periodic Reporting for period 2 - NextPerception (NextPerception - Next generation smart perception sensors and distributed intelligence for proactive human monitoring in health, wellbeing, and automotive systems)
Reporting period: 2021-05-01 to 2022-04-30
We put our lives increasingly in the hands of smart complex systems making decisions that directly affect our health and wellbeing. This is very evident in healthcare – where systems watch over your health – as well as in traffic – where autonomous driving solutions are gradually taking over control of the car. The accuracy and timeliness of the decisions depend on the systems’ ability to build a good understanding of both you and your environment, which relies on observations and the ability to reason on them.
This project will bring perception sensing technologies like Radar, LiDAR and cameras to the next level, enhancing their features to allow for more accurate detection of human behaviour and physiological parameters. Besides more accurate automotive solutions ensuring driver vigilance and pedestrian and cyclist safety, this innovation will open up new opportunities in health and wellbeing to monitor elderly people at home or unobtrusively assess health state.
To facilitate building the complex smart sensing systems envisioned and ensure their secure and reliable operation, the new Distributed Intelligence paradigm will be embraced, enhanced and supported by tools. It leverages the advantages of Edge and Cloud computing, building on the distributed computational resources increasingly available in sensors and edge components to distribute also the intelligence.
The goal of this project is to develop next generation smart perception sensors and enhance the distributed intelligence paradigm to build versatile, secure, reliable, and proactive human monitoring solutions for the health, wellbeing, and automotive domains
NextPerception aims at the exploitation of enhanced sensing and sensor networks focusing on the application domains of Health, Wellbeing and Automotive. The starting point is to acquire quality information from sensors nodes (O1). The next step is to transform the data provided by collections of networked sensors into information suitable for proactive decision making (covering a.o. clinical decision making, wellbeing coaching and vehicle control decisions (O2). The provision of a reference platform to support the design, implementation, and management of distributed sensing and intelligence solutions (O3) greatly facilitates the use and implementation of distributed sensor networks, enabling their practical application in the primary domains of healthcare, wellbeing, and automotive as well as in cross-sector applications. The demonstration of these applications (O4) is key to achieve potential products and services that can finally contribute to the solution of the Global Challenges in Health and Wellbeing, and Transport and Smart Mobility.
The project brings together 43 partners from 7 countries. The partners are major industrial players and prominent research centres and universities and will address top challenges in health, wellbeing, and automotive domains through three use cases: integral vitality monitoring for elderly and exercise, driver monitoring, and providing safety and comfort for vulnerable road users at intersections.
This project will bring perception sensing technologies like Radar, LiDAR and cameras to the next level, enhancing their features to allow for more accurate detection of human behaviour and physiological parameters. Besides more accurate automotive solutions ensuring driver vigilance and pedestrian and cyclist safety, this innovation will open up new opportunities in health and wellbeing to monitor elderly people at home or unobtrusively assess health state.
To facilitate building the complex smart sensing systems envisioned and ensure their secure and reliable operation, the new Distributed Intelligence paradigm will be embraced, enhanced and supported by tools. It leverages the advantages of Edge and Cloud computing, building on the distributed computational resources increasingly available in sensors and edge components to distribute also the intelligence.
The goal of this project is to develop next generation smart perception sensors and enhance the distributed intelligence paradigm to build versatile, secure, reliable, and proactive human monitoring solutions for the health, wellbeing, and automotive domains
NextPerception aims at the exploitation of enhanced sensing and sensor networks focusing on the application domains of Health, Wellbeing and Automotive. The starting point is to acquire quality information from sensors nodes (O1). The next step is to transform the data provided by collections of networked sensors into information suitable for proactive decision making (covering a.o. clinical decision making, wellbeing coaching and vehicle control decisions (O2). The provision of a reference platform to support the design, implementation, and management of distributed sensing and intelligence solutions (O3) greatly facilitates the use and implementation of distributed sensor networks, enabling their practical application in the primary domains of healthcare, wellbeing, and automotive as well as in cross-sector applications. The demonstration of these applications (O4) is key to achieve potential products and services that can finally contribute to the solution of the Global Challenges in Health and Wellbeing, and Transport and Smart Mobility.
The project brings together 43 partners from 7 countries. The partners are major industrial players and prominent research centres and universities and will address top challenges in health, wellbeing, and automotive domains through three use cases: integral vitality monitoring for elderly and exercise, driver monitoring, and providing safety and comfort for vulnerable road users at intersections.
The project has successfully completed the first of two cycles focusing on integrating and validating state of the art solutions for the various use cases. The second cycle has started, benefiting from the lessons learnt from the first cycle and adjusting the plans accordingly. The development in the second cycle is expected to go beyond the state of the art and validation will be done in settings further approaching real-life situations and involving actual users where possible.
The status of the project’s three use cases is currently:
UC1 Integral Vitality Monitoring – This use case focuses on elderly and activity monitoring and has four demonstrators validated in six pilots:
- Continuous activity monitoring (UC1D1) - indoor positioning, radar-based vital sign monitoring, and camera assisted activity monitoring for the elderly and activity monitoring solutions piloted in the laboratory and the Lleida living lab in Spain.
- Episodic Health Gate (UC1D2) - a radar-based vital sign monitoring and weight sensing chair for frailty detection in elderly piloted in a care facility in Kuopio, Finland.
- Wearable Activity Monitoring (UC1D3) - a wearable devices-based exercise monitoring solution including sweat analysis piloted in outdoor exercises.
- Sleep Monitoring (UC1D4) - a radar-based sleep monitoring solution piloted in Kempenhaeghe, The Netherlands
UC2 Driver monitoring – This use case addresses the topic of driver monitoring in the context of partial automated driving. It includes three demonstrators and two pilots:
- Driver simulator (UC2D1) – integrated system for assessing the Driver Complex State (DCS, a combination of cognitive, behavioural, and emotional states) and derived a “fitness to drive index” by means of different unobtrusive sensors (cameras, biofeedback sensors, driving data).
- Passenger car (UC2D2) and heavy-duty vehicle (UC2D3) – both car and truck feature an FMCW radar to monitor the driver’s vital signs as well as a gaze detection system. Pilot validation was done in real traffic situations.
UC3 Safety and Comfort at Intersections – constitutes three main topics:
- Vehicle” (UC3a) - has developed sensing solutions and intelligent algorithms for improving ego-localisation and vehicle environment monitoring – particularly for vulnerable road user (VRU) detection by means of IMU’s, Lidar, cameras and vehicle data.
- Communication + GNSS (UC3b) - worked on a UWB based positioning system with accurate synchronisation and highly secure solutions to improve GNSS based positioning robustness and reliability.
- Infrastructure” (UC3c) - focused on the development of road-side sensing solutions utilizing radar, camera, thermal camera and time of flight camera for the detection of VRU’s and their behaviour.
The status of the project’s three use cases is currently:
UC1 Integral Vitality Monitoring – This use case focuses on elderly and activity monitoring and has four demonstrators validated in six pilots:
- Continuous activity monitoring (UC1D1) - indoor positioning, radar-based vital sign monitoring, and camera assisted activity monitoring for the elderly and activity monitoring solutions piloted in the laboratory and the Lleida living lab in Spain.
- Episodic Health Gate (UC1D2) - a radar-based vital sign monitoring and weight sensing chair for frailty detection in elderly piloted in a care facility in Kuopio, Finland.
- Wearable Activity Monitoring (UC1D3) - a wearable devices-based exercise monitoring solution including sweat analysis piloted in outdoor exercises.
- Sleep Monitoring (UC1D4) - a radar-based sleep monitoring solution piloted in Kempenhaeghe, The Netherlands
UC2 Driver monitoring – This use case addresses the topic of driver monitoring in the context of partial automated driving. It includes three demonstrators and two pilots:
- Driver simulator (UC2D1) – integrated system for assessing the Driver Complex State (DCS, a combination of cognitive, behavioural, and emotional states) and derived a “fitness to drive index” by means of different unobtrusive sensors (cameras, biofeedback sensors, driving data).
- Passenger car (UC2D2) and heavy-duty vehicle (UC2D3) – both car and truck feature an FMCW radar to monitor the driver’s vital signs as well as a gaze detection system. Pilot validation was done in real traffic situations.
UC3 Safety and Comfort at Intersections – constitutes three main topics:
- Vehicle” (UC3a) - has developed sensing solutions and intelligent algorithms for improving ego-localisation and vehicle environment monitoring – particularly for vulnerable road user (VRU) detection by means of IMU’s, Lidar, cameras and vehicle data.
- Communication + GNSS (UC3b) - worked on a UWB based positioning system with accurate synchronisation and highly secure solutions to improve GNSS based positioning robustness and reliability.
- Infrastructure” (UC3c) - focused on the development of road-side sensing solutions utilizing radar, camera, thermal camera and time of flight camera for the detection of VRU’s and their behaviour.
The current proof of concept technologies and demonstrators in the project reached the current state of the art and in some cases surpassed it, while being applied to innovative use cases. All use cases showed good results with prospects to achieve the targeted results by the end of the project. In sensor development, for example the radar technology has proven capable of providing human physiological signals and behaviour in a variety of conditions. Initial algorithms e.g. for improved positioning were already deployed in various use cases. Support for the distributed platform was found e.g. in the Zenoh framework. Further development of data fusion algorithms and AI for decision making is expected to benefit from the data collected in the first cycle.
The project results will have immediate impact on the addressed domains of healthcare and mobility - particularly in the addressed use cases - as well as indirect impact on any domain deploying smart sensing systems. The distributed sensing methodology will deploy innovating sensor technologies for human sensing and facilitate the design of complex multi-sensor systems and related intelligent algorithm development and deployment, decreasing development time. Systems will become more reliable and robust, and enable improved insight in the user/environment. The project will be instrumental in leveraging the technologies for important future challenges in the health and automotive domains.
The project results will have immediate impact on the addressed domains of healthcare and mobility - particularly in the addressed use cases - as well as indirect impact on any domain deploying smart sensing systems. The distributed sensing methodology will deploy innovating sensor technologies for human sensing and facilitate the design of complex multi-sensor systems and related intelligent algorithm development and deployment, decreasing development time. Systems will become more reliable and robust, and enable improved insight in the user/environment. The project will be instrumental in leveraging the technologies for important future challenges in the health and automotive domains.