Periodic Reporting for period 2 - INSPEX (Integrated Smart Spatial Exploration System)
Periodo di rendicontazione: 2018-07-01 al 2019-12-31
Obstacle avoidance systems for automotive combine multiple sensing technologies (LiDAR, radar, IR and visual) to detect obstacles across all lighting and weather conditions. These systems are large, heavy, not fully integrated, power hungry and require large computational capabilities. They are limited to high-end vehicles. GPS and/or Inertial Measurement Unit (IMU) based traditional navigation systems do not offer safe navigation because un-mapped obstacles cannot be avoided.
INSPEX objective is to make obstacle detection currently implemented in vehicles available as a portable multi-sensor, miniaturized, low power device. The INSPEX spatial exploration system is used for 3D real-time location and warning of static and mobile obstacles under various environmental conditions (indoor and outdoor). Potential applications range from human navigation in reduced visibility (e.g. for firefighters), small robot/drone obstacle avoidance system to navigation for the visually/mobility impaired.
The INSPEX system is integrated in a regular white cane for Visually Impaired and Blind (VIB) and provides 3D spatial audio feedback to its user on obstacle location. This use-case is considered highly demanding in terms of integration challenges, power efficiency and needs for communication with the smart environment. The choice as primary use-case for a smart white cane may have societal impacts. Actually, according to WHO statistics, 285 million people are VIB worldwide and this number is expected to double by 2040, due to aging and health diseases. Moreover, navigating in a city is far from safe for VIB people and sooner or later, they sustain injuries (especially on head and chest) due to unexpected obstacles. Therefore, electronic white canes, able to detect obstacles on the whole person height should improve VIB confidence in their mobility capabilities.
INSPEX objective is to make obstacle detection currently implemented in vehicles available as a portable multi-sensor, miniaturized, low power device. The INSPEX spatial exploration system is used for 3D real-time location and warning of static and mobile obstacles under various environmental conditions (indoor and outdoor). Potential applications range from human navigation in reduced visibility (e.g. for firefighters), small robot/drone obstacle avoidance system to navigation for the visually/mobility impaired.
The INSPEX system is integrated in a regular white cane for Visually Impaired and Blind (VIB) and provides 3D spatial audio feedback to its user on obstacle location. This use-case is considered highly demanding in terms of integration challenges, power efficiency and needs for communication with the smart environment. The choice as primary use-case for a smart white cane may have societal impacts. Actually, according to WHO statistics, 285 million people are VIB worldwide and this number is expected to double by 2040, due to aging and health diseases. Moreover, navigating in a city is far from safe for VIB people and sooner or later, they sustain injuries (especially on head and chest) due to unexpected obstacles. Therefore, electronic white canes, able to detect obstacles on the whole person height should improve VIB confidence in their mobility capabilities.
The project applied a bottom-up approach, starting from end-users' interviews to collect their needs. Use-cases both from user and system perspectives have been derived together with system requirements. Then, an integration strategy has been issued to explore different integration solutions and their fit with users’ needs.
The INSPEX system architecture (HW&SW) has been defined. The system is made of three parts:
- a mobile detection device (MDD) that embeds different range sensing technologies (i.e. LiDAR and TOF-camera, UWB radar, ultrasound) and environmental sensors, an IMU and computational capabilities to fuse data and build a model of the user's surroundings. This model is then analysed to transmit information regarding potential harmful obstacles that enter the user's safety zone;
- smartphone apps that generate 3D spatial sound from information received from the MDD and from the head orientation, and provide feedback from beacon tags spread in the environment, offering context awareness to the user and helping him/her in localizing places of interest (e.g. entrance of a public place) or specific obstacles (e.g. road work barriers);
- a extra-auricular headset that embeds an IMU to send head orientation to the smartphone.
Legal and Ethical issues raised by the development of such a system have been considered from the beginning of the project, together with the legal rules the GRDP imposes.
A survey of several formal tools has been done to evaluate their use in the context of INSPEX firmware development. A bottom-up approach was used to verify key parts of the firmware and improve its reliability. A top-down approach was used to model power management strategy.
Partners brought state-of-the-art range sensors (LiDAR, UWB radar, MEMS ultrasound) to the project. These prototype sensors were first characterized. Then, they have been optimized and miniaturized with drastic effort to reduce their power consumption and meet their individual requirements. These optimized sensors have been manufactured and delivered for integration in the MDD.
Three prototypes have been developed, namely:
- the early prototype integrates off-the-shelf range sensors, IMU and demo-kit computing platforms. Its main goal was to validate the basic concepts the INSPEX system is rooted in, and develop the firmware in parallel to the hardware. Even if this early prototype does not offer the full spectrum of INSPEX functionalities, it has been used during validation tests in laboratory conditions;
- the first integrated prototype integrates part of the functionalities and sensors developed in the course of the project. It has been validated in lab. conditions, following the validation plan;
- the final prototype is an extended version of the first prototype. Extra functionalities have been added to better meet the users' expectations, and validation tests are conducted in real-life conditions.
INSPEX results have been published in world leading journals (e.g. LNCS, Sensors), and conferences (ECC 2019, ICSOFT 2018, IEEE ECTC 2018, DAC 2018, IEEE IWASI 2019, AFFORD 2019), included in the Legal and Ethical areas (e.g. European Journal of Human Rights, CPDP 2019 Conference), and patent applications have been issued on some of its results. Other publications are currently under correction or review.
Communication in large audience journals, workshops and trade fairs has also been done (e.g. International Mobility Conference 2017, Web2Day 2019, Festival della Scienza, L'usine nouvelle).
The INSPEX system architecture (HW&SW) has been defined. The system is made of three parts:
- a mobile detection device (MDD) that embeds different range sensing technologies (i.e. LiDAR and TOF-camera, UWB radar, ultrasound) and environmental sensors, an IMU and computational capabilities to fuse data and build a model of the user's surroundings. This model is then analysed to transmit information regarding potential harmful obstacles that enter the user's safety zone;
- smartphone apps that generate 3D spatial sound from information received from the MDD and from the head orientation, and provide feedback from beacon tags spread in the environment, offering context awareness to the user and helping him/her in localizing places of interest (e.g. entrance of a public place) or specific obstacles (e.g. road work barriers);
- a extra-auricular headset that embeds an IMU to send head orientation to the smartphone.
Legal and Ethical issues raised by the development of such a system have been considered from the beginning of the project, together with the legal rules the GRDP imposes.
A survey of several formal tools has been done to evaluate their use in the context of INSPEX firmware development. A bottom-up approach was used to verify key parts of the firmware and improve its reliability. A top-down approach was used to model power management strategy.
Partners brought state-of-the-art range sensors (LiDAR, UWB radar, MEMS ultrasound) to the project. These prototype sensors were first characterized. Then, they have been optimized and miniaturized with drastic effort to reduce their power consumption and meet their individual requirements. These optimized sensors have been manufactured and delivered for integration in the MDD.
Three prototypes have been developed, namely:
- the early prototype integrates off-the-shelf range sensors, IMU and demo-kit computing platforms. Its main goal was to validate the basic concepts the INSPEX system is rooted in, and develop the firmware in parallel to the hardware. Even if this early prototype does not offer the full spectrum of INSPEX functionalities, it has been used during validation tests in laboratory conditions;
- the first integrated prototype integrates part of the functionalities and sensors developed in the course of the project. It has been validated in lab. conditions, following the validation plan;
- the final prototype is an extended version of the first prototype. Extra functionalities have been added to better meet the users' expectations, and validation tests are conducted in real-life conditions.
INSPEX results have been published in world leading journals (e.g. LNCS, Sensors), and conferences (ECC 2019, ICSOFT 2018, IEEE ECTC 2018, DAC 2018, IEEE IWASI 2019, AFFORD 2019), included in the Legal and Ethical areas (e.g. European Journal of Human Rights, CPDP 2019 Conference), and patent applications have been issued on some of its results. Other publications are currently under correction or review.
Communication in large audience journals, workshops and trade fairs has also been done (e.g. International Mobility Conference 2017, Web2Day 2019, Festival della Scienza, L'usine nouvelle).
INSPEX progress beyond the state of the art includes:
- decrease of power consumption, size and weight of the range sensors integrated in the Mobile Detection Device (ratio 2 to 7 depending on the characteristic and on the sensor)(several publications and patent applications)
- system modular design, esp. of the Mobile Detection Device (HW&SW)(Best presentation award DAC'2018 IP-Track);
- construction of the 3D surroundings model in real-time on a microcontroller (the 2D model was available at the beginning of the project);
- definition of a sensor model for large field-of-view sensors used when the model is computed (patent application);
- modular firmware definition for the data acquisition subsystem (software author right protected by APP French application);
- study of the system reliability that lead to some technical choice in the design and fabrication of the different pieces, especially the mobile detection device;
- 3D audio feedback taking into account the user's head orientation to improve his/her experience and ease the true localisation of obstacles (beta version available at the end of the first reporting period);
- legal and ethical related issues part of the outcomes of the project (several publications in this field).
Note that INSPEX followed two tracks in its development and in routes for future exploitation. Basically, the white cane application, even if of high societal impact, is a niche market. Thus, the range sensors miniaturized in the course of the project willmeet other application domains (e.g. consumer domain): they are developed as independent modules with their own value in the exploitation strategy. The second track is related to the whole INSPEX system, with primary application for the VIB Community.
- decrease of power consumption, size and weight of the range sensors integrated in the Mobile Detection Device (ratio 2 to 7 depending on the characteristic and on the sensor)(several publications and patent applications)
- system modular design, esp. of the Mobile Detection Device (HW&SW)(Best presentation award DAC'2018 IP-Track);
- construction of the 3D surroundings model in real-time on a microcontroller (the 2D model was available at the beginning of the project);
- definition of a sensor model for large field-of-view sensors used when the model is computed (patent application);
- modular firmware definition for the data acquisition subsystem (software author right protected by APP French application);
- study of the system reliability that lead to some technical choice in the design and fabrication of the different pieces, especially the mobile detection device;
- 3D audio feedback taking into account the user's head orientation to improve his/her experience and ease the true localisation of obstacles (beta version available at the end of the first reporting period);
- legal and ethical related issues part of the outcomes of the project (several publications in this field).
Note that INSPEX followed two tracks in its development and in routes for future exploitation. Basically, the white cane application, even if of high societal impact, is a niche market. Thus, the range sensors miniaturized in the course of the project willmeet other application domains (e.g. consumer domain): they are developed as independent modules with their own value in the exploitation strategy. The second track is related to the whole INSPEX system, with primary application for the VIB Community.