Periodic Reporting for period 2 - HELIAUS (tHErmaL vIsion AUgmented awarenesS)
Periodo di rendicontazione: 2020-04-01 al 2021-03-31
The new ways of driving and use a vehicle as expected in the scope of Smart Mobility asks for reliable and affordable versatile perception systems. It is now clear that the interaction between the vehicles and its environment (in and out of cabin) has to be improved. As the automotive market has a strong strategy towards the development of autonomous vehicles, this need even more important in the case of autonomous driving for which the four mandatory steps (starting with “hands on” and ending with “hands off”, “eyes off” and “minds off”) are full of technical hurdles, barriers, obstacles.
Perception systems need to be accurate and reliable both for what happen in the cabin and out of the cabin. The driver and passenger monitoring needs to be improved as well as the environment surrounding monitoring, in all light and weather conditions. For both of these topics, the extension of current perception systems (mainly based on visible imaging, LIDAR and RADAR detection) to the thermal sensing is a must. Thermal sensing, especially in the Long Wavelength Infrared or LWIR (= 6 to 14 μm) bandwidth, provides valuable additional information and has to be considered as a must for the next generation of L4 (‘’Eyes off’’) and L5 (‘’Mind off’’) autonomous driving.
In this context, the HELIAUS project aims to deliver breakthrough perception systems for in-cabin passengers monitoring, as well as for the car surrounding by developing smart thermal perception systems that extend the current systems to the LWIR bandwidth.
The HELIAUS project focuses on the Sense and Think part of the perception process for both in and out-of-cabin applications.
The HELIAUS project integrates high performance and low cost LWIR thermal systems, images / data processing units and software (SW) applications into smart thermal perception systems compatible to such applications. The thermal perception systems can be used solely or in combination with others sensors commonly used for the applications, as vision sensors for in-cabin monitoring or LIDAR for out-of cabin application, leading to an augmented awareness and reliability.
From general point of view, the main objectives of the HELIAUS project and work to be carried out are as follows:
- to specify, develop, test and validate the thermal perception systems, for in-cabin and out-of cabin application,
- to develop cutting-edge and cost effective technologies leading to low cost, high performance LWIR module,
- to quantify the valuable addition of the thermal sensing into the current systems and future systems,
- to promote the benefit of such systems in the future autonomous vehicles, in particular, and for smart mobility in general.
Perception systems need to be accurate and reliable both for what happen in the cabin and out of the cabin. The driver and passenger monitoring needs to be improved as well as the environment surrounding monitoring, in all light and weather conditions. For both of these topics, the extension of current perception systems (mainly based on visible imaging, LIDAR and RADAR detection) to the thermal sensing is a must. Thermal sensing, especially in the Long Wavelength Infrared or LWIR (= 6 to 14 μm) bandwidth, provides valuable additional information and has to be considered as a must for the next generation of L4 (‘’Eyes off’’) and L5 (‘’Mind off’’) autonomous driving.
In this context, the HELIAUS project aims to deliver breakthrough perception systems for in-cabin passengers monitoring, as well as for the car surrounding by developing smart thermal perception systems that extend the current systems to the LWIR bandwidth.
The HELIAUS project focuses on the Sense and Think part of the perception process for both in and out-of-cabin applications.
The HELIAUS project integrates high performance and low cost LWIR thermal systems, images / data processing units and software (SW) applications into smart thermal perception systems compatible to such applications. The thermal perception systems can be used solely or in combination with others sensors commonly used for the applications, as vision sensors for in-cabin monitoring or LIDAR for out-of cabin application, leading to an augmented awareness and reliability.
From general point of view, the main objectives of the HELIAUS project and work to be carried out are as follows:
- to specify, develop, test and validate the thermal perception systems, for in-cabin and out-of cabin application,
- to develop cutting-edge and cost effective technologies leading to low cost, high performance LWIR module,
- to quantify the valuable addition of the thermal sensing into the current systems and future systems,
- to promote the benefit of such systems in the future autonomous vehicles, in particular, and for smart mobility in general.
Despite COVID 19 situation and planning disruption the HELIAUS project follow their objectives and major technological blocks are on track.
The Heliaus project is divided in 3 main areas:
-Project and innovation management
-Disruptive technologies development
- Demonstrator building & uses-cases evaluation
Project and innovation management actions achieved:
- Set up an efficient project organization with project planning, collaborative platform, monthly calls & reporting to overpass COVID pandemic delays
- Support innovation by creating communication tools as graphic charter and website
- Support dissemination activities targeted by the project leading to value creation
Disruptive technologies development actions achieved:
- Innovative thermal sensor and packaging design to improve cost and robustness
- Innovative process leading to reduce cost and to supply first prototypes
- Laboratory testing of first prototypes
- Design and prototypes innovative thermal image processing algorithm
- Specifications of a SW/HW Platform for thermal image processing
- Testing different ways of software embedding
- Design and prototype lenses for innovative sensors
- Design and test new lens manufacturing and assembly processes for large volumes production leading to lower cost
-
- Demonstrator building & uses-cases evaluation actions achieved
- Release specifications of the demonstrators
- Demonstrators supply to HELIAUS partners
- Thermal IR (TIR) image simulator developed for generation of physically-realistic TIR images with a large number of parameters
- Development of a multi-sensor data acquisition system for in and out of cabin
- Dataset record and labeling for Computer Vision
- Training and performance metric of Computer Vision with thermal images
The Heliaus project is divided in 3 main areas:
-Project and innovation management
-Disruptive technologies development
- Demonstrator building & uses-cases evaluation
Project and innovation management actions achieved:
- Set up an efficient project organization with project planning, collaborative platform, monthly calls & reporting to overpass COVID pandemic delays
- Support innovation by creating communication tools as graphic charter and website
- Support dissemination activities targeted by the project leading to value creation
Disruptive technologies development actions achieved:
- Innovative thermal sensor and packaging design to improve cost and robustness
- Innovative process leading to reduce cost and to supply first prototypes
- Laboratory testing of first prototypes
- Design and prototypes innovative thermal image processing algorithm
- Specifications of a SW/HW Platform for thermal image processing
- Testing different ways of software embedding
- Design and prototype lenses for innovative sensors
- Design and test new lens manufacturing and assembly processes for large volumes production leading to lower cost
-
- Demonstrator building & uses-cases evaluation actions achieved
- Release specifications of the demonstrators
- Demonstrators supply to HELIAUS partners
- Thermal IR (TIR) image simulator developed for generation of physically-realistic TIR images with a large number of parameters
- Development of a multi-sensor data acquisition system for in and out of cabin
- Dataset record and labeling for Computer Vision
- Training and performance metric of Computer Vision with thermal images
The major concrete expected impact items of the HELIAUS project are threefold and as follows:
- technological impact items expected from technology developments:
- Sensor: cost reduction thanks to innovative design and process
- Thermal image processing: optimize for the applications, markets and costs targeted
- Optics: cost reduction thanks to innovative design and process
- innovative IR modules and Smart IR modules to find the best compromise of performance and cost
- applicative impact items expected from in-cabin and car environment perception developments and evaluations.
- Specification of the system
- Demonstration of computer vision thermal-based ADAS applications improvement
- technological impact items expected from technology developments:
- Sensor: cost reduction thanks to innovative design and process
- Thermal image processing: optimize for the applications, markets and costs targeted
- Optics: cost reduction thanks to innovative design and process
- innovative IR modules and Smart IR modules to find the best compromise of performance and cost
- applicative impact items expected from in-cabin and car environment perception developments and evaluations.
- Specification of the system
- Demonstration of computer vision thermal-based ADAS applications improvement