Hardware Implementation of Pilot-Non-intrusive cOgnitive States Identification System

Project Information

HIPNOSIS

Grant agreement ID: 821466

DOI

10.3030/821466

Project closed

EC signature date 26 September 2018

Start date 1 November 2018

End date 31 October 2021

Funded under

• SOCIETAL CHALLENGES - Smart, Green And Integrated Transport

Total cost

€ 968 166,25

EU contribution

€ 848 963,13

Coordinated by

CSEM CENTRE SUISSE D'ELECTRONIQUE ET DE MICROTECHNIQUE SA - RECHERCHE ET DEVELOPPEMENT
Switzerland

Periodic Reporting for period 2 - HIPNOSIS (Hardware Implementation of Pilot-Non-intrusive cOgnitive States Identification System)

Reporting period: 2020-05-01 to 2021-10-31

Summary of the context and overall objectives of the project

Human fatigue is a serious issue affecting the safety of the traveling public in all modes of transportation. There is evidence that fatigue reduces the capacity for situational awareness, impairs decision making and slows down reaction time. In the safety-pioneering aviation sector, crew monitoring is seen as key to maintaining safety in case of incapacitation, stress, or exhaustion of crew members and for reduced crew operations.

The HIPNOSIS consortium, led by CSEM, has provided the hardware and software implementation of smart sensors to monitor cognitive states of pilots, such as drowsiness, which will allow a great leap in flight safety in the next generation cockpit. The HIPNOSIS approach is based on two sensing systems:
- A vision-based system integrated in the cockpit dashboard to continuously monitor the pilots in the cockpit and detect behavior linked to drowsiness in real time.
- A smart wristband to sense several bio signals through advanced optical sensing technologies, before, during and after the flight, in order to detect physiological measures linked to drowsiness.
The goal of such a user-centric approach is to improve fatigue-related safety and situational awareness and support disruptive cockpit operations. In order to fulfill the abovementioned objectives, HIPNOSIS brings together the latest artificial intelligence (AI) and computer vision techniques as well as optical sensing technologies with aeronautics expertise, contributing to the advent of next-generation cockpits. For such a monitoring system to be successfully adopted, an extremely high level of accuracy and robustness is required, considering aviation-specific environmental factors such as extreme lighting conditions and vibrations.

To design such a tailor-made solution for the aviation industry, the consortium partners worked in close collaboration with the topic leader Honeywell Aerospace, a leader in aerospace development and the manufacturer of avionics solutions. CSEM developed the vision system and machine learning algorithms for detecting behavioral features linked to drowsiness, addressing the unique challenges and requirements in the aeronautic context. In addition, CSEM has developed a smart wristband, based on optical sensing technologies, which allows direct on-body analysis of a pilot’s bio signals. SERMA Ingénierie was responsible for developing the HIPNOSIS vision system hardware, particularly the near-infrared illuminators (in compliance with eye safety regulations) and a system for synchronizing them with image capture, as well as integrating the system into the cockpit simulator. French startup Innov+, which already commercializes similar solutions for the automotive industry, contributed to the development of the HIPNOSIS software API.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

In order to design and implement a system for pilot monitoring in the cockpit, system specifications were first identified on the basis of the requirements in the aeronautics.

A non-intrusive intelligent vision system was designed and developed to monitor pilots and detect signs of drowsiness. A first version of the vision system was implemented and installed in the cockpit simulator for the data collection. The hardware and software systems were then modified towards a second system design.
A multi-camera system was developed to meet the requirements established in the first phase, including 4 near infrared (NIR) off-the-shelf cameras and 4 custom-made illuminators. The illuminators allow the selection of pulse duration and illumination intensity, so that images can be captured with a short exposure time for sharper images when there is motion. It also automatically regulates the orders received so as not to exceed the safety and health limits.
State-of-the-art computer vision and machine learning techniques were implemented to provide behavioral features linked to drowsiness in real time at 40 frames per second. The developed algorithmic pipeline, which is a combination of data-driven and analytics approaches, includes face detection and facial landmark localization, as well as the iris, glints, and pupil detection modules. The key features extracted by the vision system are 3D head pose, blink rate and duration, percentage of eye closure (PERCLOS), and mean capped eyelid closure over time (MCEC). These key behavioral features are indicators of the pilots’ drowsiness level.
The algorithmic pipeline was developed and iteratively optimized to obtain the speed and accuracy required for the aviation use cases. The speed requirement is critical for some of the features to be detected, such as blinks which have an average duration of 100-150ms. The vision system software, including the data acquisition module and the algorithmic pipeline, runs in real time within less than 25ms on the HIPNOSIS computer thanks to the multi-threading architecture, with a latency of about 50ms.

The HIPNOSIS wristband, based on optical sensing technologies, allows direct on-body analysis of pilot’s bio signals. The wristband includes monitoring of heart rate, heart rate variability, breathing rate, skin temperature, activity, and sleep analysis. CSEM optical Heart Rate Variability (HRV) monitoring relies on photoplethysmography (PPG) signals of high temporal resolution. This technology allows the detection of single cardiac beat events and the measurement of HRV in the time and frequency domains (as defined in the European HRV guidelines). This is computed via CSEM proprietary analysis of arterial pulsatility patterns at any location on body with sufficient perfusion.

The work performed in the context of the project was presented internally by the consortium partners inside their organizations. A press release was made at the beginning of the project, which achieved good coverage by the media, including several interviews and articles. The project results have been presented to potential industrial entities who have shown interest in the technology, which has led to discussions on possible future partnerships and successful acquisition of new projects.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

By providing hardware and software implementation of smart sensors to monitor cognitive states of pilots, such as drowsiness, HIPNOSIS will give a new dimension and new functionalities to cockpit instruments, allowing a great leap towards improving situation awareness and flight safety.

HIPNOSIS will have a positive impact on the European aviation industry based on the acquired knowledge and experience with the first TRL 6 cognitive states detection system coupled with avionics. The industry (avionics and aircraft manufacturers, airlines) will be well positioned to develop systems that make use of the data generated, thanks to the consortium’s proven successes in maturing technologies developed in Cleansky, to high TRL industrial developments. This can lead to lower costs for European airlines by reducing the incidence of equipment failures (better understanding of instruments) and accidents.

Beyond the aeronautical domain, the developed system will have a potential impact on smart vision systems for drowsiness detection in general, after it has been tested and validated in the demanding environment of a cockpit. Automotive applications and the railway industry to quote a few, can strongly benefit from these improvements.

The HIPNOSIS prototype installed in the cockpit simulator,with C1-4(cameras) and I1-4(illuminators)

HIPNOSIS wristband design architecture.

HIPNOSIS illuminators’ design

Detection of face, facial landmarks, head orientation, iris, glints, and pupil using the H