Skip to main content

Hardware Implementation of Pilot-Non-intrusive cOgnitive States Identification System

Periodic Reporting for period 1 - HIPNOSIS (Hardware Implementation of Pilot-Non-intrusive cOgnitive States Identification System)

Reporting period: 2018-11-01 to 2020-04-30

European Clean Sky project HIPNOSIS
Artificial intelligence to monitor pilot drowsiness

Neuchatel, 25 February, 2019—Providing tools to evaluate pilots’ fatigue state: this is the aim of the Clean Sky project HIPNOSIS. Coordinated by CSEM and under the guidance of Honeywell, the project will combine artificial intelligence (AI) with aeronautics expertise, contributing to the advent of next- generation cockpits. Consisting of smart cameras and wearable electronics, a safety kit will enable the real-time detection of signs of drowsiness, thus improving fatigue-risk management.
Last November, an Australian pilot fell asleep while operating a passenger flight, overshooting its destination by 50 kilometers. A few months earlier, in the US, investigators found that an air disaster had been narrowly avoided in San Francisco the previous year. Once again, the danger had been brought about by a pilot’s lack of sleep.
Fortunately these incidents had no serious repercussions, but they are not isolated cases. Human fatigue is a serious issue affecting the safety of the traveling public in all modes of transportation. Nearly 20 percent of the major US Transportation Safety Board investigations completed between 2001 and 2012 identified fatigue as a probable cause, contributing factor, or a finding.
To address this issue, the HIPNOSIS consortium, led by CSEM, aims to improve the evaluation of pilot fatigue by providing innovative monitoring tools—namely, a specific vision-based system combined with a bio-physiological signal sensor.
Machine learning at the service of onboard safety
HIPNOSIS won the tender launched by the Clean Sky 2 Joint Undertaking, a European research program dedicated to aeronautics. Andrea Dunbar, head of Embedded Vision Systems at CSEM, explains its main features: “We will implement computer vision and machine learning algorithms in order to detect signs of drowsiness in pilots in real time.” These algorithms will be integrated into a specific camera developed by the French startup Innov+, which already commercializes similar solutions for the automotive industry.
“CSEM will also use its know-how in the measurement of physiological parameters to develop a wearable sensor that monitors pilots before and during a flight,” Dunbar adds. “The collected data will be fused with eye-gaze-related measures as well as head pose, observed by the vision system.” The French company SERMA Ingénierie will be in charge of integrating HIPNOSIS into a cockpit prototype for preliminary testing.
A tailor-made solution for the aviation industry
Honeywell Aerospace—Clean Sky II Core Partner and a leader in aerospace development and the manufacturer of avionics solutions—defines requirements for the technology developed by the HIPNOSIS project and integrates it into the overall pilot monitoring system. “HIPNOSIS will deliver key enablers for the introduction of pilot monitoring and for turning this technology into aviation reality,” enthuses Bohdan Blaha, project manager at Honeywell Aerospace. “HIPNOSIS provides building blocks critical to successfully demonstrating the potential of this technology and its benefit to the aviation industry.” Final results are expected in 2021.
Main achievements during the first reporting period:

Specification of sensors for the aircraft environment
- Vision system specifications, for HW and SW, were identified based on the constraint in the cockpit simulator at Honeywell and according the requirements defined by the topic leader.
- Wristband specifications were identified based on the requirements defined by the topic leader.

Development of the first system design and installation in the cockpit demonstrator
- A multi-camera system was designed according to the use cases and constraints
- Off-the-shelf HW components were selected based on the identified requirements
- The active illuminators were evaluated in terms of the photobiological risks in the expected operation conditions in the cockpit simulator to ensure
- HIPNOSIS API software was developed to acquire synchronized data from the multi-camera system
- Camera calibration algorithms (internal and external calibration) were developed and integrated in the HIPNOSIS SW
- The HIPNOSIS system was installed in the cockpit simulator at Honeywell and its functionality was validated

Development of the second system design
- New illuminators are being developed which are more powerful and provide maximum illumination according to the health limits
- Vision system algorithms for face detection, facial landmark localization, head-pose estimation, and iris detection were developed and optimized for the multi-camera system
- The design and development of the wristband HW and firmware is being carried out and the required algorithms for extracting the physiological features, such as heart rate, breathing rate, etc. are being developed

- Data management plan was established
- Plan for communication, dissemination and exploitation of the project results was established
- Infrastructure for storing 10s of TB of video data on CSEM servers prepared, in compliance with GDPR
HIPNOSIS will contribute to develop the next generation cockpit by providing a hardware and SW implementation of smart sensors to monitor cognitive states such as drowsiness, which will allow a great leap towards improving flight safety.
By providing a validated system to detect the cognitive states of pilots, HIPNOSIS will give a new dimension and new functionalities to cockpit instruments.
HIPNOSIS will have a positive impact on the European aviation industry by generating knowledge and experience with the first TRL6 cognitive states detection system coupled with avionics. The industry (avionics and aircraft manufacturers, airlines) will be well positioned to develop systems that make use of data generated, also thanks to the consortium’s proven successes in maturing technologies developed in Cleansky, to high TRL industrial developments.

Beyond, the aeronautical domain, the developed system will have a potential impact on smart vision systems for drowsiness detection in general, after it has been tested and validated in the demanding environment of a cockpit. Automotive applications, railway industry to quote a few, can strongly benefit from these improvements.

Main expected results:
- Developing a calibrated multi-camera system, capable of extracting features related to drowsiness (e.g. PERCLOS, blink rate/duration, etc.) in real time (60 frames per second).
- Developing a wristband to provide physiological signals related to drowsiness (e.g. heart rate, breathing rate, etc.).
Wristband design architecture.
HIPNOSIS hardware system installed in the cockpit simulator at Honeywell.
Results of iris detection on a single frame.
Hardware design for the new illuminators.
Results of face detection and facial landmark localization on the 5 camera views.