Cockpit camera system monitors pilots’ eye-gaze and tracks hand-gestures
Nowadays, a typical commercial cockpit features several large computer monitors, many secondary dials, and hundreds of switches. Although such designs work well, with higher degrees of automation in the cockpits, it is crucial to optimise human-computer interaction to enhance pilots’ efficiency and flight safety. The EU-funded PEGGASUS project has advanced a novel human-machine interface system, addressing the unique challenges in the aeronautic environment. A dashboard-mounted camera system was developed which tracks the pilots’ gaze direction while also recognising their hand gestures. Its purpose is to allow smooth feedback to the pilots, helping to reduce workload and improve situational awareness when they handle multiple actions.
Tracking eye gaze
The system features several cameras that face the pilot. The multi-camera system operates in the near-infrared range, necessary because cockpits experience extreme variation in light intensity, being sometimes very bright and at other times, dark. The cameras pair with state-of-the-art Edge AI allowing real-time determination of where the pilots are looking. “We have developed an algorithmic pipeline, combining data-driven and analytics approaches, to accurately detect eye gaze in real time and with minimum latency,” says Sareh Saeedi, project manager at PEGGASUS host company, Swiss Center for Electronics and Microtechnology (CSEM). State-of-the-art computer vision and machine learning techniques have been used. “First, the pilot’s face is detected in the images. Then, several landmarks on the face and around the eyes are localised, followed by the detection of pupil and the estimation of the centre of the corneal curvature,” explains Saeedi. The result is a multi-camera vision system, capable of operating in real time at 60 frames per second with a latency of 32 ms, which achieves better than 1° accuracy for gaze detection.
Gesture-recognition
The system also tracks five predefined hand gestures. To do so, researchers have developed a deep neural network model, which was incorporated in the system to provide a new interaction modality. The PEGGASUS system was installed in a cockpit simulator at Lufthansa Aviation Training in Switzerland, where 10 professional pilots evaluated it. “We got very positive feedback from the pilots,” adds Gilad Scherpf, head of Aviation Training Development Lufthansa Group. “Overall, they rated the PEGGASUS vision system better than head-mounted eye-tracking systems in terms of comfort and low distraction.” Testing also showed that the gesture-recognition acknowledgement of air-traffic-control voice messages worked faultlessly when pilots were engaged in simulated traffic-collision avoidance scenarios. This highlights the potential of adopting such systems in future cockpits.
Collaboration for success
A tailored solution was developed thanks to a very productive collaboration. CSEM developed the vision system algorithms, while SERMA Ingénierie was responsible for adapting the system hardware and integrating it into a cockpit demonstrator. ETH Zurich performed the design and execution of the studies with pilots and Swiss International Airlines supported the consortium with expertise on the use cases. The system is not intended to radically redesign cockpits. However, the combination of gesture recognition and eye tracking should help pilots concentrate on their most important tasks, minimise distractions and improve flight safety.
Keywords
PEGGASUS, aircraft, cockpit, pilot, eye tracking, gesture recognition, vison system, machine learning, intelligent vision system
