Skip to main content

Highly Automated Air Traffic Controller Workstations with Artificial Intelligence Integration

Periodic Reporting for period 1 - HAAWAII (Highly Automated Air Traffic Controller Workstations with Artificial Intelligence Integration)

Reporting period: 2020-06-01 to 2021-05-31

The Controller Working Position (CWP) is the centre of the human-machine interactions for Air Traffic Controllers (ATCos). The tools and platforms used by controllers shape and inform their way of operating, their mental models and ultimately their performance. This in turn promotes the safe and efficient management of air traffic. Voice communications between ATCos and pilots is still not digitalised and, therefore, not accessible for machine analysis.
To change the current status, the SESAR Master Plan vision includes increasing digitalisation and automation of Air Traffic Control (ATC). Digitization of ATCo-pilot communication by Automatic Speech Recognition (ASR) is a cornerstone. Data communication, i.e. Controller Pilot Data Link Communication (CPDLC) or text-based transmission of data and ASR are no competing, but complementary. ASR can e.g. replace ATCos mouse-based inputs into the assistant systems. Voice is the most efficient way of human-human communication. Why not benefitting from ASR in human-machine communication? Recently, solution PJ.16-04 of SESAR 2020 Wave-1 has demonstrated that the current ASR systems are mainly targeting the every-day consumer market and are not suited to safety and time critical application areas such as Air Traffic Management (ATM). As shown in the past, trust and acceptance by ATCos of low fidelity tools is an obstacle for deployment.
Nevertheless, several past projects have indicated the usefulness and utility of ASR: (i) AcListant® quantified the benefits with respect to workload reduction and performance increase, (ii) MALORCA project demonstrated commercial viability and enhanced models, (iii) SESAR 2020 solution PJ.16-04 explored industrial integration and requirements building.
So far, these precursor projects have focused on research and development of a set of models, further deployed in relatively simple and controlled application domains. The HAAWAII project proposes to target complex and challenging environments and, more importantly, wider applications of automatically recognized voice communications. More specifically, HAAWAII proposes two general objectives:
1. Research and develop data-driven (machine learning oriented) approaches to be deployed for novel and complex environments from two large ANSPs, demonstrating an increased validity of the tools;
2. Demonstrate the wider applicability of the tools in ATM, focussing on generating benefits for the ATCos and the ANSPs, i.e. reducing workload, increasing both efficiency and safety.
Overall, HAAWAII intends to focus on the following applications:
• Pilot readback error detection,
• Modelling and being able to anticipate controller behaviour,
• Pre-filling radar labels and CPDLC messaging using the automatic speech recognition,
• Human performance metric extraction.
• NATS and Isavia ANS have provided each two months of voice and corresponding surveillance recordings. This includes manual transcription and manual checking of more than seven hours of silence reduced voice data for each partner, i.e. more than 4000 utterances are transcribed by each of the two ANSPs.
• NATS Human Performance has drafted the Human Performance Metrics Evaluation concept in collaboration with all partners. First progress has been made in understanding and interpreting the HP aspects of the ASR output to date.
• The project phase is extended by three months, the final report is published three months later
• All 15 project deliverables scheduled for the first reporting period are submitted to SJU.
• A command recognition rate of 84.5% for ATCOs and 78.8% for pilots is already achieved. The already achieved command recognition error of 10.1% for ATCos, 13.3% for pilots need to be reduced
• A callsign recognition rate of 91.2% for ATCos and 92.1% for pilots is already achieved for London TMA.
In concrete HAAWAII objectives are:
• Demonstration airspaces are London TMA of NATS and enroute and oceanic traffic controlled by Isavia ANS. MALORCA has chosen the medium size airports of Prague and Vienna.
• Running in the ops room environment, i.e. the validation data is not recorded in lab environment, but directly in the ops room environment of NATS and Isavia ANS.
• Exploiting massive amounts of unlabelled voice data through new unsupervised learning algorithms to training an Assistant Based Speech Recognition (ABSR) system. Due to COVID-19 situation at least 1000 hours per demonstrations airspace. MALORCA only uses 150 hours and SESAR 2020 solution PJ.16-04 does not use training from untranscribed models at all.
• Automatic recognition of the controller and pilot communication for London TMA and Isavia enroute airspace by targeting a recognition rate on command level of greater 85% for ATCos and 75% for pilots, an error rate on command level of less than 3% for ATCos and less than 5% for pilots and a callsign recognition rate of better than 95% for ATCos and 90% for pilots. Previous projects did not target the pilot side at all.
• Automatic readback error detection by targeting that at least 50% of the readback errors are detected and the false alarm rate is below 10%. Previous projects did not target to recognize pilots at all. It is for the first time, that read back error detection by ASR is performed in Europe at all.
• Proof-of-concept for Pre-filling of radar labels and CPDLC messaging via non-integrated applications. A command recognition rate of better than 90% is targeted, although SESAR 2020 solution PJ.16-04 achieved only 70% for Prague and 55% for Vienna in the lab environment.
• Improve ATCo staffing, rostering and flow management planning and reaction for the London TMA by measuring and anticipating the workload from voice communication. A user acceptance of better than 75% is targeted and the perceived workload reduction when predicting task load, also used as indicator of objective metrics validity should be confirmed by at least 80% of the study participants.
Last, but not least, data privacy issues should be sufficiently considered, i.e. a minimum amount of anonymized data is stored.