Deciphering Intents of Air traffic controllers, workLOad assessment and Gaze analysis to enable their efficient and trustworthy collaboration with AI

Project Information

DIALOG

Grant agreement ID: 101166886

DOI

10.3030/101166886

EC signature date 28 June 2024

Start date 1 September 2024

End date 28 February 2027

Funded under

• Climate, Energy and Mobility

Total cost

€ 1 974 387,50

EU contribution

€ 1 974 387,50

Coordinated by

SINTEF AS
Norway

Periodic Reporting for period 1 - DIALOG (Deciphering Intents of Air traffic controllers, workLOad assessment and Gaze analysis to enable their efficient and trustworthy collaboration with AI)

Reporting period: 2024-09-01 to 2025-08-31

Summary of the context and overall objectives of the project

The overall objective of the DIALOG project is to enable efficient and trustworthy collaboration between air traffic controllers (ATCOs) and artificial intelligence systems. To achieve this, the project focuses on three main goals: first, to infer ATCOs’ intent and goals by applying speech recognition and understanding of pilot-controller exchanges; second, to develop unobtrusive, real-time methods for assessing ATCOs’ workload and attention using multimodal data such as voice, physiological signals, and behavioral indicators; and third, to design a digital Teamwork Assistant that dynamically allocates tasks between human and AI agents based on context, workload, and intent.

The pathway to impact builds on these objectives by delivering scientific, technological, and societal benefits. Scientifically, DIALOG advances research in AI, human factors, and neuroscience through new models for intent inference, workload assessment, and human–AI teaming principles. Technologically, the project will validate a Teamwork Assistant integrated with ATCO workstations at TRL2, improving operational efficiency, reducing workload, and increasing airspace capacity. Economically, these innovations promise cost reductions for air traffic management and airlines through more optimal trajectories. Societally, the project promotes human-centric AI design to foster trust and usability, supporting safe adoption of AI in air traffic management. In addition, by enabling more efficient flight paths, DIALOG contributes to reducing aviation’s environmental impact and mitigating climate change.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

During the first reporting period, the project focused on building the technical and scientific foundations for human–AI collaboration in air traffic management. The operational concept and use cases were refined and validated through expert input and advisory board reviews. A key achievement was the development of an ontology for pilot requests and the creation of task models representing air traffic controller responses. These models were validated by operational experts and translated into workflows for software implementation. Based on this work, a first prototype of the Intent Inferring Service was delivered, capable of recognizing pilot requests and predicting controller actions.

Significant progress was also made toward unobtrusive, real-time assessment of workload and attention. Multimodal data collection sessions have been conducted to capture synchronized streams of speech, physiological signals, and gaze data from licensed controllers. Machine learning models were trained on these datasets to estimate workload levels, with promising results in subject-dependent scenarios. Additional experiments explored multimodal fusion and attention estimation, laying the groundwork for a service that provides real-time awareness of controller state.

In parallel, the design of effective human–AI teaming solutions advanced. Task allocation schemes were defined and validated through theoretical frameworks and operational feedback. The development of a digital assistant and a controller working position interface is progressing as planned. Guidelines and design patterns for human–AI interaction were drafted to ensure usability and trust.

Finally, the validation framework was established, including scenarios, KPIs, and multimodal measurement protocols for upcoming human-in-the-loop evaluations. Integration planning for the core services into a simulation environment was initiated, ensuring readiness for technology readiness level 2 validation.

Overall, the project has delivered validated models, initial prototypes, and methodological frameworks that form a strong basis for integrated demonstrations and impact assessment in the next phase.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The potential impacts of these results span scientific, technological, and societal domains. Scientifically, the project contributes new knowledge in AI, human factors, and neuroscience, particularly in intent inference and multimodal workload modeling. Technologically, the validated Teamwork Assistant and associated services promise to improve operational efficiency, reduce controller workload, and increase airspace capacity. Economically, these innovations could lower costs for air navigation service providers and airlines through optimized trajectories. Societally, the human-centric design approach fosters trust and usability, supporting safe adoption of AI in critical operations and enabling more efficient flight paths that reduce environmental impact.

To ensure further uptake and success, several key needs have been identified. Continued research is required to enhance model robustness and generalization, particularly for workload and attention estimation across diverse operational contexts. Large-scale demonstrations and human-in-the-loop validations will be essential to build confidence among stakeholders and meet SESAR maturity criteria. Access to markets and finance will support the transition from research to deployment, while clear strategies for commercialization and intellectual property management will enable sustainable exploitation. International collaboration and alignment with regulatory and standardization frameworks will be critical to ensure interoperability and compliance in global air traffic management. Finally, engagement with end-users and industry stakeholders must remain central to guarantee operational relevance and accelerate adoption.