Periodic Reporting for period 1 - HARVIS (HUMAN AIRCRAFT ROADMAP FOR VIRTUAL INTELLIGENT SYSTEM)
Reporting period: 2019-01-01 to 2020-06-30
Flight movements are growing significantly in Europe, with no trend reversal expected. The integration of unmanned aircraft into the air space will make traffic management even more complex.
A significant impact on pilots’ job is inevitable, with increasing information to deal with and new tasks to accomplish. Framing the human-machine interaction in terms of partnership will help building capacity in machines to better understand humans, and in people to engage collaboratively with them. In the cockpit, this partnership will lead to pilots using a set of new technologies, capable of self-learning, to anticipate needs and to adapt to pilots’ mental states.
Single Pilot Operations (SPO) is being regarded as the next phase of a decades-long downward trend in the minimum number of cockpit crew and gaining relevance within the scientific community. A significant impact on pilots’ job is inevitable consequence, with increasing information to deal with and new tasks to accomplish. Cognitive Computing algorithms and adaptive automation implemented in a digital assistant concept could support the single pilots’ performance and decision-making in complex situations.
HARVIS project is developing a cockpit digital assistant concept able to partner and support the pilot to support anticipating needs and make decisions in complex scenarios. The impact of the cockpit assistant concept will be assessed under the two following use cases: (1) Go around decision under unstable approach and (2) Diversion to alternate airfield after an emergency. Both demonstrators are being developed in the context of Project HARVIS (www.harvis-project.eu).
The objective of this WP is to define the state of the art and past research done in cognitive computing, and to define what are the more relevant case studies to be covered by the proof of concept. In the deliverable 1.1 an assessment of the future aerospace sector has been carried out, considering future systems and tasks as well as human factor. Additionally, a state-of-the-art review in machine learning and cognitive computing algorithms and their integration into several sectors such as aerospace, automotive, healthcare, etc. has been made. Furthermore, a research of the different proposal for the implementation of the Single Pilot Operation (SPO) scenario has been done, analysing the different approaches and current regulations.
This deliverable defined the context in which the HARVIS roadmap is being developed, specifying the steps needed, in terms of technology development, interaction design and training, to develop such an assistant.
It has been also useful to build specific use cases, to make specific examples of an AI based virtual assistant able to enable and support Single Pilot Operations.
WP2: Human-Machine Partnership
Analysis of Potential Cognitive computing aided Tasks: the objective was to highlight the situations where a digital assistant would be relevant by analyzing the work of pilots in the cockpit and studying the already existing virtual assistant concept. This document was part of the UC selection process that led to the two UC that were identified as relevant to be continued and tested during experimentation. These Uses Cases are
• The Non-stabilized approach support: During the high workload phases of the flight such as the approach, the PF has the support of the PM for the trajectory monitoring and the Go Around decision. Nowadays, 97% of Non-Stabilized Approach (NSA) are continued until landing going against Standard Operational Procedures (SOP), therefore, an assistant solely based on SOP will potentially lead to conflicts with PF decisions. Moreover, the PM is aware of what the PF is doing and can decide if a trajectory deviation announce is relevant or not according to the situation. Our assistant will provide support for monitoring and go around decision making. For the Go Around decision making support assistant, our innovative approach is to use expertise from a large number of pilots on many relevant scenarios to classify human judgement on real flight trajectories instead of relying on rules like SOP. Relevant segments of NSA based on Flight Data Recordings (FDR) and operational constraints will be presented on a web interface displaying parts of the flight deck for the pilots to give their judgement about the necessity to go-around or not. Thanks to this labelling, an assistant will be trained to recognize these situations and will provide real time support to decision-making. For the trajectory monitoring task usually done by the PM, we intend to use an eye tracker in order to check if the single pilot is looking frequently at the deviating parameters in order to reduce the announcements to the minimum. We intend to run non-stabilized approach simulation and compare the behavior of a single pilot with a single pilot supported by our assistant.
• The Aircraft dynamic rerouting support: Diversion is often required after system failure, medical emergency, or just for weather phenomena (dense fog, storms, etc.) in the approaching. During regular operation if a diversion is needed the pilot in command and first officer discuss on the multiple options they have and try to find out the one they think is the best. The AI assistant will take into account characteristics of nearby airports, METAR at destination, and facilities to take care of passengers, among other factors. It may then consider several options, assess the risks and benefits of each one, and finally inform the pilot accordingly. In this scenario, the digital assistant takes care of the Options and Risks in a FORDEC procedure. HARVIS is committed to develop a use case demonstrator for the arrival rerouting. Two technical challenges are contemplated: Firstly, the collection of a relevant and representative dataset to train the AI from real flights and secondly, to develop the proper human-machine interface between the virtual assistant and the pilot for a satisfactory experience.
After the two use cases are implemented and validated by mid-2021, a systematic analysis will be carried to asses the performance of the human-machine collaboration in the cockpit in high workload situations. Feedback from several roles will be also collected and included in the analysis. At the end of the project, the amin outcomes will be collected and summarized in two public deliverables:
• D4.2 Analysis Report: This deliverable will include the multi-criteria analysis performed on the basis of the simulations.
• D4.3 Technologies Roadmap: This deliverable will present the roadmap of technologies for the next generation digital assistant.
Th results of this project will be made public, so any A/C related stakeholder can take the results and analysis of the project as a base step for future projects and developments.