Periodic Reporting for period 2 - HARVIS (HUMAN AIRCRAFT ROADMAP FOR VIRTUAL INTELLIGENT SYSTEM)
Periodo di rendicontazione: 2020-07-01 al 2021-12-31
Flight movements are growing significantly in Europe, with no trend reversal expected. The integration of unmanned aircraft into the air space will make traffic management even more complex. The HARVIS project aims to answer to this by developing a future concept of Artificial Intelligence in the cockpit.
A significant impact on pilots’ job is inevitable, with increasing information to deal with and new tasks to accomplish. Framing the human-machine interaction in terms of partnership will help building capacity in machines to better understand humans, and in people to engage collaboratively with them. In the cockpit, this partnership will lead to pilots using a set of new technologies, capable of self-learning, to anticipate needs and to adapt to pilots’ mental states.
SPO are being regarded as the next phase of a decades-long downward trend in the minimum number of cockpit crew and gaining relevance within the scientific community. Cognitive Computing algorithms and adaptive automation implemented in a digital assistant concept could support the single pilots’ performance and decision-making in complex situations.
HARVIS has developed a cockpit digital assistant concept able to partner and support the pilot, anticipating needs and make decisions in complex scenarios. The impact of the cockpit assistant concept has been assessed under the two following use cases: (1) Go around decision under unstable approach and (2) Diversion to alternate airfield after an emergency. Both demonstrators have been developed in the context of HARVIS (www.harvis-project.eu).
A significant impact on pilots’ job is inevitable, with increasing information to deal with and new tasks to accomplish. Framing the human-machine interaction in terms of partnership will help building capacity in machines to better understand humans, and in people to engage collaboratively with them. In the cockpit, this partnership will lead to pilots using a set of new technologies, capable of self-learning, to anticipate needs and to adapt to pilots’ mental states.
SPO are being regarded as the next phase of a decades-long downward trend in the minimum number of cockpit crew and gaining relevance within the scientific community. Cognitive Computing algorithms and adaptive automation implemented in a digital assistant concept could support the single pilots’ performance and decision-making in complex situations.
HARVIS has developed a cockpit digital assistant concept able to partner and support the pilot, anticipating needs and make decisions in complex scenarios. The impact of the cockpit assistant concept has been assessed under the two following use cases: (1) Go around decision under unstable approach and (2) Diversion to alternate airfield after an emergency. Both demonstrators have been developed in the context of HARVIS (www.harvis-project.eu).
WP1: State of the art Review and Case studies
The objective of this WP was to define the state of the art and past research done in cognitive computing, and to define what are the more relevant case studies to be covered by the POC. In D1.1 an assessment of the future aerospace sector has been done, considering future systems, tasks and human factors. It has been also useful to build specific UCs.
WP2: Human-Machine Partnership
The objective was to highlight the situations where a digital assistant would be relevant by analysing the work of pilots in the cockpit and studying the already existing virtual assistant concept. The two UC that were identified as relevant to be continued and tested during experimentation are:
- The Non-stabilized approach support: It will support in monitoring and go around decision making. For this assistant, our innovative approach is to use expertise from many pilots on many relevant scenarios to classify human judgement on real flight trajectories instead of relying on rules like SOP. Relevant segments of NSA based on FDR and operational constraints were presented on a web interface displaying parts of the flight deck for the pilots to give their judgement about the necessity to go-around or not. With this, an assistant was trained to recognize these situations and provide real time decision-making support. For the trajectory monitoring task usually done by the PM, we used an eye tracker to check if the single pilot is looking frequently at the deviating parameters to reduce the announcements to the minimum. As part of WP4 activities, we ran NSA simulations and compared the behaviour of a single pilot with a single pilot supported by our assistant.
- The Aircraft dynamic rerouting support: During regular operations if a diversion is needed, the pilot in command and first officer discuss on the multiple options they have and try to find out the one they think is the best. The AI assistant considers characteristics of nearby airports, METAR at destination, and facilities to take care of passengers, etc. It then calculates the trajectories, assess the risks and benefits of each one and finally presents the best rerouting alternatives to the pilot. Two technical challenges were contemplated: Firstly, the collection of a relevant and representative dataset to train the AI from real flights and secondly, to develop the proper HMI between the virtual assistant and the pilot for a satisfactory experience.
The last task of this WP, focused on the analysis of data necessary to make the solutions effective and how pilots should be trained in such a context to make the solution robust and effective (machine learning approach), the results are detailed in D2.3. Feedback gathered during the 3rd Advisory Board meeting was also considered.
WP3: Use Case Implementation
The objective of this WP is the development of the proof of concept of cognitive computing assistants described above.
For both assistants developed, feedback from pilots and from experts participating in the 2nd Advisory Board meeting was used to refine the implementations before the validation sessions. The prototypes developed are described in D3.2 and D3.1. The prototypes, their functionalities and characteristics and the algorithms used are described in these documents.
WP4: Validation and Roadmap
The objective of this WP was to run the simulation activities to validate the digital assistants’ behaviour while performing the use cases defined in WP3. We investigated the impact of the digital assistant on pilots and provide a multi-criteria analysis and a technology roadmap validation.
The work carried out in the validation activities, consisted in stablishing the validation objectives and associated success criteria and the plan for the validation sessions with pilots corresponding to the next task, including scenario definition, and methods and technics for the demonstration.
The aim of D 4.1 is to outline the experimental plan, validation methods and for the final project validation. The first part of the document outlines the project validation approach which is built upon three different levels: (1) the advisory board meetings and workshops with stakeholders, (2) the intermediate validation and (3) the final demonstration. The validation objectives (operational impact, impact on human performance, technical feasibility, and impact on safety) are also described along with associated success criteria, demonstration platform and methods and techniques.
We gathered a group of 8 expert pilots, which tested both assistants and provided feedback during the validation sessions. The results of the validation sessions (data gathered and debriefing comments) were analysed afterwards, the information can be found in D4.2.
The last task consisted in producing a roadmap with the expected evolution of the technologies that are considered as key enablers for the digital assistant, using the conclusions after the analysis of the validation results. Feedback from experts collected during the 3rd Advisory Board meeting was also considered. The results were included in D4.3.
The objective of this WP was to define the state of the art and past research done in cognitive computing, and to define what are the more relevant case studies to be covered by the POC. In D1.1 an assessment of the future aerospace sector has been done, considering future systems, tasks and human factors. It has been also useful to build specific UCs.
WP2: Human-Machine Partnership
The objective was to highlight the situations where a digital assistant would be relevant by analysing the work of pilots in the cockpit and studying the already existing virtual assistant concept. The two UC that were identified as relevant to be continued and tested during experimentation are:
- The Non-stabilized approach support: It will support in monitoring and go around decision making. For this assistant, our innovative approach is to use expertise from many pilots on many relevant scenarios to classify human judgement on real flight trajectories instead of relying on rules like SOP. Relevant segments of NSA based on FDR and operational constraints were presented on a web interface displaying parts of the flight deck for the pilots to give their judgement about the necessity to go-around or not. With this, an assistant was trained to recognize these situations and provide real time decision-making support. For the trajectory monitoring task usually done by the PM, we used an eye tracker to check if the single pilot is looking frequently at the deviating parameters to reduce the announcements to the minimum. As part of WP4 activities, we ran NSA simulations and compared the behaviour of a single pilot with a single pilot supported by our assistant.
- The Aircraft dynamic rerouting support: During regular operations if a diversion is needed, the pilot in command and first officer discuss on the multiple options they have and try to find out the one they think is the best. The AI assistant considers characteristics of nearby airports, METAR at destination, and facilities to take care of passengers, etc. It then calculates the trajectories, assess the risks and benefits of each one and finally presents the best rerouting alternatives to the pilot. Two technical challenges were contemplated: Firstly, the collection of a relevant and representative dataset to train the AI from real flights and secondly, to develop the proper HMI between the virtual assistant and the pilot for a satisfactory experience.
The last task of this WP, focused on the analysis of data necessary to make the solutions effective and how pilots should be trained in such a context to make the solution robust and effective (machine learning approach), the results are detailed in D2.3. Feedback gathered during the 3rd Advisory Board meeting was also considered.
WP3: Use Case Implementation
The objective of this WP is the development of the proof of concept of cognitive computing assistants described above.
For both assistants developed, feedback from pilots and from experts participating in the 2nd Advisory Board meeting was used to refine the implementations before the validation sessions. The prototypes developed are described in D3.2 and D3.1. The prototypes, their functionalities and characteristics and the algorithms used are described in these documents.
WP4: Validation and Roadmap
The objective of this WP was to run the simulation activities to validate the digital assistants’ behaviour while performing the use cases defined in WP3. We investigated the impact of the digital assistant on pilots and provide a multi-criteria analysis and a technology roadmap validation.
The work carried out in the validation activities, consisted in stablishing the validation objectives and associated success criteria and the plan for the validation sessions with pilots corresponding to the next task, including scenario definition, and methods and technics for the demonstration.
The aim of D 4.1 is to outline the experimental plan, validation methods and for the final project validation. The first part of the document outlines the project validation approach which is built upon three different levels: (1) the advisory board meetings and workshops with stakeholders, (2) the intermediate validation and (3) the final demonstration. The validation objectives (operational impact, impact on human performance, technical feasibility, and impact on safety) are also described along with associated success criteria, demonstration platform and methods and techniques.
We gathered a group of 8 expert pilots, which tested both assistants and provided feedback during the validation sessions. The results of the validation sessions (data gathered and debriefing comments) were analysed afterwards, the information can be found in D4.2.
The last task consisted in producing a roadmap with the expected evolution of the technologies that are considered as key enablers for the digital assistant, using the conclusions after the analysis of the validation results. Feedback from experts collected during the 3rd Advisory Board meeting was also considered. The results were included in D4.3.
HARVIS project presents a roadmap for the future digital assistant in the cockpit in the framework of SPO For this, several technologies were evaluated in the cockpit by implementing the two use cases described above.
After this, a systematic analysis was carried out to assess the performance of the human-machine collaboration in the cockpit in high workload situations. Feedback from experts was also collected and included in the analysis. The results of this project will be made public in the project website, so any A/C related stakeholder can take the results and analysis of the project as a base step for future projects and developments.
After this, a systematic analysis was carried out to assess the performance of the human-machine collaboration in the cockpit in high workload situations. Feedback from experts was also collected and included in the analysis. The results of this project will be made public in the project website, so any A/C related stakeholder can take the results and analysis of the project as a base step for future projects and developments.