Periodic Reporting for period 2 - Co2Team (Cognitive Collaboration for Teaming)
Reporting period: 2020-07-01 to 2022-03-31
The growth in air traffic and the complexity of the systems in the cockpit, demand more and more skills from pilots and a higher workload. Co2Team proposes to help pilots in their tasks without changing the captain's task as the sole master on board of the aircraft and the decisions made using the power of the Artificial Intelligence to assist them. It is a real challenge in terms of human factors (taking into account the human) for the pilot and the AI to collaborate efficiently and as naturally as possible. The Co2Team project aims to propose to major European airframers and avionics systems providers, a roadmap on how Cognitive Technology and Artificial Intelligence could help pilots in this context.
Co2Team aims to create an artificial intelligence teammate to collaborate with the pilot built on top of the state-of-the art of human factors and a pilot-centred approach. This AI based on Cognitive Computing Technologies (CCT) allows a natural communication with the pilot. In other words, the pilot will be able to communicate with the intelligent teammate as he/she will do with a human teammate, by voice, gestures, gaze. An innovative bidirectional and multimodal communication solution will be tested. The pilot remains the captain on board and pilot-in-command, the CCT provides information to the pilot for better data driven decision making. This intelligent teammate will be able to consider the pilot and the context to best adapt information, communication and collaboration.
Co2team approach is based on 5 objectives:
• Identify the technical and methodological potential of cognitive computing in the different areas of application to cockpit operations
• Push up the role of the pilot in a collaborative relationship with very complex systems, with a Human-centred approach, the « Man in the Loop » concept for an optimized allocation of tasks and roles
• Construct an inventory of cognitive technologies and human to identify the potential of cognitive computing in cockpit operations
• Demonstrate the added value in flight safety and pilots’ acceptability of a cognitive collaborative teaming
• Develop a roadmap for an optimized and accepted collaboration pilot/system supported by cognitive computing technologies
Co2Team consortium is composed of five main partners. Each one of them contributes to the project with unique expertise: experts in human factors in aeronautics and cognitive technologies (Bordeaux INP-CATIE), experts in Artificial Intelligence (DFKI) and expert pilots (ECAIR,AKIANI). The project started on January 2019 and it will finish on December 2021. More information and news available at https://www.catie.fr/en/co2team-en/.
Co2Team aims to create an artificial intelligence teammate to collaborate with the pilot built on top of the state-of-the art of human factors and a pilot-centred approach. This AI based on Cognitive Computing Technologies (CCT) allows a natural communication with the pilot. In other words, the pilot will be able to communicate with the intelligent teammate as he/she will do with a human teammate, by voice, gestures, gaze. An innovative bidirectional and multimodal communication solution will be tested. The pilot remains the captain on board and pilot-in-command, the CCT provides information to the pilot for better data driven decision making. This intelligent teammate will be able to consider the pilot and the context to best adapt information, communication and collaboration.
Co2team approach is based on 5 objectives:
• Identify the technical and methodological potential of cognitive computing in the different areas of application to cockpit operations
• Push up the role of the pilot in a collaborative relationship with very complex systems, with a Human-centred approach, the « Man in the Loop » concept for an optimized allocation of tasks and roles
• Construct an inventory of cognitive technologies and human to identify the potential of cognitive computing in cockpit operations
• Demonstrate the added value in flight safety and pilots’ acceptability of a cognitive collaborative teaming
• Develop a roadmap for an optimized and accepted collaboration pilot/system supported by cognitive computing technologies
Co2Team consortium is composed of five main partners. Each one of them contributes to the project with unique expertise: experts in human factors in aeronautics and cognitive technologies (Bordeaux INP-CATIE), experts in Artificial Intelligence (DFKI) and expert pilots (ECAIR,AKIANI). The project started on January 2019 and it will finish on December 2021. More information and news available at https://www.catie.fr/en/co2team-en/.
During the first eight months, the consortium completed the first phase of the project, providing state of the arts about cognitive computing technologies and learning from rare events and states.
As a first step, the Co2Team consortium produced a state of the art (SoA) of cognitive computing and its use in different application domains and described the difference with a normal AI. Both use the same machine learning or deep learning technologies and can solve very complex problems in a short period of time, but their approach and objective are very different. In addition to focusing on natural communication, CCT supports human decision making while AI aims to solve problems without human intervention. Thus, the pilot's skills are increased rather than adding more AI-based automation. The advantage of CCT is that it can process a large amount of data, monitor and cross-reference it, to provide the pilot with reliable and understandable information. To learn, these technologies need a large amount of data, but the intelligent teammate must be able to help the pilot in abnormal situations and therefore learn with little data. A deliverable is therefore dedicated to the learning of rare events and intelligibility of Machine Learning.
The integration of an intelligent teammate in a cockpit requires a thorough analysis of the human factors involved. The Co2team consortium worked for eight months on Human Factors (HF) involved in a Human intelligent Machine Team, in parallel with the SoA on cognitive computing. We looked at past and present research and attempts in different fields of application. The knowledge acquired, with SoA in parallel, on CCTs has made it possible to extend research and to move away from the state of the art on human factors related to system automation. We provided a state of the art of the human factors involved in a hybrid team (Human-Cognitive Computing Teammate) and how to form this team that communicates, shares knowledge, information, collaborates and trusts each other to ensure flight safety at the highest level. As we had the objective of flight safety, we reviewed human factors recommendations and obligations and how they could be updated and implemented for SPO with a CCTeammate. In addition we interviewed pilots to make an in-depth analysis of all the tasks and knowledge required by pilots to carry out their duties and get their views on the implementation and interactions with the CCT and their ideas and recommendations for better acceptability.
After submitting those four deliverables, Co2Team consortium completed the second milestone of the project and launched a review of Cognitive technologies for bidirectional Human-System collaboration and the definition of the use case.
Co2Team consortium identified a specific use case and a very complex and risky flight scenario designed with pilots and human factors experts to challenge the HiMT. The scenario includes several flight phases (descent, approach, Go Around, and landing) the most accident prone according to Airbus and Boeing figures. We submitted and discussed the use case scenario with Airbus, Dassault and Thales who found it full of interest, useful and in relation to their expectations.
As a first step, the Co2Team consortium produced a state of the art (SoA) of cognitive computing and its use in different application domains and described the difference with a normal AI. Both use the same machine learning or deep learning technologies and can solve very complex problems in a short period of time, but their approach and objective are very different. In addition to focusing on natural communication, CCT supports human decision making while AI aims to solve problems without human intervention. Thus, the pilot's skills are increased rather than adding more AI-based automation. The advantage of CCT is that it can process a large amount of data, monitor and cross-reference it, to provide the pilot with reliable and understandable information. To learn, these technologies need a large amount of data, but the intelligent teammate must be able to help the pilot in abnormal situations and therefore learn with little data. A deliverable is therefore dedicated to the learning of rare events and intelligibility of Machine Learning.
The integration of an intelligent teammate in a cockpit requires a thorough analysis of the human factors involved. The Co2team consortium worked for eight months on Human Factors (HF) involved in a Human intelligent Machine Team, in parallel with the SoA on cognitive computing. We looked at past and present research and attempts in different fields of application. The knowledge acquired, with SoA in parallel, on CCTs has made it possible to extend research and to move away from the state of the art on human factors related to system automation. We provided a state of the art of the human factors involved in a hybrid team (Human-Cognitive Computing Teammate) and how to form this team that communicates, shares knowledge, information, collaborates and trusts each other to ensure flight safety at the highest level. As we had the objective of flight safety, we reviewed human factors recommendations and obligations and how they could be updated and implemented for SPO with a CCTeammate. In addition we interviewed pilots to make an in-depth analysis of all the tasks and knowledge required by pilots to carry out their duties and get their views on the implementation and interactions with the CCT and their ideas and recommendations for better acceptability.
After submitting those four deliverables, Co2Team consortium completed the second milestone of the project and launched a review of Cognitive technologies for bidirectional Human-System collaboration and the definition of the use case.
Co2Team consortium identified a specific use case and a very complex and risky flight scenario designed with pilots and human factors experts to challenge the HiMT. The scenario includes several flight phases (descent, approach, Go Around, and landing) the most accident prone according to Airbus and Boeing figures. We submitted and discussed the use case scenario with Airbus, Dassault and Thales who found it full of interest, useful and in relation to their expectations.
Co2Team contributes directly to the long-term aviation vision published by the EC, in which Clean Sky aims to strengthen the collaboration, global leadership and competitiveness of the European aviation industry. The CS2 Cognitive Computing topic is a significant opportunity to understand how Cognitive Computing could influence the way to consider Human Machine interaction in the future.
Clean Sky 2 Cognitive Computing topic is specifically designed to contribute to the following expected impacts:
• Intuitive and multimodal pilot interfaces
• Increased and more efficient communication with the ground
• Aircraft status, pilot behaviour monitoring
• Systems management improvement
• Reduced pilot workload
• Augmented vision systems with head-up/head Down displays
Co2Team’s objective of contributing to these expected impacts is specifically addressed in the four technical work packages, completed and in progress: WP1, State of the art about cognitive technologies, WP2 State of the art about man-machine teaming, WP3 Cognitive technologies for bidirectional Human-System collaboration, WP4 Case studies and demonstration in a flight simulator.
Clean Sky 2 Cognitive Computing topic is specifically designed to contribute to the following expected impacts:
• Intuitive and multimodal pilot interfaces
• Increased and more efficient communication with the ground
• Aircraft status, pilot behaviour monitoring
• Systems management improvement
• Reduced pilot workload
• Augmented vision systems with head-up/head Down displays
Co2Team’s objective of contributing to these expected impacts is specifically addressed in the four technical work packages, completed and in progress: WP1, State of the art about cognitive technologies, WP2 State of the art about man-machine teaming, WP3 Cognitive technologies for bidirectional Human-System collaboration, WP4 Case studies and demonstration in a flight simulator.