Periodic Reporting for period 2 - E-PILOTS (Evolution of cockPIt operations Levering on cOgnitive compuTing Services)
Période du rapport: 2020-07-01 au 2021-12-31
E-PILOTS aims at contributing to enhancement of the safety and overall performance of flight deck operations by elaborating a roadmap of new cockpit supporting tools relying on present cognitive computing technologies paving the way for a safe transition to the “Single Pilot Operation” (SPO) framework, guaranteeing that only 1 cabin pilot would be able to manage the different tasks avoiding the negative impact of interruptions in complex scenarios which could lead to time-out of some pending tasks and impacting on the safety indicators.
E-PILOTS addresses the following key objectives:
• To setup a modelling framework to synthesize the fascination with cognitive computing in some areas with its inertial proclivity towards automation and potential displacement of some pilot tasks considering rigorous analysis of cockpit operations and cognitive gaps in unexpected and complex situations.
• Identify functional and interface requirements to monitor the cognitive state of the pilot and predict its dynamic evolution to better characterize and detail the scope of cockpit pilot supporting skill-based, rule-based and knowledge-based tasks considering expected future events.
• Extend the "human-in-the-loop" behavioural analysis of the “human-in-the-mesh” behaviour given the importance of the prospective memory items caused by the frequent interruptions affecting the PF performance.
• Context-aware mechanism to enhance pilot situational awareness considering a shared situational awareness framework among the different actors to better support knowledge-based pilot tasks.
• To design a service oriented architecture to coordinate and enable efficient and effective supporting tools in different operational conditions maintaining the performance variability under a performance envelope.
• Implementation of a cognitive infrastructure to support the benefits of a symbiosis during the decision making process when unexpected or very complex situations arise, in which tacit knowledge will be explicit shared with the pilot flying. Cognitive infrastructure implementation will be guided by what the pilot needs to know, when the information should be shared and how pilot tasks could be supported.
• Development of a roadmap from cognitive computing technological and flight deck operational points of view. Lessons learnt during validation exercises will drive the selection of technologies to accomplish a given cockpit operational development level.
E-PILOTS addresses the following key objectives:
• To setup a modelling framework to synthesize the fascination with cognitive computing in some areas with its inertial proclivity towards automation and potential displacement of some pilot tasks considering rigorous analysis of cockpit operations and cognitive gaps in unexpected and complex situations.
• Identify functional and interface requirements to monitor the cognitive state of the pilot and predict its dynamic evolution to better characterize and detail the scope of cockpit pilot supporting skill-based, rule-based and knowledge-based tasks considering expected future events.
• Extend the "human-in-the-loop" behavioural analysis of the “human-in-the-mesh” behaviour given the importance of the prospective memory items caused by the frequent interruptions affecting the PF performance.
• Context-aware mechanism to enhance pilot situational awareness considering a shared situational awareness framework among the different actors to better support knowledge-based pilot tasks.
• To design a service oriented architecture to coordinate and enable efficient and effective supporting tools in different operational conditions maintaining the performance variability under a performance envelope.
• Implementation of a cognitive infrastructure to support the benefits of a symbiosis during the decision making process when unexpected or very complex situations arise, in which tacit knowledge will be explicit shared with the pilot flying. Cognitive infrastructure implementation will be guided by what the pilot needs to know, when the information should be shared and how pilot tasks could be supported.
• Development of a roadmap from cognitive computing technological and flight deck operational points of view. Lessons learnt during validation exercises will drive the selection of technologies to accomplish a given cockpit operational development level.
The following work has been performed:
WP1- “Cockpit Operations & Case Studies”: , the Operational Concept for the full project has been finished with a definition of the use case and the E-Pilots simulation framework functionalities. A deep state of the art review on flight deck Cognitive Computing supporting tools has been the baseline to identify present gaps and opportunities to improve PF performance while preserving safety in a SPO framework. A Roadmap explaining the need on further research for a safety deployment of cognitive computing supporting services in the flight deck has been documented.
WP2- “Cognitive Sensor Net”: different physiological sensors has been considered and tested to characterize the pilot cognitive state together with the algorithms to translate their output into a cognitive scale. An architecture integration of the sensor with the Rolls Royce Future Flight Simulator provided by Cranfield University for validation purposes has been described together with a set of experiments for validation purposes. The original architecture has been upgraded to perform the experiments in an A-320 cockpit simulator. In this WP, it has been implemented 3 different serious games in which the human operator creates Prospective Memory Items for data gathering purposes.
WP3, “Cognitive Computing”: The methods and development of the predictive algorithm to improve the pilot situational awareness by informing about the probability of a hard landing have been described, implemented and tested using data from Flight Monitoring System (FMS) of 377,446 flights. The implementation of a ML algorithm to monitor the cognitive status of the PF has been implemented using EEG and ECG data. The results achieved in WP3, confirms that ECG is more stable and results can be more easily transferred at the cost of a limited capability for discriminating among different cognitive states and detecting states not related to stress and heart rate (like mental collapse). Also, it has strong dependency on the baseline state of the subject which suggests that the best use of ECG could be the detection of relative mental alterations across the flight. On the contrary, EEG is stable to discriminate different mental states at the cost of a lower task transfer power and a demand of a larger data set for training models. Data has been collected from serious games exercises and A-320 Cockpit simulations.
WP4 - “Socio-Technological Simulation Model”: It has been analysed the Standard Operating Procedures (SOP) from the “Approach Briefing” to “Reaching Minima” to specify the different PF and PM tasks using the FRAM formalism. This model will be the baseline to identify the need of cognitive computing supporting tools avoiding prospective memory items which impact on the pilot performance. The causal model has been implemented in FRAM formalism and adapted to SPO by assuming PM uncapacitated. The models has been validated by means of different experiments in an A-320 cockpit simulator. The model provides excellent tool to predict the PF workload, and allows the design of mitigation mechanism to avoid the creation of prospective memory items when PF is attending concurrent actions.
WP5 - “Verification and Validation” : It has been analysed the functional and non-functional requirements of the simulation framework to support the use case exercises, in order to get a better understanding of the barriers and enablers of cognitive computing supporting tools in the knowledge-based pilot flying tasks. The validation exercises for the ML2 and the socio-technical model were implemented in an A-320 cockpit simulator. An experienced line pilot was monitored with the EEG and ECG devices when flying the different scenarios.
WP6 - “Dissemination, exploitation and IPR management”, a detailed dissemination plan has been defined, with the high-level communication objectives, the target stakeholders, messages to be delivered, channels to be used, and activities to be performed.Worthwhile to emntion the publication of 5 scientific papers in Q1/Q2 journals.
Overview of exploitable results:
1.- Socio-Technological Simulation Framework.
2.- Machine Learning to predict a hard landing.
3.- Machine learning to monitor Huamn operator Cognitive status.
WP1- “Cockpit Operations & Case Studies”: , the Operational Concept for the full project has been finished with a definition of the use case and the E-Pilots simulation framework functionalities. A deep state of the art review on flight deck Cognitive Computing supporting tools has been the baseline to identify present gaps and opportunities to improve PF performance while preserving safety in a SPO framework. A Roadmap explaining the need on further research for a safety deployment of cognitive computing supporting services in the flight deck has been documented.
WP2- “Cognitive Sensor Net”: different physiological sensors has been considered and tested to characterize the pilot cognitive state together with the algorithms to translate their output into a cognitive scale. An architecture integration of the sensor with the Rolls Royce Future Flight Simulator provided by Cranfield University for validation purposes has been described together with a set of experiments for validation purposes. The original architecture has been upgraded to perform the experiments in an A-320 cockpit simulator. In this WP, it has been implemented 3 different serious games in which the human operator creates Prospective Memory Items for data gathering purposes.
WP3, “Cognitive Computing”: The methods and development of the predictive algorithm to improve the pilot situational awareness by informing about the probability of a hard landing have been described, implemented and tested using data from Flight Monitoring System (FMS) of 377,446 flights. The implementation of a ML algorithm to monitor the cognitive status of the PF has been implemented using EEG and ECG data. The results achieved in WP3, confirms that ECG is more stable and results can be more easily transferred at the cost of a limited capability for discriminating among different cognitive states and detecting states not related to stress and heart rate (like mental collapse). Also, it has strong dependency on the baseline state of the subject which suggests that the best use of ECG could be the detection of relative mental alterations across the flight. On the contrary, EEG is stable to discriminate different mental states at the cost of a lower task transfer power and a demand of a larger data set for training models. Data has been collected from serious games exercises and A-320 Cockpit simulations.
WP4 - “Socio-Technological Simulation Model”: It has been analysed the Standard Operating Procedures (SOP) from the “Approach Briefing” to “Reaching Minima” to specify the different PF and PM tasks using the FRAM formalism. This model will be the baseline to identify the need of cognitive computing supporting tools avoiding prospective memory items which impact on the pilot performance. The causal model has been implemented in FRAM formalism and adapted to SPO by assuming PM uncapacitated. The models has been validated by means of different experiments in an A-320 cockpit simulator. The model provides excellent tool to predict the PF workload, and allows the design of mitigation mechanism to avoid the creation of prospective memory items when PF is attending concurrent actions.
WP5 - “Verification and Validation” : It has been analysed the functional and non-functional requirements of the simulation framework to support the use case exercises, in order to get a better understanding of the barriers and enablers of cognitive computing supporting tools in the knowledge-based pilot flying tasks. The validation exercises for the ML2 and the socio-technical model were implemented in an A-320 cockpit simulator. An experienced line pilot was monitored with the EEG and ECG devices when flying the different scenarios.
WP6 - “Dissemination, exploitation and IPR management”, a detailed dissemination plan has been defined, with the high-level communication objectives, the target stakeholders, messages to be delivered, channels to be used, and activities to be performed.Worthwhile to emntion the publication of 5 scientific papers in Q1/Q2 journals.
Overview of exploitable results:
1.- Socio-Technological Simulation Framework.
2.- Machine Learning to predict a hard landing.
3.- Machine learning to monitor Huamn operator Cognitive status.
It is expected that E-PILOTS operational framework supports:
• A test-bed to identify those combination of events that overload pilots capacity to manage and interleave the pending tasks on time, and which cognitive support could be provided.
• A simulation environment to validate the benefits of flight deck cognitive supporting tools by monitoring the cognitive state of the pilot
• A cost-effective framework to evaluate cockpit solutions to move towards Single Pilot Operation framework.
• Demonstration and quantification of the potential for cognitive computing supporting tools in the cockpit as well as provision of the benefits in safety and efficiency.
• A test-bed to identify those combination of events that overload pilots capacity to manage and interleave the pending tasks on time, and which cognitive support could be provided.
• A simulation environment to validate the benefits of flight deck cognitive supporting tools by monitoring the cognitive state of the pilot
• A cost-effective framework to evaluate cockpit solutions to move towards Single Pilot Operation framework.
• Demonstration and quantification of the potential for cognitive computing supporting tools in the cockpit as well as provision of the benefits in safety and efficiency.