Final Report Summary - MAN4GEN (Manual Operation for 4th Generation Airliners)
Executive Summary:
While aviation is an extremely safe mode of transport, there has recently been a steady rise in the number of accidents that are attributed to limited manual handling skills by pilots. These accidents are often due to a combination of the crew not managing the aircraft systems effectively after an unexpected event, and being unable to apply appropriate manual handling skills. Man4Gen aims to identify the common thread behind the events that lead to these accidents, and to recommend short-term changes to operational procedures, training and aircraft systems technology in order to mitigate this threat to aviation safety.
A procedure has been developed to assist crews in managing surprising, unexpected and ambiguous situations. Three philosophy concepts come back in this procedure: manage time criticality, manage (un)certainty and plan for contingencies and changes. The procedure is supplementing the manufacturer procedures and checklists, by providing a strategy for flight crew to deal with situations for which there is no procedure available. The training solution focuses on train competencies instead of pre-described flight manoeuvers. To support this, a scenario development method has been worked out to generate scenarios for competency-based training. The improved cockpit system development focuses on supporting the crew’s abilities to perform the modified procedure. It consists of additional pages of the Electronic Centralized Aircraft Monitor (ECAM) to summarize the current aircraft situation in terms of controllability, performance and endurance.The implemented solutions for procedures, training and cockpit displays, have been evaluated in experiments in either a research or a training simulator setting where the behaviour of the flight crew to the modifications was evaluated. In addition to the three planned experiments at the NLR, DLR and GTA simulators, an additional training experiment was carried out at the training centre of a major airline outside of the consortium.
The analyses for the three operational research strands were brought together with the results of the exploratory experiments for the final analysis. The findings of the operational work are that there are improvements that can be made to current practices that can address the potential for problems in modern airline operations. It has been identified that it is possible to support and prepare flight crew to deal with unexpected events, potentially including in the form of additional systems support in the longer term. The main outcomes from the two research strands, SA and sensemaking, were analysed. Next to conventional assessment techniques, methods were applied that are new and innovative, such as fMRI, (social) SA, and sensemaking. The combination of these techniques results in a mutual validation of the applied approaches. Therefore, we are now better able to test the interfacing between human and machine (HMI), the required levels of training, and associated factors in a dynamic test and simulation environment, which allows developing new HMI configurations.
Project Context and Objectives:
Modern highly automated aircraft (4th Generation airliners) are extremely safe, and there is a particularly low chance of an accident when operating these aircraft. Automation clearly plays a very positive role in enhancing aviation safety and preventing accidents. However, there has recently been a steady rise in the number of accidents that are attributed to the ability of flight crew to assess and understand an unexpected situation, and consequently respond appropriately to handle the situation, and eventually in some cases limited manual handling skills by pilots.
The main objectives therefore of the project were twofold:
- To identify the factors in highly automated, 4th generation, aircraft that affect the ability of the flight crew to handle unexpected events and gradually deteriorating conditions to maintain effective control of the aircraft.
- To subsequently identify methods to prepare flight crew to deal with unexpected events, modifying the training, procedures and systems available to operators of highly automated aircraft today.
The project was set up to achieve these objectives using a phased approach. The initial phase of the project focused on establishing the background of the problem from an academic and an operational perspective. This was then applied in an experimental setting to reproduce the challenges faced by flight crews in a way that their behaviour and responses can be studied. The results of the experiments formed the basis for an initial concept development to improve the flight crew’s ability to handle challenging situations, and subsequently focus on the development of aviation industry guidelines. The guidelines were applied and evaluated in an operational/experimental setting to validate their application.
In the first period of the Man4Gen project a literature study was performed and this was used to identify the roadmap for the project. Research methods to investigate how crew and aircraft successfully handle unexpected events from a Cognitive Systems Engineering (CSE), or sensemaking, perspective have been developed. Exploratory experiments in research flight simulators have been executed to identify the processes that lead to confusion or loss of SA when using automation. In addition to the flight simulator experiments, experiments have been carried out in an fMRI environment to establish the brain regions that exhibit significant activity in response to situational awareness changes. These experiments are intended to demonstrate a link between known brain activity and situational awareness, to be able to examine situational awareness further from a theoretical and neuropsychological perspective in the forthcoming analysis work.
In the second phase of the project the results of the exploratory experiments have been analysed focusing on three research tracks: operational analysis, sensemaking and Situational Awareness (SA). The sensemaking and SA analysis highlights a number of issues and gaps in communication, information and uncertainty management and prioritising implying the need to support the crew in these areas. The operational analysis identifies competencies where high-performing crews are highly-rated. With these results concepts of solutions for the above mentioned problems have been identified in three areas: procedures, training and cockpit design. A selection has been made of these solutions for further development in the Man4Gen project.
This final report addresses the results of the Man4Gen project.
a. Overview of the results
b. Conclusions on the project
c. Exploitation and dissemination
Project Results:
2 FINAL TECHNICAL REPORT
2.1. Overview of the results
In the first period of the Man4Gen project a literature study was performed and this was used to identify the roadmap for the project. From this literature study the following problem statement for the project has been derived: Despite the substantial and proven safety benefits of automation systems in 3rd and 4th generation aircraft, evidence indicates that when faced with unexpected and challenging situations, pilots sometimes have difficulties in quickly responding to situations which require a rapid transition in their activity from monitors of very reliable systems, to active and authoritative decision-makers exercising manual control of the aircraft.
Research methods to investigate how crew and aircraft successfully handle unexpected events from a Cognitive Systems Engineering (CSE), or sensemaking, perspective have been developed.
Exploratory experiments in research flight simulators have been executed to identify the processes that lead to confusion or loss of SA when using automation.
In addition to the flight simulator experiments, experiments have been carried out in an fMRI environment to establish the brain regions that exhibit significant activity in response to situational awareness changes. These experiments are intended to demonstrate a link between known brain activity and situational awareness, to be able to examine situational awareness further from a theoretical and neuropsychological perspective in the forthcoming analysis work.
In the second phase of the project the results of the exploratory experiments have been analysed focusing on three research tracks: operational analysis, sensemaking and Situational Awareness (SA). The sensemaking and SA analysis highlights a number of issues and gaps in communication, information and uncertainty management and prioritising implying the need to support the crew in these areas. The operational analysis identifies competencies where high-performing crews are highly-rated. With these results concepts of solutions for the above mentioned problems have been identified in three areas: procedures, training and cockpit design. A selection has been made of these solutions for further development in the Man4Gen project.
A procedure has been developed to assist crews in managing surprising, unexpected and ambiguous situations. Three philosophy concepts come back in this procedure: manage time criticality, manage (un)certainty and plan for contingencies and changes. The procedure is supplementing the manufacturer procedures and checklists, by providing a strategy for flight crew to deal with situations for which there is no procedure available. The training solution focuses on train competencies instead of pre-described flight manoeuvers. To support this, a scenario development method has been worked out to generate scenarios for competency-based training. The improved cockpit system development focuses on supporting the crew’s abilities to perform the modified procedure. It consists of additional pages of the Electronic Centralized Aircraft Monitor (ECAM) to summarize the current aircraft situation in terms of controllability, performance and endurance.
The implemented solutions for procedures, training and cockpit displays, have been evaluated in experiments in either a research or a training simulator setting where the behaviour of the flight crew to the modifications was evaluated. In addition to the three planned experiments at the NLR, DLR and GTA simulators, an additional training experiment was carried out at the training centre of a major airline outside of the consortium. The analyses for the three operational research strands were brought together with the results of the exploratory experiments for the final analysis. The findings of the operational work are that there are improvements that can be made to current practices that can address the potential for problems in modern airline operations. It has been identified that it is possible to support and prepare flight crew to deal with unexpected events, potentially including in the form of additional systems support in the longer term.
The main outcomes from the two research strands, SA and sensemaking, were analysed. Next to conventional assessment techniques, methods were applied that are new and innovative, such as fMRI, (social) SA, and sensemaking. The combination of these techniques results in a mutual validation of the applied approaches. Therefore, we are now better able to test the interfacing between human and machine (HMI), the required levels of training, and associated factors in a dynamic test and simulation environment, which allows developing new HMI configurations.
The following sections describe in more detail the stages that were carried out in the project:
• Exploratory Experiments
• Development of concepts to improve safety in the operation
• The Man4Gen recommendations
• The conclusions from the final validation experiments
3 ANALYSIS OF THE EXPLORATORY EXPERIMENTS
At the end of the work package 2 exploratory experiments an initial analysis was carried out to collect and report the measures that were described in the experiment plan. This initial breakdown was reported at the end of the exploratory experiments and formed the starting point for the analysis work for the recommendations. The analysis was carried out by the project partners applying three research foci: Operational analysis (Airbus, Boeing, NLR), Sensemaking (Linköping University, NLR), Situational Awareness (Vienna University, DLR).
3.1. Initial Findings SA Analysis
Crew analyses revealed that the following gaps in SA occurred: lack of weather awareness during ILS (i.e. wind), lack of system awareness after the bird strike (i.e. engine status), lack of correct application of procedures and checklists after the bird strike, lack of energy awareness after the bird strike, and lack of flexibility in decision making.
RQ: Why did those gaps in SA occur and why did crews perform better or worse?
(1) In our initial analysis we evaluated the data from the various questionnaires (i.e. NASA TLX raw, SART, SSA-VAS) and found a significant change in workload and SA over the scenario. We performed a correlation analysis and found a significant relationship between workload and SA. Workload and stress patterns increased during surprising and challenging flight phases and at the same time SA decreased. Thus, in our exploratory study an increase in workload and stress were linked with a decline in different types of SA.
(2) In the fMRI experiments we found evidence that pilots accompany monitoring, understanding, anticipation and decision making by mentally verbalizing or quietly murmuring. This highlights that communication is a very important factor for SA as well as for all shared forms of SA. Additionally, this finding supports the hypothesis of “shared mechanism”, that language and sensorimotor cognition are linked. Furthermore, we found that communication (with self or others) directly is encoded with cognitive processes that are linked to the experience collected in the named context (i.e. memory). Consequently, less effective oral and non-verbal communication in the cockpit can be related to SA and a possible worse performance.
(3) Similar to the fMRI results, the role of effective communication also was an issue identified in the simulator studies, where crews were rated better if they successfully communicated. Consequently, communication seems to serve various purposes within a socio-technical system: (A) assisting first, second and third level SA (i.e. perception, understanding or making sense, anticipating), (B) supporting recall from memory, which is linked to point (A), and (C) sharing one’s mental model and consequently fostering shared and team SA. We observed different communication patterns in crews. Amongst other things, the linear model showed that crews were rated better if PM was able to recall information from memory, checklists or the system by himself and if he was more active in explaining. Framed in differently, crews were rated worse if PM requested more information and explained actions less during surprising and challenging situations, i.e. bird strike.
(4) PF behaviour that emerged as being relevant for better crew performance was an increased amount of explaining actions during go-around and during final approach and land. This implies that sharing one’s mental model has a positive effect on shared and team SA. Additionally, better crew performance was linked to the PF appropriately selecting the type and amount of communication. The PM in crews which received higher rating scores seemed to be better able and more ready to receive information. A lack in these abilities, i.e. the ability and amount of sharing mental models and verbalizing current and planned actions, had an impact on the different gaps in SA observed in our scenario.
(5) If PF used initiative and gave directions when required, then the crew received better ratings.
(6) Our assessment of non-verbal communication showed that crews were rated better if PF used greater amounts of non-verbal communication during final approach and land and PM used greater amounts during initial approach. Iconic gestures are known to accompany speech in several ways: (A) redundant (e.g. “to the right” and gesture pointing to the right), (B) supplementary (e.g. “this direction” and gesture depicting the path to the left), and (C) complementary (e.g. “this” and gesture pointing to an engine on the engine display). Consequently, non-verbal communication conveys information, but also catches attention and supports understanding. Speakers’ gestures affect listeners’ (re-)actions. Effective non-verbal communication by PF and PM during different stages in the scenario correlated with better crew ratings.
(7) We found evidence from the SSA evaluations that the PF’s and PM’s abilities to assess their colleague’s level of stress and control were diminished during challenging and stressful flight phases. Our results show that stress affects the perception and understanding of others, i.e. leads to decreased empathic accuracy. The latter again might lead to decreased abilities to understand the current emotional and cognitive state of the other crew members and therefore decrease SSA. Consequently, we propose continuing to focus on social SA topics in aviation.
3.2. Initial Findings Sensemaking Analysis
The analysis of debrief data highlighted a number of issues implying the need for support for crews to search for information, manage uncertainties, prioritise and make trade-offs between goals at various levels of granularity, assess risks, consider options and anticipate future events. Combined with the contextual control strategy analysis we can conclude that some crews do this well by aligning priorities and risk assessments through an interaction between discussion considering options, risks, and expectations, and action acting upon these assessments, followed by monitoring of results of these actions against high-level goals.
The extent and content of the assessments made by the individual crews varies considerably, from crews that agree they should have taken more time to assess the best runway once engines were stabilized, to crews who feel there was no other option but to land on the nearest runway. Improving the crew-automation system’s ability (e.g. through training, procedures, display design) to improve the understanding of technical issues and assessing their potential consequences therefore seems to be needed. Also, considerable differences were found between the two crew member’s assessment of the situation, what variables are considered, what risks, goals, and priorities are identified, and what actions should be prioritised. The overall ability to assess whether immediate action is needed, how much time is available to assess and problem-solve, based on an identification and appreciation of risks and goals, seems to be a subject of considerable variability and therefore deserves attention while this project generates its recommendations. Variations and misunderstandings in the use of autopilot and engine management imply that this ability to appreciate risks and time constraints and possibilities goes hand-in-hand with the technical understanding of the complex systems that are difficult to understand in pressed situations, making these joint crew-automation issues rather than issues of individual pilots.
3.3. Initial Findings Operational Analysis
The “Desired Flight Crew Performance” (DFCP) analysis displays the flight crew’s overall performance in the operational scenario. Results show that high-performing crews in this scenario were highly rated in Communication, Leadership and Teamwork, Problem Solving and Decision Making, Situation Awareness, and Workload Management. These competencies need to be paired together since some of them are a consequence of good performance in the others. For example, Communication by itself is not indicative of good performance since this competency is only a medium to propagate good behaviour in the other competencies identified here. In fact, as noticed with poor-performing crews, communication needs to be effective and clear to guarantee that the recipients understand and acknowledge what is being said. If that is not the case, it can lead to a performance decrease in the other core competencies (e.g. loss of Situation Awareness).
Reflecting on the results from this analysis, poor-performing crews showed difficulties in the competencies where high-performing crews were strong, especially during high-workload situations. These poor-performing crews completely skipped the planning flight phase which had a high impact during the execution flight phase, shown by the several below average and poor performance comments. Also, the heat-map shows that these crews already have difficulties in Application of Procedures during low-workload situations (flight phases 1 and 2) and in manual flight throughout the scenario. High-performing crews, on the other hand, do not show negative comments for these competencies during these flight phases, yet positive comments were not present since conducting the required procedures here is not considered as above average performance. Despite the predictive asymmetry preventing the prediction of positive performance, it can at least be premised that poor performance for the overall flight can be predicted from low workload situations. All in all the collection of observed competencies are able to draw a clear picture of the differences between high and poor performing crews.
This analysis has identified the competencies that are most helpful in managing unexpected and challenging events, in addition to those competencies whose absence is most likely to lead to poor performance and unsafe outcomes. The desirable competencies identified by the analysis of crew responses to this scenario are: Leadership & teamwork, communication and problem solving & decision making.
4 CONCEPT DEVELOPMENT
4.1. Solution recommendations for procedures
At this time, four concepts have been identified which may be worthwhile to investigate as (partial) solutions for the above problems. The solution to the first problem may consist of defining and implementing an explicit threat-assessment step into existing emergency procedures. A second solution aspect for the first problem may be to introduce a methodology to address and manage uncertainties concerning the aircraft and system states. The solution for the second problem may lie in assisting a process of review and rechecking, possibly by the use of mnemonics. The solution to the third problem may involve redesigning non-normal procedure checklists to include all information required to solve the problem, as opposed to referencing other documents, procedures or people.
4.1.1. Threat assessment solution path
Experiment results show that crews require better guidance on how to systematically assess the situation and aircraft state following a failure, not only by looking at the most present failure, but also looking at other threats related to the failure. This concept builds upon existing failure and threat-management elements that are already part of the pilot training curriculum, although these competencies are trained sparingly (usually part of one-time command training). Examining a good contemporary example of such a failure management process, this procedure can be reduced to the simplified diagram shown in Figure 1.
Figure 1: Failure Management Process extended with threat identification.
In this process, crews collect information from different sources in the flight deck, prioritize the information and determine the status of the aircraft (systems). They must determine what information should be used or responded to right away, what information can be ignored temporarily and what information can be left aside. This process can help prioritize immediate actions, and manage the time-criticality of the situation. The crew then has to anticipate the situation, in the first place by selecting the appropriate procedure to solve or contain the problem. Finally, the crew should consider the various scenarios (return, continue or divert), taking into account the status of the aircraft and environmental conditions, and make a decision on the final plan. This process is designed to increase situation awareness and help crews systematically identify threats and then decide on an action plan, based on the status of the aircraft and environmental conditions.
In the experiments, most crews tended to focus initially on the most obvious problems (engines surging and failing), which often resulted in correctly applying the first few steps of this failure-management process. However, after completing the memory items and procedures for the engine problems, many crews failed to continue and perform an assessment of their aircraft state, and create a suitable long term plan. Instead, the crews terminated the failure-management procedure and decided to land the aircraft (as soon as possible) without evaluating for instance aircraft controllability, structural integrity, aircraft performance or environment (i.e. weather). As a result, some safety margins were exceeded, which in reality would have required re-training.
Regarding the main purpose of the threat-management procedure as a means to understand the full situation and condition of the aircraft and ensure operation within the margins of safety (with respect to for instance attitude, altitude, speeds, [approach] stability, thrust versus drag and energy management). Besides the recommendation to fortify threat-management training, it is proposed that the process is adapted to improve crew awareness during such situations. The proposed adaptation introduced a dedicated threat-identification and prioritization step. This step, although partially present as long term planning in the current process, must occur earlier and thoroughly. The reason for this is that crews become aware of the level of temporal pressure, or lack thereof. By being mindful of the actual temporal pressure, crews may either create more time by prioritizing time critical threats, and/or be able to create more time to further understand their aircraft state, diagnose the problem and plan effective actions. Keeping this in mind, the current aspects of the failure management procedure have been reviewed and it is proposed to improve the process as depicted in Figure 2.
Figure 2: Failure Management Process extended with threat identification.
4.1.2. Managing uncertainty
A second solution concept which may prove to be helpful toward proper assessment of the aircraft state is that of managing uncertainty. Lipshitz & Strauss (1997) investigated the connection between types of uncertainties and various coping mechanisms to reduce the uncertainty. Types of uncertainties may include lack of information, unreliable information, inadequate understanding and undifferentiated alternatives. Types of coping tactics are collecting information and advice, assumption based reasoning, avoiding irreversible action, forestalling and weighing pros and cons. The experiment asked 102 students of the Israeli Defence Force (IDF) to write a case of decision making under uncertainty, from their own experience. These cases were subsequently analysed for the types of uncertainty and the coping mechanisms used. The resulting connection between types of uncertainties and the effective resolution strategies bore a new naturalistic decision making heuristic called RAWFS: Reduction, Assumption-based reasoning, Weighing pros and cons, Forestalling, Suppression. The RAWFS heuristic has been hypothesized as presented in Figure 4. The RAWFS concept may work as a mnemonic of sorts, aimed specifically at resolving uncertain and ambiguous situations.
4.2. Solution recommendations for training
The identifiable connection between crew performance and specific competencies indicates those competencies that are most helpful in managing unexpected and challenging events, in addition to those whose absence is most linked to poor performance and unsafe outcomes. Results indicate that flight crews that are strong in leadership & teamwork, communication, and problem solving & decision making are better prepared to handle challenging and unexpected flight events.
Based on these results, it is recommended to investigate whether it is possible to adapt training in such a way to maintain specific skills in order to cope with unexpected situations. Competency-Based Training is a method that is currently being introduced by some operators and training organisations to fulfil this training need. Such training would adapt training scenarios in simulators in order to create opportunities for frequent exposure to demanding and unexpected situations that cover the most helpful core competencies, preferably in scenarios that are relevant for the operation of modern airliners. Competency-Based Training is expected to supersede conventional event-based training, which has the risk of the development of memorized skills that only work for specific events occurring during training (Casner, Geven & Williams, 2012). It must be stated that despite this study indicating that crew performance can be reflected in specific competencies, it is yet uncertain to what extent training those specific competencies will reflect in better performance for other flight crews not only in a specific scenario but also in other challenging scenarios. Based on the results from this exploratory experimental campaign we propose the following (tentative) recommendations:
• Resilient flight crews need to be strong in the following core competencies:
o leadership & teamwork
o communication
o problem solving & decision making
• Training scenarios should expose flight crews to several challenging and unexpected situations. These must include scenario elements specifically targeted at the above competencies. Additionally, scenarios may be sufficiently diverse to cover most or all of the operational settings and various types of challenges that crews may be exposed to.
• The operational scenarios should be different to avoid memorization, but should cover similar core competencies.
Since the previous experiment contained a single operational scenario for each group of crew, and a limited number of crews, it is necessary to broaden the experimental evaluation for validation of this recommendation. In the next phase it is desirable to test whether flight crews flying different scenarios that focus on the same competencies show the same performance in these competencies.
4.3. Solution recommendations for cockpit design
The above three problems (energy management, engine management & wind-shift awareness) may find their resolution in novel HMI display concepts, which will be detailed in this section. It should be mentioned that these displays are a first move at a novel cockpit design philosophy that focusses on providing crews with positive system state information (i.e. what is still possible, as opposed to what has failed), in order to expedite the process of recovering awareness of the situation and changes by removing uncertainties in aircraft limits and capabilities. In addition to assisting state awareness, this philosophy can also benefit flight planning because crews can more easily generate options are they are presented with possibilities as opposed to restrictions. This new concept will also require some level of training and/or familiarisation, which is not explicitly covered in this section, but may find overlap with any training embedded in the procedural solutions.
4.3.1. Energy display
The flight crews’ awareness of the aircraft’s energy state and their energy management capabilities could possibly be improved by implementing an energy display into the flight deck. The display uses a layout of a vertical situation display, but gives the crew a clear indication of the aircraft’s current kinetic energy status in terms speed deviations relative to the selected (or managed) target speed (in magenta). Furthermore, the display prompts the crew at the time when they have to select new target speeds and aircraft configurations (flaps and gear).
4.3.2. Mitigate aircraft state uncertainty by presenting “positive limitations”
As proposed in the industry’s instructor comments’ analysis means to “internalize external parameters”, that is directly communicating effects of perturbations, were proposed. For this task, the ECAM, PFD and ND displays could be modified in order to directly communicate the perturbations’ effects to the crew in a very simplified way. The most profound change with respect to current displays is the display of “positive limitations”, in other words what controllability, performance and systems are possible, instead of only communicating failures and restrictions. This philosophy should assist crews in more easily mitigating any uncertainty as the new philosophy aims to present information about the pertinent effects of the system state change, as opposed to information about the change itself. The events of the first set of M4G experiments mainly had an impact on the aircraft’s performance. Therefore, it would be beneficial to have the performance implications in terms of the new flight envelope displayed to the crew.
4.3.3. Modified wind indication on the ND
Two possible display modifications are proposed. The idea is to catch the pilots’ attention in wind change situations by temporarily changing the wind arrows size and colour on the ND:
1) When the change rates of wind direction or wind speed are exceeding predefined thresholds, temporarily increase wind arrow’s and colour code it amber.
2) When the wind components for cross and/or tailwind are exceeding the aircraft’s operational limits, temporarily display a magnified red flashing wind arrow.
5 MAN4GEN RECOMMENDATIONS
The operational recommendations that were developed within the Man4Gen project covered three areas: Procedures, Training and Systems.
5.1. Procedural Development
The procedure aims to assist crews in managing surprising, unexpected and diffuse/ambiguous situations where conventional training and published procedures may be insufficient. Some characteristics of such situations:
- These situations are not immediately clear. They require time and attention to understand.
- These situations can develop fast or slowly over time (i.e. dynamic), and often consist of multiple events/failures for which there are not standard checklists.
- These situations test the boundary of the current electronic support (e.g. ECAM) and procedural training that crews get.
- These situations are often untrained and crews are not familiar with them.
The procedure will be a strategy, based on the results from the exploratory experiments, which assists crews in these situations. The term strategy is used explicitly, as the procedure must not allude to a strict checklist or do-list. Rather it must support a crew’s cognitive flexibility, judgement and airmanship to properly assess and manage a complex and ambiguous situation. It must not, however, be so high-level that it becomes too unclear what is meant (this is the risk that a basic mnemonic design has).
This procedure is designed to limit the cognitive load on pilots at the start of the procedure (in consideration of stress and emotional responses), and increases this load as the procedure attempts to increase the cognitive capacity available (by “calming” or focusing the crew). We cannot expect them to do a full diagnosis before executing important memory items. Similarly, we should not waste the capacity they have once they have time and (comparative) peace of mind to diagnose the situation and make a plan.
The strategy encompasses three philosophy concepts:
1) Manage time criticality, such that you have time/calm to...
2) Manage (un)certainty, such that you can effectively...
3) Plan for contingencies and changes
Similar to the purposeful order of the above concepts, the procedure itself is structured into six phases to ascertain sufficient crew and aircraft capacity for each task required:
Phase 1: Stabilize flightpath
Phase 2: Manage immediate threats
Phase 3: Short term planning
Phase 4: Identify situation
Phase 5: Perform appropriate actions
Phase 6: Long term planning
This procedure should by no means contradict manufacturer procedures, checklists and guidelines. As a matter of fact, we are supplementing their published resolution strategies with one for situations they cannot cover. The key aim of this procedure is to provide a strategy for flight crew to deal with unexpected situations for which there is no procedure available. This final procedure concept is designed to support flight crews in both critical acute events as well as (possibly subsequent) complex, ambiguous situations. Within the validation experiments, we intended to evaluate both these aspects of the procedure. Evaluation of the procedure will focus on the overall safety performance of the crew, and compare this to a measure of application of the procedure and its components. These observations will be supplemented with subjective reviews from subject pilots. This evaluation should elicit whether crews trained in and using the procedure perform better, as well, different or perhaps worse than comparable crews without the procedure and its training. In addition, the evaluation will indicate which aspects of the procedure are associated with a performance delta, and how the performance changes.
The procedure will also be evaluated in conjunction with the display improvements in a second experiment at DLR. This experiment has been setup to evaluate the collective design philosophy shared between both solution developments. A similar procedure evaluation will be done, and a comparison with the procedure-only experiment results may indicate whether the addition of a display affects how crews use the procedure, and the performance they achieve.
Both experimental results will indicate what a final tuned version of the procedure might be, by retaining the effective components, and adapting or removing others. Also the training and interface with display improvements will be reviewed to indicate how procedure effectiveness may be increased.
5.2. Training Development
A possible solution identified to improve flight training would be to train competencies instead of pre-described flight maneuvers, many of those being non-technical. Such competency transfer between scenarios could create more resilient flight crews that are more prepared to handle operational events instead of the scripted and sometimes predictable scenarios currently used in training. A way to do this would be the implementation of Evidence-Based Training (EBT) concepts. To support this training methodology, this deliverable presents a scenario development method to generate scenarios for competency-based training.
5.2.1. Desktop evaluation
The objective of the evaluation phase is to assess the flight crew competency level. Therefore, the scenario designed for this phase should be general so that all pilot core competencies can be evaluated in order to identify the main training needs. This is opposed to the training phase scenarios that should trigger the competencies lacking in certain flight crews. The evaluation phase scenarios are then more general because they are used to evaluate the flight crew proficiency in all competencies. Although the evaluation phase could be done with a general flight scenario in a full-flight simulator, in the Man4Gen project we are trying to evaluate these competencies with a non-flight deck related desktop simulation.
5.2.2. Training Phase
The main objective of the training phase is to improve the flight crew proficiency on the pilot core competencies where more difficulties have been previously identified. Therefore, for optimal training the scenarios should be personalized to the flight crew. For the Man4Gen scenarios, we used the process developed earlier in the project to generate the experimental scenarios. This process focused in the problem statement themes, rather than competencies, because here we were not developing scenarios to trigger the needs of an individual flight crew. The goal of the Man4Gen experiments was not to train the needs of a flight crew, as it should be during the training phase, but to investigate whether competencies transfer between different scenarios.
The scenario development method described in this document is only effective if the competencies transfer between scenarios. This is, if the competency proficiency of a certain flight crew is similar between different flight scenarios that require similar competencies in order to be handled successfully. This competency transfer was addressed in the second experimental campaign of the Man4Gen project. If the results from these deliverables are positive, it means that we can observe competency proficiency from scenarios that require certain competencies. This would also be indicative that this scenario development method is appropriate since it focuses the scenario development around specific key competencies that are required by certain flight crews. However, we still need to show that exposure to certain competencies will indeed increase flight crew proficiency on them. That was outside the scope of the Man4Gen experiments.
Additionally, if the competency transfer works not only for the flight scenarios, but also for the desktop exercise, it means that this method can be used in the evaluation phase described in the EBT manual. The potential savings here could be significant if full flight simulator usage can be limited or even scrapped in the evaluation phase. Also, if competencies can be trained by exposure, flight crew training becomes more effective because it can focus training on the flight crew specific difficulties which will lead to more resilient flight crews.
5.3. Systems Technology Development
The improved cockpit system development thus focusses on supporting the crew’s capabilities to perform the modified procedure. The new systems provides six additional pages on the lower ECAM that summarize the current situation in terms of controllability, performance, endurance and automation as well as the impact on the upcoming flight phases. This information shall enhance situation awareness (SA) and support the short-term as well as the long-term planning of the crew as it is intended by the modified procedure. Important principles in designing this modified display are that it does not conflict with existing procedures, does not add to the design complexity, does not require significant additional certification, and provides an ancillary level of support to flight crews.
The Risk Information System consists of six additional pages located on the lower ECAM display; besides the regular system synopsis pages and the status page. These six pages provide collective interpretations of the effects of an augmented aircraft state, based on information available to the aircraft information systems, notable the flight warning system. These interpretations and their structure should serve to support flights crews in situation analysis and awareness, the subsequent decision making process, when confronted with an uncertain aircraft state and uncertainty in how to proceed. These six new pages are accessible via a set of buttons on the transponder (XPDR) panel.
The cockpit system modifications implemented in the project nclude additional pages for the lower ECAM of the Airbus A320. These pages shall provide an overview of the situation on an overall aircraft level and not on a system level. The design of the pages is still preliminary and should be considered as first prototype to assess the usefulness of such a display concept. Hence the task of the simulator experiments is to perform a proof of concept. The aim of the validation experiments is to analyse what impact the new Risk Information System has on the overall crew performance and especially on the crew’s decision making process during the scenario.
6 CONCLUSIONS OF THE PROJECT
The conclusions of the project have been presented in the form of deliverable reports that will be made available to the aviation and research stakeholders.
6.1. Operational Conclusions
The primary conclusion of the project was that there is a benefit in examining the preparation of flight crew to deal with unexpected events. During this project it became clear to the researchers that there are areas where improvements can be made in order to work towards the desired reduction in accident rates. By including the three different areas – training, procedures and systems – the project team has covered recommendations for both shorter and medium term concepts. The aim of the project was to deliver recommendations that could be applied by industry stakeholders such as regulators, manufacturers and operators. The final results of the project take the form of material that can be used by the stakeholders to support development and changes in training for example, or can be used as a foundation to re-examine existing procedures and display designs.
The findings of the operational work are that there are improvements that can be made to current practices – particularly with reference to training and operational practices in the shorter term – that can address the potential for problems in modern airline operations. It has been identified that it is possible to support and prepare flight crew to deal with unexpected events, potentially including in the form of additional systems support in the longer term. It is hoped that the results of the Man4Gen research can be picked up by the industry in continuing work to further improve the safety of operations, thus preparing the operation to handle unexpected events.
- D6.1 Report on Operational Procedures recommendations: This report contains the recommendations for airlines and industry for operational procedures improvements as result of the evaluations at NLR’s GRACE simulator.
- D6.2 Report on Training recommendations: This report contains the recommendations for training improvement based on the experiment results from the training experiments in the GTA simulator.
- D6.3 Report on Systems recommendations: This report contains the recommendations for developments in aircraft and cockpit systems technology.
- D6.4 Final Report of Operational Guidelines: This report brings together the recommendations and findings of the three areas of research for the use of aviation stakeholders such as regulators, airlines and manufacturers.
6.2. Research Conclusions
In addition to the operational conclusions that the project has delivered, the research team have developed and investigated different research methodologies that can be applied in the fields of cognitive science, neuroergonomics, and cognitive systems engineering. Within cognitive science the project team focused on the perspective of “situation(al) awareness”, and within cognitive systems engineering the sensemaking perspective was applied. Methodological considerations from each perspective were discussed. A triangulation of methods has been carried out in the project, including the two research strands and the industry based assessment. Next to conventional assessment techniques, methods were applied that are new and innovative, especially in the field of aviation (such as fMRI, social SA tools, and ECOM). The combination of the analyses and outcomes of these techniques (fMRI, [social] SA and sensemaking) resulted in a mutual validation of the applied approaches. Therefore, we are now better able to test the interfacing between man and machine (MMI), the required levels of training, and associated factors in a dynamic test and simulation environment, which allows developing new MMI configurations. The results from SA and SM have individually been linked to the industry assessment and part of the results from the two research perspectives was combined to demonstrate how different methods can complement and validate each other. The overall results from the Man4Gen experiments show that there is a great variation in how the crews understand, prioritise, manage uncertainties and take action following the unexpected events introduced in the scenarios. Operational recommendations and the findings that support them on how to improve crew abilities are summarised in the final report D6.7. The project has opened up a new domain in research options and techniques that can be instrumental for cockpit design, layout and training. The findings further raise several important questions regarding challenges and possibilities for pilots to maintain control in surprise situation.
- D6.5 Report on Situational Awareness research: This report summarises the main conclusions regarding Neuro-ergonomics and Situational Awareness in 4th generation aircraft cockpits based on the evaluations of the experiments.
- D6.6 Report on Resilience research: This report includes the findings from Man4Gen experiments in the context of the resilience engineering sense-making research. The research results should be described as part of the application to the operational requirements for operating 4th generation aircraft.
- D6.7 Final report of research methods and results: This deliverable brings together the research findings throughout the project regarding methods for identifying, analysing and assessing sensemaking and situation awareness in operational environments. The report includes the outcome of the experiments in the research simulators and the application of fMRI technology in research of pilot flying and pilot monitoring tasks.
Potential Impact:
7 POTENTIAL IMPACT OF THE PROJECT
The research methodological work carried out within the project has developed methods and scientific results that have an immediate application within the research fields: cognitive science, neuroergonomics and cognitive systems engineering. For example the methods developed in this project are already being picked up by other EU research projects for application and further development to measure and report human performance in similar aviation experiments.
The results of the project have a potential impact on the operational sphere of the aviation industry. The results were developed and presented with the primary aviation industry stakeholders in mind. The external advisors of the project made up a combination of expertise from regulators, manufacturers and operators, who all indicated that the research results of the project would be a useful contribution to existing developments within the industry.
• Training: The aviation industry is currently reviewing the training methodologies that are used, Competency Based Training and Evidence Based Training are methodologies that are increasingly being applied by operators and manufacturers. The regulators are also adopting these methodologies within regulations and advisory material. The results of the Man4Gen project demonstrate how competencies can be designed into training scenarios, how they can be assessed in training, and how alternative training tools can contribute to the training. Through the broader application of competency based training the training of flight crew can lead to competency development for a variety of situations, and for unexpected situations.
• Procedures: The potential advantage of a strategy that can be applied in the form of a procedure or guidance for flight crew is that this can be adopted by operators with local training. This is therefore a conclusion that can be implemented in the short-term. The impact of the research that was conducted in this strand of the project is to inform the ways that flight crew operate and function in response to unexpected events. Additionally it was noted by the researchers that the procedure developed in the project may be useful as a training tool, to promote a strategic approach to dealing with unexpected events. These results of the Man4Gen project have been presented and discussed with aviation professionals and there is interest in taking the concepts of the project for further development and application.
• Systems: The concept that was investigated in the systems research strand – the risk management display – is closely aligned to the strategy that was developed in the procedural concept within the project. The display promoted a discussion within the aviation industry experts in the advisory board as to the different ways to communicate the information with the flight crew about the risks and threats to the aircraft. This research strand presented a concept that can be picked up for future research within other projects.
The partners of the project have taken the results and are investigating ways to further disseminate and exploit the results of the project beyond the activities that were already carried out within the project timeframe.
8 EXPLOITATION AND DISSEMINATION
8.1. Man4Gen website
A web site has been put in place with information about the Man4Gen project to inform the general public. The project website describes the consortium, project goals, purpose and objectives, schedule and agenda, upcoming events, results and reports, links to articles, links to social media, online discussion forum, information for the general public. The website address is: www.man4gen.eu.
8.2. Man4Gen video
At the end of the project a video has been produced that explains the main research questions, the experimental approach of Man4Gen and the main conclusion of the project. It is presented in a visually attractive way with a narrative explanation. The video has be placed at youtube: https://www.youtube.com/watch?v=mVLuck49Vk0
8.3. Dissemination
Several presentations and papers about the research have been disseminated. An overview is presented in the table below:
Event Type Date Audience
European Association of Aviation Psychology Papers and Presentations September 2012 Aviation Psychology researchers and aviation professionals
2nd Annual CogSci "LORENZ-Lecture" Poster presentation 7 November 2012 Rectorate ViU, scientists in the fields of cognitive science, animal cognition, psychology, biology and students
5th symposium on Resilience Engineering Paper and presentation 25-27 June 2013 researchers/practitioners within the resilience engineering community
Annual Conference on Neuroeconomics: Decision Making and the Brain Poster presentation and abstract 26-29 September 2013 Scientists
3rd Annual CogSci "Bühler-Lecture" Poster presentation 20-21 January 2014 Rectorate ViU, scientists in the fields of cognitive science, animal cognition, psychology, biology and students
Psychoneuroendocrinology Journal Journal article January 2014 Scientists and researchers
ICAO Loss of Control – In-flight symposium Paper and presentation May 2014 Aviation professionals, International policy makers (Government and Industry)
EASA Loss of Control – In Flight workshop Paper and presentation September 2014 European regulators, aviation professionals
Aviation Psychology and Applied Human Factors Journal article December 2014 Aviation psychologists, Human factors researchers and practictioners
Organisation for Human Brain Mapping Poster and Presentation June 2015 Neuroergonomics, researchers, scientists and medical professionals
NeuroImage journal Journal article July 2015 Medical scientists and researchers
AeroSafetyWorld Publication article July 2015 Aviation industry members
International Society of Aviation Psychologists Paper and presentation June 2015 Aviation psychology researchers, aviation professionals.
European Association of Aviation Psychology Papers and Presentations September 2014 Aviation Psychology researchers and aviation professionals
6th symposium on Resilience Engineering Paper and presentation June 2015 researchers/practitioners within the resilience engineering community
Aerodays Presentation October 2015 European Commission, Industry, Research and Politicians
AIAA SciTech Forum and Exposition Paper and presentation January 2016 Aviation professionals and researchers
In addition to these dissemination activities, the results of the project are currently being written up in the form of papers and journal articles that will be submitted by the project partners in the coming months.
8.4. Exploitation of Results
The results of the Man4Gen project have been discussed and picked up by a number of organisations in the aviation industry. There is interest in the project, and project partners have received questions and expressions of interest in the results.
From the publication of the results from the exploratory experiments during the project a number of international airlines made contact with the project team to find out more about the work that was being carried out. These airlines have been in contact with the project team to investigate how the concepts investigated in the project can be applied within their training programmes.
Additionally, a study is being carried out for EASA where the results from the Man4Gen project are used as the starting point for research into startle effect management for flight crew. The results of the project are being directly picked up and applied further to ensure that the research work of the Man4Gen project contributes to the recommendations being developed by EASA.
The expectation of the project partners is that further organisations, operators, airlines, manufacturers, regulators and researchers will take the concepts and results of the Man4Gen project and apply them further in the future.
9 EXTERNAL ADVISORY BOARD
The aim of the EAB will develop as the project continues. As a whole, the EAB is intended to scrutinise the work that the consortium is carrying out, and ensure that our work is applicable outside the project. For that reason the EAB includes advisors from several important areas of the aviation and (scientific) research communities. There are 25 international subject matter experts from the aviation industry, airliners, training providers, regulation authorities, and universities in the EAB.
Two meetings with the External Advisory Board (EAB) have been organised during this reporting period. The first meeting was organised after the first results from the literature study and automation analysis. These results were presented and the problem statement has been discussed with the EAB members. As outcome of the discussion the problem statement has been adjusted and some useful recommendations for the direction of the research methods and the design of the experiments have been received. During the second meeting the design of and the scenario for the simulator experiments at DLR and NLR have been presented and discussed. Some useful comments were received to sharpen the experiment aims.
EAB meetings:
Location Date Number of EAB members
NLR, Amsterdam 21 March 2013 7
Swiss Aviation Training, Zurich 18 December 2013 5
DLR, Oberpfaffenhofen 9 December 2014 3
Boeing, Seattle 6-7 and 18 March 2015 2
London 6-7 and 18 March 2015 1
Vienna University, Vienna 6 November 2015 2
EASA, Cologne 10 December 2015 6
List of Websites:
Website: http://man4gen.eu/
Email: man4gen@nlr.nl
Coordinators Email: arjan.lemmers@nlr.nl joris.field@nlr.nl
While aviation is an extremely safe mode of transport, there has recently been a steady rise in the number of accidents that are attributed to limited manual handling skills by pilots. These accidents are often due to a combination of the crew not managing the aircraft systems effectively after an unexpected event, and being unable to apply appropriate manual handling skills. Man4Gen aims to identify the common thread behind the events that lead to these accidents, and to recommend short-term changes to operational procedures, training and aircraft systems technology in order to mitigate this threat to aviation safety.
A procedure has been developed to assist crews in managing surprising, unexpected and ambiguous situations. Three philosophy concepts come back in this procedure: manage time criticality, manage (un)certainty and plan for contingencies and changes. The procedure is supplementing the manufacturer procedures and checklists, by providing a strategy for flight crew to deal with situations for which there is no procedure available. The training solution focuses on train competencies instead of pre-described flight manoeuvers. To support this, a scenario development method has been worked out to generate scenarios for competency-based training. The improved cockpit system development focuses on supporting the crew’s abilities to perform the modified procedure. It consists of additional pages of the Electronic Centralized Aircraft Monitor (ECAM) to summarize the current aircraft situation in terms of controllability, performance and endurance.The implemented solutions for procedures, training and cockpit displays, have been evaluated in experiments in either a research or a training simulator setting where the behaviour of the flight crew to the modifications was evaluated. In addition to the three planned experiments at the NLR, DLR and GTA simulators, an additional training experiment was carried out at the training centre of a major airline outside of the consortium.
The analyses for the three operational research strands were brought together with the results of the exploratory experiments for the final analysis. The findings of the operational work are that there are improvements that can be made to current practices that can address the potential for problems in modern airline operations. It has been identified that it is possible to support and prepare flight crew to deal with unexpected events, potentially including in the form of additional systems support in the longer term. The main outcomes from the two research strands, SA and sensemaking, were analysed. Next to conventional assessment techniques, methods were applied that are new and innovative, such as fMRI, (social) SA, and sensemaking. The combination of these techniques results in a mutual validation of the applied approaches. Therefore, we are now better able to test the interfacing between human and machine (HMI), the required levels of training, and associated factors in a dynamic test and simulation environment, which allows developing new HMI configurations.
Project Context and Objectives:
Modern highly automated aircraft (4th Generation airliners) are extremely safe, and there is a particularly low chance of an accident when operating these aircraft. Automation clearly plays a very positive role in enhancing aviation safety and preventing accidents. However, there has recently been a steady rise in the number of accidents that are attributed to the ability of flight crew to assess and understand an unexpected situation, and consequently respond appropriately to handle the situation, and eventually in some cases limited manual handling skills by pilots.
The main objectives therefore of the project were twofold:
- To identify the factors in highly automated, 4th generation, aircraft that affect the ability of the flight crew to handle unexpected events and gradually deteriorating conditions to maintain effective control of the aircraft.
- To subsequently identify methods to prepare flight crew to deal with unexpected events, modifying the training, procedures and systems available to operators of highly automated aircraft today.
The project was set up to achieve these objectives using a phased approach. The initial phase of the project focused on establishing the background of the problem from an academic and an operational perspective. This was then applied in an experimental setting to reproduce the challenges faced by flight crews in a way that their behaviour and responses can be studied. The results of the experiments formed the basis for an initial concept development to improve the flight crew’s ability to handle challenging situations, and subsequently focus on the development of aviation industry guidelines. The guidelines were applied and evaluated in an operational/experimental setting to validate their application.
In the first period of the Man4Gen project a literature study was performed and this was used to identify the roadmap for the project. Research methods to investigate how crew and aircraft successfully handle unexpected events from a Cognitive Systems Engineering (CSE), or sensemaking, perspective have been developed. Exploratory experiments in research flight simulators have been executed to identify the processes that lead to confusion or loss of SA when using automation. In addition to the flight simulator experiments, experiments have been carried out in an fMRI environment to establish the brain regions that exhibit significant activity in response to situational awareness changes. These experiments are intended to demonstrate a link between known brain activity and situational awareness, to be able to examine situational awareness further from a theoretical and neuropsychological perspective in the forthcoming analysis work.
In the second phase of the project the results of the exploratory experiments have been analysed focusing on three research tracks: operational analysis, sensemaking and Situational Awareness (SA). The sensemaking and SA analysis highlights a number of issues and gaps in communication, information and uncertainty management and prioritising implying the need to support the crew in these areas. The operational analysis identifies competencies where high-performing crews are highly-rated. With these results concepts of solutions for the above mentioned problems have been identified in three areas: procedures, training and cockpit design. A selection has been made of these solutions for further development in the Man4Gen project.
This final report addresses the results of the Man4Gen project.
a. Overview of the results
b. Conclusions on the project
c. Exploitation and dissemination
Project Results:
2 FINAL TECHNICAL REPORT
2.1. Overview of the results
In the first period of the Man4Gen project a literature study was performed and this was used to identify the roadmap for the project. From this literature study the following problem statement for the project has been derived: Despite the substantial and proven safety benefits of automation systems in 3rd and 4th generation aircraft, evidence indicates that when faced with unexpected and challenging situations, pilots sometimes have difficulties in quickly responding to situations which require a rapid transition in their activity from monitors of very reliable systems, to active and authoritative decision-makers exercising manual control of the aircraft.
Research methods to investigate how crew and aircraft successfully handle unexpected events from a Cognitive Systems Engineering (CSE), or sensemaking, perspective have been developed.
Exploratory experiments in research flight simulators have been executed to identify the processes that lead to confusion or loss of SA when using automation.
In addition to the flight simulator experiments, experiments have been carried out in an fMRI environment to establish the brain regions that exhibit significant activity in response to situational awareness changes. These experiments are intended to demonstrate a link between known brain activity and situational awareness, to be able to examine situational awareness further from a theoretical and neuropsychological perspective in the forthcoming analysis work.
In the second phase of the project the results of the exploratory experiments have been analysed focusing on three research tracks: operational analysis, sensemaking and Situational Awareness (SA). The sensemaking and SA analysis highlights a number of issues and gaps in communication, information and uncertainty management and prioritising implying the need to support the crew in these areas. The operational analysis identifies competencies where high-performing crews are highly-rated. With these results concepts of solutions for the above mentioned problems have been identified in three areas: procedures, training and cockpit design. A selection has been made of these solutions for further development in the Man4Gen project.
A procedure has been developed to assist crews in managing surprising, unexpected and ambiguous situations. Three philosophy concepts come back in this procedure: manage time criticality, manage (un)certainty and plan for contingencies and changes. The procedure is supplementing the manufacturer procedures and checklists, by providing a strategy for flight crew to deal with situations for which there is no procedure available. The training solution focuses on train competencies instead of pre-described flight manoeuvers. To support this, a scenario development method has been worked out to generate scenarios for competency-based training. The improved cockpit system development focuses on supporting the crew’s abilities to perform the modified procedure. It consists of additional pages of the Electronic Centralized Aircraft Monitor (ECAM) to summarize the current aircraft situation in terms of controllability, performance and endurance.
The implemented solutions for procedures, training and cockpit displays, have been evaluated in experiments in either a research or a training simulator setting where the behaviour of the flight crew to the modifications was evaluated. In addition to the three planned experiments at the NLR, DLR and GTA simulators, an additional training experiment was carried out at the training centre of a major airline outside of the consortium. The analyses for the three operational research strands were brought together with the results of the exploratory experiments for the final analysis. The findings of the operational work are that there are improvements that can be made to current practices that can address the potential for problems in modern airline operations. It has been identified that it is possible to support and prepare flight crew to deal with unexpected events, potentially including in the form of additional systems support in the longer term.
The main outcomes from the two research strands, SA and sensemaking, were analysed. Next to conventional assessment techniques, methods were applied that are new and innovative, such as fMRI, (social) SA, and sensemaking. The combination of these techniques results in a mutual validation of the applied approaches. Therefore, we are now better able to test the interfacing between human and machine (HMI), the required levels of training, and associated factors in a dynamic test and simulation environment, which allows developing new HMI configurations.
The following sections describe in more detail the stages that were carried out in the project:
• Exploratory Experiments
• Development of concepts to improve safety in the operation
• The Man4Gen recommendations
• The conclusions from the final validation experiments
3 ANALYSIS OF THE EXPLORATORY EXPERIMENTS
At the end of the work package 2 exploratory experiments an initial analysis was carried out to collect and report the measures that were described in the experiment plan. This initial breakdown was reported at the end of the exploratory experiments and formed the starting point for the analysis work for the recommendations. The analysis was carried out by the project partners applying three research foci: Operational analysis (Airbus, Boeing, NLR), Sensemaking (Linköping University, NLR), Situational Awareness (Vienna University, DLR).
3.1. Initial Findings SA Analysis
Crew analyses revealed that the following gaps in SA occurred: lack of weather awareness during ILS (i.e. wind), lack of system awareness after the bird strike (i.e. engine status), lack of correct application of procedures and checklists after the bird strike, lack of energy awareness after the bird strike, and lack of flexibility in decision making.
RQ: Why did those gaps in SA occur and why did crews perform better or worse?
(1) In our initial analysis we evaluated the data from the various questionnaires (i.e. NASA TLX raw, SART, SSA-VAS) and found a significant change in workload and SA over the scenario. We performed a correlation analysis and found a significant relationship between workload and SA. Workload and stress patterns increased during surprising and challenging flight phases and at the same time SA decreased. Thus, in our exploratory study an increase in workload and stress were linked with a decline in different types of SA.
(2) In the fMRI experiments we found evidence that pilots accompany monitoring, understanding, anticipation and decision making by mentally verbalizing or quietly murmuring. This highlights that communication is a very important factor for SA as well as for all shared forms of SA. Additionally, this finding supports the hypothesis of “shared mechanism”, that language and sensorimotor cognition are linked. Furthermore, we found that communication (with self or others) directly is encoded with cognitive processes that are linked to the experience collected in the named context (i.e. memory). Consequently, less effective oral and non-verbal communication in the cockpit can be related to SA and a possible worse performance.
(3) Similar to the fMRI results, the role of effective communication also was an issue identified in the simulator studies, where crews were rated better if they successfully communicated. Consequently, communication seems to serve various purposes within a socio-technical system: (A) assisting first, second and third level SA (i.e. perception, understanding or making sense, anticipating), (B) supporting recall from memory, which is linked to point (A), and (C) sharing one’s mental model and consequently fostering shared and team SA. We observed different communication patterns in crews. Amongst other things, the linear model showed that crews were rated better if PM was able to recall information from memory, checklists or the system by himself and if he was more active in explaining. Framed in differently, crews were rated worse if PM requested more information and explained actions less during surprising and challenging situations, i.e. bird strike.
(4) PF behaviour that emerged as being relevant for better crew performance was an increased amount of explaining actions during go-around and during final approach and land. This implies that sharing one’s mental model has a positive effect on shared and team SA. Additionally, better crew performance was linked to the PF appropriately selecting the type and amount of communication. The PM in crews which received higher rating scores seemed to be better able and more ready to receive information. A lack in these abilities, i.e. the ability and amount of sharing mental models and verbalizing current and planned actions, had an impact on the different gaps in SA observed in our scenario.
(5) If PF used initiative and gave directions when required, then the crew received better ratings.
(6) Our assessment of non-verbal communication showed that crews were rated better if PF used greater amounts of non-verbal communication during final approach and land and PM used greater amounts during initial approach. Iconic gestures are known to accompany speech in several ways: (A) redundant (e.g. “to the right” and gesture pointing to the right), (B) supplementary (e.g. “this direction” and gesture depicting the path to the left), and (C) complementary (e.g. “this” and gesture pointing to an engine on the engine display). Consequently, non-verbal communication conveys information, but also catches attention and supports understanding. Speakers’ gestures affect listeners’ (re-)actions. Effective non-verbal communication by PF and PM during different stages in the scenario correlated with better crew ratings.
(7) We found evidence from the SSA evaluations that the PF’s and PM’s abilities to assess their colleague’s level of stress and control were diminished during challenging and stressful flight phases. Our results show that stress affects the perception and understanding of others, i.e. leads to decreased empathic accuracy. The latter again might lead to decreased abilities to understand the current emotional and cognitive state of the other crew members and therefore decrease SSA. Consequently, we propose continuing to focus on social SA topics in aviation.
3.2. Initial Findings Sensemaking Analysis
The analysis of debrief data highlighted a number of issues implying the need for support for crews to search for information, manage uncertainties, prioritise and make trade-offs between goals at various levels of granularity, assess risks, consider options and anticipate future events. Combined with the contextual control strategy analysis we can conclude that some crews do this well by aligning priorities and risk assessments through an interaction between discussion considering options, risks, and expectations, and action acting upon these assessments, followed by monitoring of results of these actions against high-level goals.
The extent and content of the assessments made by the individual crews varies considerably, from crews that agree they should have taken more time to assess the best runway once engines were stabilized, to crews who feel there was no other option but to land on the nearest runway. Improving the crew-automation system’s ability (e.g. through training, procedures, display design) to improve the understanding of technical issues and assessing their potential consequences therefore seems to be needed. Also, considerable differences were found between the two crew member’s assessment of the situation, what variables are considered, what risks, goals, and priorities are identified, and what actions should be prioritised. The overall ability to assess whether immediate action is needed, how much time is available to assess and problem-solve, based on an identification and appreciation of risks and goals, seems to be a subject of considerable variability and therefore deserves attention while this project generates its recommendations. Variations and misunderstandings in the use of autopilot and engine management imply that this ability to appreciate risks and time constraints and possibilities goes hand-in-hand with the technical understanding of the complex systems that are difficult to understand in pressed situations, making these joint crew-automation issues rather than issues of individual pilots.
3.3. Initial Findings Operational Analysis
The “Desired Flight Crew Performance” (DFCP) analysis displays the flight crew’s overall performance in the operational scenario. Results show that high-performing crews in this scenario were highly rated in Communication, Leadership and Teamwork, Problem Solving and Decision Making, Situation Awareness, and Workload Management. These competencies need to be paired together since some of them are a consequence of good performance in the others. For example, Communication by itself is not indicative of good performance since this competency is only a medium to propagate good behaviour in the other competencies identified here. In fact, as noticed with poor-performing crews, communication needs to be effective and clear to guarantee that the recipients understand and acknowledge what is being said. If that is not the case, it can lead to a performance decrease in the other core competencies (e.g. loss of Situation Awareness).
Reflecting on the results from this analysis, poor-performing crews showed difficulties in the competencies where high-performing crews were strong, especially during high-workload situations. These poor-performing crews completely skipped the planning flight phase which had a high impact during the execution flight phase, shown by the several below average and poor performance comments. Also, the heat-map shows that these crews already have difficulties in Application of Procedures during low-workload situations (flight phases 1 and 2) and in manual flight throughout the scenario. High-performing crews, on the other hand, do not show negative comments for these competencies during these flight phases, yet positive comments were not present since conducting the required procedures here is not considered as above average performance. Despite the predictive asymmetry preventing the prediction of positive performance, it can at least be premised that poor performance for the overall flight can be predicted from low workload situations. All in all the collection of observed competencies are able to draw a clear picture of the differences between high and poor performing crews.
This analysis has identified the competencies that are most helpful in managing unexpected and challenging events, in addition to those competencies whose absence is most likely to lead to poor performance and unsafe outcomes. The desirable competencies identified by the analysis of crew responses to this scenario are: Leadership & teamwork, communication and problem solving & decision making.
4 CONCEPT DEVELOPMENT
4.1. Solution recommendations for procedures
At this time, four concepts have been identified which may be worthwhile to investigate as (partial) solutions for the above problems. The solution to the first problem may consist of defining and implementing an explicit threat-assessment step into existing emergency procedures. A second solution aspect for the first problem may be to introduce a methodology to address and manage uncertainties concerning the aircraft and system states. The solution for the second problem may lie in assisting a process of review and rechecking, possibly by the use of mnemonics. The solution to the third problem may involve redesigning non-normal procedure checklists to include all information required to solve the problem, as opposed to referencing other documents, procedures or people.
4.1.1. Threat assessment solution path
Experiment results show that crews require better guidance on how to systematically assess the situation and aircraft state following a failure, not only by looking at the most present failure, but also looking at other threats related to the failure. This concept builds upon existing failure and threat-management elements that are already part of the pilot training curriculum, although these competencies are trained sparingly (usually part of one-time command training). Examining a good contemporary example of such a failure management process, this procedure can be reduced to the simplified diagram shown in Figure 1.
Figure 1: Failure Management Process extended with threat identification.
In this process, crews collect information from different sources in the flight deck, prioritize the information and determine the status of the aircraft (systems). They must determine what information should be used or responded to right away, what information can be ignored temporarily and what information can be left aside. This process can help prioritize immediate actions, and manage the time-criticality of the situation. The crew then has to anticipate the situation, in the first place by selecting the appropriate procedure to solve or contain the problem. Finally, the crew should consider the various scenarios (return, continue or divert), taking into account the status of the aircraft and environmental conditions, and make a decision on the final plan. This process is designed to increase situation awareness and help crews systematically identify threats and then decide on an action plan, based on the status of the aircraft and environmental conditions.
In the experiments, most crews tended to focus initially on the most obvious problems (engines surging and failing), which often resulted in correctly applying the first few steps of this failure-management process. However, after completing the memory items and procedures for the engine problems, many crews failed to continue and perform an assessment of their aircraft state, and create a suitable long term plan. Instead, the crews terminated the failure-management procedure and decided to land the aircraft (as soon as possible) without evaluating for instance aircraft controllability, structural integrity, aircraft performance or environment (i.e. weather). As a result, some safety margins were exceeded, which in reality would have required re-training.
Regarding the main purpose of the threat-management procedure as a means to understand the full situation and condition of the aircraft and ensure operation within the margins of safety (with respect to for instance attitude, altitude, speeds, [approach] stability, thrust versus drag and energy management). Besides the recommendation to fortify threat-management training, it is proposed that the process is adapted to improve crew awareness during such situations. The proposed adaptation introduced a dedicated threat-identification and prioritization step. This step, although partially present as long term planning in the current process, must occur earlier and thoroughly. The reason for this is that crews become aware of the level of temporal pressure, or lack thereof. By being mindful of the actual temporal pressure, crews may either create more time by prioritizing time critical threats, and/or be able to create more time to further understand their aircraft state, diagnose the problem and plan effective actions. Keeping this in mind, the current aspects of the failure management procedure have been reviewed and it is proposed to improve the process as depicted in Figure 2.
Figure 2: Failure Management Process extended with threat identification.
4.1.2. Managing uncertainty
A second solution concept which may prove to be helpful toward proper assessment of the aircraft state is that of managing uncertainty. Lipshitz & Strauss (1997) investigated the connection between types of uncertainties and various coping mechanisms to reduce the uncertainty. Types of uncertainties may include lack of information, unreliable information, inadequate understanding and undifferentiated alternatives. Types of coping tactics are collecting information and advice, assumption based reasoning, avoiding irreversible action, forestalling and weighing pros and cons. The experiment asked 102 students of the Israeli Defence Force (IDF) to write a case of decision making under uncertainty, from their own experience. These cases were subsequently analysed for the types of uncertainty and the coping mechanisms used. The resulting connection between types of uncertainties and the effective resolution strategies bore a new naturalistic decision making heuristic called RAWFS: Reduction, Assumption-based reasoning, Weighing pros and cons, Forestalling, Suppression. The RAWFS heuristic has been hypothesized as presented in Figure 4. The RAWFS concept may work as a mnemonic of sorts, aimed specifically at resolving uncertain and ambiguous situations.
4.2. Solution recommendations for training
The identifiable connection between crew performance and specific competencies indicates those competencies that are most helpful in managing unexpected and challenging events, in addition to those whose absence is most linked to poor performance and unsafe outcomes. Results indicate that flight crews that are strong in leadership & teamwork, communication, and problem solving & decision making are better prepared to handle challenging and unexpected flight events.
Based on these results, it is recommended to investigate whether it is possible to adapt training in such a way to maintain specific skills in order to cope with unexpected situations. Competency-Based Training is a method that is currently being introduced by some operators and training organisations to fulfil this training need. Such training would adapt training scenarios in simulators in order to create opportunities for frequent exposure to demanding and unexpected situations that cover the most helpful core competencies, preferably in scenarios that are relevant for the operation of modern airliners. Competency-Based Training is expected to supersede conventional event-based training, which has the risk of the development of memorized skills that only work for specific events occurring during training (Casner, Geven & Williams, 2012). It must be stated that despite this study indicating that crew performance can be reflected in specific competencies, it is yet uncertain to what extent training those specific competencies will reflect in better performance for other flight crews not only in a specific scenario but also in other challenging scenarios. Based on the results from this exploratory experimental campaign we propose the following (tentative) recommendations:
• Resilient flight crews need to be strong in the following core competencies:
o leadership & teamwork
o communication
o problem solving & decision making
• Training scenarios should expose flight crews to several challenging and unexpected situations. These must include scenario elements specifically targeted at the above competencies. Additionally, scenarios may be sufficiently diverse to cover most or all of the operational settings and various types of challenges that crews may be exposed to.
• The operational scenarios should be different to avoid memorization, but should cover similar core competencies.
Since the previous experiment contained a single operational scenario for each group of crew, and a limited number of crews, it is necessary to broaden the experimental evaluation for validation of this recommendation. In the next phase it is desirable to test whether flight crews flying different scenarios that focus on the same competencies show the same performance in these competencies.
4.3. Solution recommendations for cockpit design
The above three problems (energy management, engine management & wind-shift awareness) may find their resolution in novel HMI display concepts, which will be detailed in this section. It should be mentioned that these displays are a first move at a novel cockpit design philosophy that focusses on providing crews with positive system state information (i.e. what is still possible, as opposed to what has failed), in order to expedite the process of recovering awareness of the situation and changes by removing uncertainties in aircraft limits and capabilities. In addition to assisting state awareness, this philosophy can also benefit flight planning because crews can more easily generate options are they are presented with possibilities as opposed to restrictions. This new concept will also require some level of training and/or familiarisation, which is not explicitly covered in this section, but may find overlap with any training embedded in the procedural solutions.
4.3.1. Energy display
The flight crews’ awareness of the aircraft’s energy state and their energy management capabilities could possibly be improved by implementing an energy display into the flight deck. The display uses a layout of a vertical situation display, but gives the crew a clear indication of the aircraft’s current kinetic energy status in terms speed deviations relative to the selected (or managed) target speed (in magenta). Furthermore, the display prompts the crew at the time when they have to select new target speeds and aircraft configurations (flaps and gear).
4.3.2. Mitigate aircraft state uncertainty by presenting “positive limitations”
As proposed in the industry’s instructor comments’ analysis means to “internalize external parameters”, that is directly communicating effects of perturbations, were proposed. For this task, the ECAM, PFD and ND displays could be modified in order to directly communicate the perturbations’ effects to the crew in a very simplified way. The most profound change with respect to current displays is the display of “positive limitations”, in other words what controllability, performance and systems are possible, instead of only communicating failures and restrictions. This philosophy should assist crews in more easily mitigating any uncertainty as the new philosophy aims to present information about the pertinent effects of the system state change, as opposed to information about the change itself. The events of the first set of M4G experiments mainly had an impact on the aircraft’s performance. Therefore, it would be beneficial to have the performance implications in terms of the new flight envelope displayed to the crew.
4.3.3. Modified wind indication on the ND
Two possible display modifications are proposed. The idea is to catch the pilots’ attention in wind change situations by temporarily changing the wind arrows size and colour on the ND:
1) When the change rates of wind direction or wind speed are exceeding predefined thresholds, temporarily increase wind arrow’s and colour code it amber.
2) When the wind components for cross and/or tailwind are exceeding the aircraft’s operational limits, temporarily display a magnified red flashing wind arrow.
5 MAN4GEN RECOMMENDATIONS
The operational recommendations that were developed within the Man4Gen project covered three areas: Procedures, Training and Systems.
5.1. Procedural Development
The procedure aims to assist crews in managing surprising, unexpected and diffuse/ambiguous situations where conventional training and published procedures may be insufficient. Some characteristics of such situations:
- These situations are not immediately clear. They require time and attention to understand.
- These situations can develop fast or slowly over time (i.e. dynamic), and often consist of multiple events/failures for which there are not standard checklists.
- These situations test the boundary of the current electronic support (e.g. ECAM) and procedural training that crews get.
- These situations are often untrained and crews are not familiar with them.
The procedure will be a strategy, based on the results from the exploratory experiments, which assists crews in these situations. The term strategy is used explicitly, as the procedure must not allude to a strict checklist or do-list. Rather it must support a crew’s cognitive flexibility, judgement and airmanship to properly assess and manage a complex and ambiguous situation. It must not, however, be so high-level that it becomes too unclear what is meant (this is the risk that a basic mnemonic design has).
This procedure is designed to limit the cognitive load on pilots at the start of the procedure (in consideration of stress and emotional responses), and increases this load as the procedure attempts to increase the cognitive capacity available (by “calming” or focusing the crew). We cannot expect them to do a full diagnosis before executing important memory items. Similarly, we should not waste the capacity they have once they have time and (comparative) peace of mind to diagnose the situation and make a plan.
The strategy encompasses three philosophy concepts:
1) Manage time criticality, such that you have time/calm to...
2) Manage (un)certainty, such that you can effectively...
3) Plan for contingencies and changes
Similar to the purposeful order of the above concepts, the procedure itself is structured into six phases to ascertain sufficient crew and aircraft capacity for each task required:
Phase 1: Stabilize flightpath
Phase 2: Manage immediate threats
Phase 3: Short term planning
Phase 4: Identify situation
Phase 5: Perform appropriate actions
Phase 6: Long term planning
This procedure should by no means contradict manufacturer procedures, checklists and guidelines. As a matter of fact, we are supplementing their published resolution strategies with one for situations they cannot cover. The key aim of this procedure is to provide a strategy for flight crew to deal with unexpected situations for which there is no procedure available. This final procedure concept is designed to support flight crews in both critical acute events as well as (possibly subsequent) complex, ambiguous situations. Within the validation experiments, we intended to evaluate both these aspects of the procedure. Evaluation of the procedure will focus on the overall safety performance of the crew, and compare this to a measure of application of the procedure and its components. These observations will be supplemented with subjective reviews from subject pilots. This evaluation should elicit whether crews trained in and using the procedure perform better, as well, different or perhaps worse than comparable crews without the procedure and its training. In addition, the evaluation will indicate which aspects of the procedure are associated with a performance delta, and how the performance changes.
The procedure will also be evaluated in conjunction with the display improvements in a second experiment at DLR. This experiment has been setup to evaluate the collective design philosophy shared between both solution developments. A similar procedure evaluation will be done, and a comparison with the procedure-only experiment results may indicate whether the addition of a display affects how crews use the procedure, and the performance they achieve.
Both experimental results will indicate what a final tuned version of the procedure might be, by retaining the effective components, and adapting or removing others. Also the training and interface with display improvements will be reviewed to indicate how procedure effectiveness may be increased.
5.2. Training Development
A possible solution identified to improve flight training would be to train competencies instead of pre-described flight maneuvers, many of those being non-technical. Such competency transfer between scenarios could create more resilient flight crews that are more prepared to handle operational events instead of the scripted and sometimes predictable scenarios currently used in training. A way to do this would be the implementation of Evidence-Based Training (EBT) concepts. To support this training methodology, this deliverable presents a scenario development method to generate scenarios for competency-based training.
5.2.1. Desktop evaluation
The objective of the evaluation phase is to assess the flight crew competency level. Therefore, the scenario designed for this phase should be general so that all pilot core competencies can be evaluated in order to identify the main training needs. This is opposed to the training phase scenarios that should trigger the competencies lacking in certain flight crews. The evaluation phase scenarios are then more general because they are used to evaluate the flight crew proficiency in all competencies. Although the evaluation phase could be done with a general flight scenario in a full-flight simulator, in the Man4Gen project we are trying to evaluate these competencies with a non-flight deck related desktop simulation.
5.2.2. Training Phase
The main objective of the training phase is to improve the flight crew proficiency on the pilot core competencies where more difficulties have been previously identified. Therefore, for optimal training the scenarios should be personalized to the flight crew. For the Man4Gen scenarios, we used the process developed earlier in the project to generate the experimental scenarios. This process focused in the problem statement themes, rather than competencies, because here we were not developing scenarios to trigger the needs of an individual flight crew. The goal of the Man4Gen experiments was not to train the needs of a flight crew, as it should be during the training phase, but to investigate whether competencies transfer between different scenarios.
The scenario development method described in this document is only effective if the competencies transfer between scenarios. This is, if the competency proficiency of a certain flight crew is similar between different flight scenarios that require similar competencies in order to be handled successfully. This competency transfer was addressed in the second experimental campaign of the Man4Gen project. If the results from these deliverables are positive, it means that we can observe competency proficiency from scenarios that require certain competencies. This would also be indicative that this scenario development method is appropriate since it focuses the scenario development around specific key competencies that are required by certain flight crews. However, we still need to show that exposure to certain competencies will indeed increase flight crew proficiency on them. That was outside the scope of the Man4Gen experiments.
Additionally, if the competency transfer works not only for the flight scenarios, but also for the desktop exercise, it means that this method can be used in the evaluation phase described in the EBT manual. The potential savings here could be significant if full flight simulator usage can be limited or even scrapped in the evaluation phase. Also, if competencies can be trained by exposure, flight crew training becomes more effective because it can focus training on the flight crew specific difficulties which will lead to more resilient flight crews.
5.3. Systems Technology Development
The improved cockpit system development thus focusses on supporting the crew’s capabilities to perform the modified procedure. The new systems provides six additional pages on the lower ECAM that summarize the current situation in terms of controllability, performance, endurance and automation as well as the impact on the upcoming flight phases. This information shall enhance situation awareness (SA) and support the short-term as well as the long-term planning of the crew as it is intended by the modified procedure. Important principles in designing this modified display are that it does not conflict with existing procedures, does not add to the design complexity, does not require significant additional certification, and provides an ancillary level of support to flight crews.
The Risk Information System consists of six additional pages located on the lower ECAM display; besides the regular system synopsis pages and the status page. These six pages provide collective interpretations of the effects of an augmented aircraft state, based on information available to the aircraft information systems, notable the flight warning system. These interpretations and their structure should serve to support flights crews in situation analysis and awareness, the subsequent decision making process, when confronted with an uncertain aircraft state and uncertainty in how to proceed. These six new pages are accessible via a set of buttons on the transponder (XPDR) panel.
The cockpit system modifications implemented in the project nclude additional pages for the lower ECAM of the Airbus A320. These pages shall provide an overview of the situation on an overall aircraft level and not on a system level. The design of the pages is still preliminary and should be considered as first prototype to assess the usefulness of such a display concept. Hence the task of the simulator experiments is to perform a proof of concept. The aim of the validation experiments is to analyse what impact the new Risk Information System has on the overall crew performance and especially on the crew’s decision making process during the scenario.
6 CONCLUSIONS OF THE PROJECT
The conclusions of the project have been presented in the form of deliverable reports that will be made available to the aviation and research stakeholders.
6.1. Operational Conclusions
The primary conclusion of the project was that there is a benefit in examining the preparation of flight crew to deal with unexpected events. During this project it became clear to the researchers that there are areas where improvements can be made in order to work towards the desired reduction in accident rates. By including the three different areas – training, procedures and systems – the project team has covered recommendations for both shorter and medium term concepts. The aim of the project was to deliver recommendations that could be applied by industry stakeholders such as regulators, manufacturers and operators. The final results of the project take the form of material that can be used by the stakeholders to support development and changes in training for example, or can be used as a foundation to re-examine existing procedures and display designs.
The findings of the operational work are that there are improvements that can be made to current practices – particularly with reference to training and operational practices in the shorter term – that can address the potential for problems in modern airline operations. It has been identified that it is possible to support and prepare flight crew to deal with unexpected events, potentially including in the form of additional systems support in the longer term. It is hoped that the results of the Man4Gen research can be picked up by the industry in continuing work to further improve the safety of operations, thus preparing the operation to handle unexpected events.
- D6.1 Report on Operational Procedures recommendations: This report contains the recommendations for airlines and industry for operational procedures improvements as result of the evaluations at NLR’s GRACE simulator.
- D6.2 Report on Training recommendations: This report contains the recommendations for training improvement based on the experiment results from the training experiments in the GTA simulator.
- D6.3 Report on Systems recommendations: This report contains the recommendations for developments in aircraft and cockpit systems technology.
- D6.4 Final Report of Operational Guidelines: This report brings together the recommendations and findings of the three areas of research for the use of aviation stakeholders such as regulators, airlines and manufacturers.
6.2. Research Conclusions
In addition to the operational conclusions that the project has delivered, the research team have developed and investigated different research methodologies that can be applied in the fields of cognitive science, neuroergonomics, and cognitive systems engineering. Within cognitive science the project team focused on the perspective of “situation(al) awareness”, and within cognitive systems engineering the sensemaking perspective was applied. Methodological considerations from each perspective were discussed. A triangulation of methods has been carried out in the project, including the two research strands and the industry based assessment. Next to conventional assessment techniques, methods were applied that are new and innovative, especially in the field of aviation (such as fMRI, social SA tools, and ECOM). The combination of the analyses and outcomes of these techniques (fMRI, [social] SA and sensemaking) resulted in a mutual validation of the applied approaches. Therefore, we are now better able to test the interfacing between man and machine (MMI), the required levels of training, and associated factors in a dynamic test and simulation environment, which allows developing new MMI configurations. The results from SA and SM have individually been linked to the industry assessment and part of the results from the two research perspectives was combined to demonstrate how different methods can complement and validate each other. The overall results from the Man4Gen experiments show that there is a great variation in how the crews understand, prioritise, manage uncertainties and take action following the unexpected events introduced in the scenarios. Operational recommendations and the findings that support them on how to improve crew abilities are summarised in the final report D6.7. The project has opened up a new domain in research options and techniques that can be instrumental for cockpit design, layout and training. The findings further raise several important questions regarding challenges and possibilities for pilots to maintain control in surprise situation.
- D6.5 Report on Situational Awareness research: This report summarises the main conclusions regarding Neuro-ergonomics and Situational Awareness in 4th generation aircraft cockpits based on the evaluations of the experiments.
- D6.6 Report on Resilience research: This report includes the findings from Man4Gen experiments in the context of the resilience engineering sense-making research. The research results should be described as part of the application to the operational requirements for operating 4th generation aircraft.
- D6.7 Final report of research methods and results: This deliverable brings together the research findings throughout the project regarding methods for identifying, analysing and assessing sensemaking and situation awareness in operational environments. The report includes the outcome of the experiments in the research simulators and the application of fMRI technology in research of pilot flying and pilot monitoring tasks.
Potential Impact:
7 POTENTIAL IMPACT OF THE PROJECT
The research methodological work carried out within the project has developed methods and scientific results that have an immediate application within the research fields: cognitive science, neuroergonomics and cognitive systems engineering. For example the methods developed in this project are already being picked up by other EU research projects for application and further development to measure and report human performance in similar aviation experiments.
The results of the project have a potential impact on the operational sphere of the aviation industry. The results were developed and presented with the primary aviation industry stakeholders in mind. The external advisors of the project made up a combination of expertise from regulators, manufacturers and operators, who all indicated that the research results of the project would be a useful contribution to existing developments within the industry.
• Training: The aviation industry is currently reviewing the training methodologies that are used, Competency Based Training and Evidence Based Training are methodologies that are increasingly being applied by operators and manufacturers. The regulators are also adopting these methodologies within regulations and advisory material. The results of the Man4Gen project demonstrate how competencies can be designed into training scenarios, how they can be assessed in training, and how alternative training tools can contribute to the training. Through the broader application of competency based training the training of flight crew can lead to competency development for a variety of situations, and for unexpected situations.
• Procedures: The potential advantage of a strategy that can be applied in the form of a procedure or guidance for flight crew is that this can be adopted by operators with local training. This is therefore a conclusion that can be implemented in the short-term. The impact of the research that was conducted in this strand of the project is to inform the ways that flight crew operate and function in response to unexpected events. Additionally it was noted by the researchers that the procedure developed in the project may be useful as a training tool, to promote a strategic approach to dealing with unexpected events. These results of the Man4Gen project have been presented and discussed with aviation professionals and there is interest in taking the concepts of the project for further development and application.
• Systems: The concept that was investigated in the systems research strand – the risk management display – is closely aligned to the strategy that was developed in the procedural concept within the project. The display promoted a discussion within the aviation industry experts in the advisory board as to the different ways to communicate the information with the flight crew about the risks and threats to the aircraft. This research strand presented a concept that can be picked up for future research within other projects.
The partners of the project have taken the results and are investigating ways to further disseminate and exploit the results of the project beyond the activities that were already carried out within the project timeframe.
8 EXPLOITATION AND DISSEMINATION
8.1. Man4Gen website
A web site has been put in place with information about the Man4Gen project to inform the general public. The project website describes the consortium, project goals, purpose and objectives, schedule and agenda, upcoming events, results and reports, links to articles, links to social media, online discussion forum, information for the general public. The website address is: www.man4gen.eu.
8.2. Man4Gen video
At the end of the project a video has been produced that explains the main research questions, the experimental approach of Man4Gen and the main conclusion of the project. It is presented in a visually attractive way with a narrative explanation. The video has be placed at youtube: https://www.youtube.com/watch?v=mVLuck49Vk0
8.3. Dissemination
Several presentations and papers about the research have been disseminated. An overview is presented in the table below:
Event Type Date Audience
European Association of Aviation Psychology Papers and Presentations September 2012 Aviation Psychology researchers and aviation professionals
2nd Annual CogSci "LORENZ-Lecture" Poster presentation 7 November 2012 Rectorate ViU, scientists in the fields of cognitive science, animal cognition, psychology, biology and students
5th symposium on Resilience Engineering Paper and presentation 25-27 June 2013 researchers/practitioners within the resilience engineering community
Annual Conference on Neuroeconomics: Decision Making and the Brain Poster presentation and abstract 26-29 September 2013 Scientists
3rd Annual CogSci "Bühler-Lecture" Poster presentation 20-21 January 2014 Rectorate ViU, scientists in the fields of cognitive science, animal cognition, psychology, biology and students
Psychoneuroendocrinology Journal Journal article January 2014 Scientists and researchers
ICAO Loss of Control – In-flight symposium Paper and presentation May 2014 Aviation professionals, International policy makers (Government and Industry)
EASA Loss of Control – In Flight workshop Paper and presentation September 2014 European regulators, aviation professionals
Aviation Psychology and Applied Human Factors Journal article December 2014 Aviation psychologists, Human factors researchers and practictioners
Organisation for Human Brain Mapping Poster and Presentation June 2015 Neuroergonomics, researchers, scientists and medical professionals
NeuroImage journal Journal article July 2015 Medical scientists and researchers
AeroSafetyWorld Publication article July 2015 Aviation industry members
International Society of Aviation Psychologists Paper and presentation June 2015 Aviation psychology researchers, aviation professionals.
European Association of Aviation Psychology Papers and Presentations September 2014 Aviation Psychology researchers and aviation professionals
6th symposium on Resilience Engineering Paper and presentation June 2015 researchers/practitioners within the resilience engineering community
Aerodays Presentation October 2015 European Commission, Industry, Research and Politicians
AIAA SciTech Forum and Exposition Paper and presentation January 2016 Aviation professionals and researchers
In addition to these dissemination activities, the results of the project are currently being written up in the form of papers and journal articles that will be submitted by the project partners in the coming months.
8.4. Exploitation of Results
The results of the Man4Gen project have been discussed and picked up by a number of organisations in the aviation industry. There is interest in the project, and project partners have received questions and expressions of interest in the results.
From the publication of the results from the exploratory experiments during the project a number of international airlines made contact with the project team to find out more about the work that was being carried out. These airlines have been in contact with the project team to investigate how the concepts investigated in the project can be applied within their training programmes.
Additionally, a study is being carried out for EASA where the results from the Man4Gen project are used as the starting point for research into startle effect management for flight crew. The results of the project are being directly picked up and applied further to ensure that the research work of the Man4Gen project contributes to the recommendations being developed by EASA.
The expectation of the project partners is that further organisations, operators, airlines, manufacturers, regulators and researchers will take the concepts and results of the Man4Gen project and apply them further in the future.
9 EXTERNAL ADVISORY BOARD
The aim of the EAB will develop as the project continues. As a whole, the EAB is intended to scrutinise the work that the consortium is carrying out, and ensure that our work is applicable outside the project. For that reason the EAB includes advisors from several important areas of the aviation and (scientific) research communities. There are 25 international subject matter experts from the aviation industry, airliners, training providers, regulation authorities, and universities in the EAB.
Two meetings with the External Advisory Board (EAB) have been organised during this reporting period. The first meeting was organised after the first results from the literature study and automation analysis. These results were presented and the problem statement has been discussed with the EAB members. As outcome of the discussion the problem statement has been adjusted and some useful recommendations for the direction of the research methods and the design of the experiments have been received. During the second meeting the design of and the scenario for the simulator experiments at DLR and NLR have been presented and discussed. Some useful comments were received to sharpen the experiment aims.
EAB meetings:
Location Date Number of EAB members
NLR, Amsterdam 21 March 2013 7
Swiss Aviation Training, Zurich 18 December 2013 5
DLR, Oberpfaffenhofen 9 December 2014 3
Boeing, Seattle 6-7 and 18 March 2015 2
London 6-7 and 18 March 2015 1
Vienna University, Vienna 6 November 2015 2
EASA, Cologne 10 December 2015 6
List of Websites:
Website: http://man4gen.eu/
Email: man4gen@nlr.nl
Coordinators Email: arjan.lemmers@nlr.nl joris.field@nlr.nl
