Periodic Reporting for period 4 - MOTO (the embodied reMOte Tower)
Reporting period: 2017-12-01 to 2018-05-31
MOTO address two major research opportunities:
• The first one is to consider the role of all the human senses for tower operations. The approach of Embodied Cognition could be applied to achieve a full understanding on the use of all senses for controllers, besides the visual one.
• A second opportunity is to build new human-system interaction concepts on the understanding of embodied aspects of ATM Human Performance. The final goal is to enhance human performance by augmenting multisensory stimuli – including but not limited to the already overloaded visual channel - without increasing the ATCO workload.
The overall objective of the project is to identify the key multimodal stimuli required on RTO to enhance the sense of Presence experienced by ATCOs.
The state-of-the-art of cognitive neuroscience has shown that for humans to really believe to be there – and act accordingly - the integration of multisensory information is more important than the high fidelity of the visual channel in isolation.
MOTO project detailed objectives are the following:
• Objective 1: assessment of the role of Embodied Cognition in control tower operations (i.e. the role of multimodal acquisition of information in current control tower operations)
• Objective 2: definition of user requirements for a multimodal Remote Tower, to reconstruct multimodal perception in a remote tower simulation platform ( enclosing the development of augmented multimodal prototypes exploiting visual channel and sound spatialisation)
• Objective 3: definition of brain-physiological indexes, customized for Remote Tower operations, to monitor aspects of Human Performance like: workload, situation awareness,sense of presence
• Objective 4: Validate the above results in realistic ATM operational conditions through simulation facilities.
Current target scenarios include:Single remote tower and multiple remote tower scenarios
The first stream of work performed in MOTO, ‘Realistic Remote Tower’, was explored through the virtual reality platform using the HTC Vive Head Mounted Display. The outcomes of this stream of work showed that ATCOs performance could be enhanced by integrating visual channel with one sensory modality at a time (either audio or haptic). In line with this finding, workload assessment showed that by integrating only Audio channel to the Video one induced a decreasing in ATCO’ experienced workload.
However, attention should pay to not “overload” ATCOs providing visual and both Audio and Vibrotactile stimuli. Results seem to suggest that the full combination may induce a degradation of performance, likely due to distracting stimuli contribution that is higher than their informative content. Finally, results suggest that in the ‘Visual + Auditory’ condition ATCOs sense of presence was improved, consistently with performance and workload improvement. These results are detailed in D4.2 (First Validation Report).
The second stream of work, ‘Augmented Remote Tower’, exploited the possibilities offered by the multimodal stimuli, translating them into a technologies to enhance human performance, for instance by augmenting perception.
A set of augmented multimodal Remote Tower concepts (TRL2) were designed and a subset were actually tested during the second MOTO validation providing an initial insight of the potential benefits in terms of performance. The augmented multimodal concepts validated in the second MOTO validation showed performance advantage using a spatialized auditory warning: ATCO reported faster reaction time in detecting abnormal situation such as an unauthorized clearance in the apron or an on-going runway incursion (in this latter a vibration feedback was reinforced by an auditory spatialized warning). However, both subjective and neurophysiological results suggested an increase of experienced workload if augmented solutions were activated. Such perception could also be attributed to the fact that ATCOs needed a longer familiarization process with a more enriched sensorial environment for tower operations, since they are usually used to work only with information provided by the visual sensory channel. Further studies are then recommended to verify that performance benefits could be extended to the whole set of proposed solutions. A set of exploratory concepts for multimodal multiple Remote Tower featured by augmented interactions have been also defined at lower TRL1.
Another outcome of the project was the neurophysiological classifiers developed to monitor cognitive workload and sense of presence. These classifiers were used in both validation experiments to measure the ATCOs mental states during the execution of air traffic control tasks. The Sense of Presence Index was explored more into detailed during the first validation. This index is intended as an objective measure of the sense of presence in an immersive remote environment (in this case also virtual). The index was estimated from the possible modulations of the GSR responses induced by an emotionally salient virtual context in RTO environment, and results from GSR and EEG were consistent along the whole activity. Some limitations have been identified during the validatoins and they need to be carefully considered in the interpretation of the results.
Hovewer, MOTO outcomes in terms of requirements and solutions for multimodal Remote Towers represents a first step to integrate “realistic” multimodal feedback in current industrial solutions and to drive further R&D activities.
• Use of multimodal technologies in the remote tower environment, by partially off-loading the visual channel and relying on other channels (i.e. audio or haptic feedbacks) for monitoring out-of-the-current-view airports.
• Combination of neurophysiological measurements to estimate operator mental states and sense of presence along the execution of tasks in a simulated virtual environment.
• Potential support of the augmented multimodal technologies to enhance ATCO situation awareness and performance during remote towers operations.
Particularly, MOTO explored the role of the multimodal stimuli in the remote tower operations. Neurophysiological index has been used to estimate controller workload and to investigate the relevance of the sense of presence in the remote operations. The enhanced Sense of presence is a key requirement to preserve human performance as in real operations, in order to facilitate the transfer of real-world knowledge and skills into the RTO.
Finally MOTOprovided insights on the potentiality of the use of multimodal augmented technologies in RTO, supporting ATCO performance and situation awareness thought auditory and vibrotactile feedback.