Final Report Summary - ISMINO (Facilitation and Inhibition of Multisensory Integration via Neural Oscillations)
In our everyday interactions with the environment, the brain is bombarded with information from multiple senses: sometimes a common event (or source) generates both signals, but other times two separate independent events (or sources) generate the distinct signals. For instance, in the pub, we should integrate the facial movements of the speaker we are listening to with the speech he produces whilst ignoring other voices. Put another way, the brain is challenged to decide when to integrate sensory signals that emanate from a common event, but segregate those from different events. To infer whether sensory signals are generated by a common event and should be bound into a coherent percept, the brain needs to combine multiple ‘bottom-up’ congruency relations with ‘top-down’ prior knowledge. How does the brain put together multiple ‘bottom-up’ correspondences such as spatial congruence, temporal (a)synchrony, or motion congruence? How do ‘top-down’ prior expectations influence whether and how we bind sensory signals into a unified percept? Despite its importance for many daily situations, little has been known about how the brain arbitrates between integration and segregation of sensory information in such an effortless manner, when so many factors must come together to create quickly a percept, especially when cues are in conflict with each other.
Dr Zumer’s project has built on previous work in several ways. The project was divided primarily into two main streams: one stream first established the neural mechanisms associated with varying levels of temporal asynchrony, as one type of ‘bottom up’ cue for multisensory integration; the second stream elaborated by combining bottom-up and top-down cues, with an examination of both the behavioural effects and neural mechanisms associated.
Specific to the first stream, Dr Zumer revealed novel distinct neural mechanisms that process multisensory temporal correspondence (i.e. asynchrony) for multisensory integration. Importantly, the brain should integrate sensory signals only when they co-occur within a ‘temporal integration window’ and are hence likely to originate from a common source. This study examining both psychophysics and electroencephalography (EEG) data unravelled a multitude of neural interactions governed by different temporal constraints: interactions were confined to a behaviourally-relevant ‘temporal integration window’ for evoked response potentials, were specific for one particular asynchrony for inter-trial coherence, and extended beyond the behavioural ‘temporal integration window’ for induced low frequency (4-20 Hz) power. This diversity of temporal profiles demonstrates that distinct neural mechanisms mediate a cascade of multisensory integration processes. A part of these findings is in line with established hypotheses (e.g. phase-resetting as a means of enabling multisensory integration). These findings go beyond previous work to relate, across multisensory asynchronies and across neural mechanistic features (e.g. neural activity phase-locked and non-phase-locked to stimulus timing), the specific multisensory interactive responses in the brain. This work has been presented by Dr Zumer: as a poster at both national and international conferences, orally during an invited talk at another university, and as a detailed manuscript made public as a preprint on bioRxiv (and submitted for peer-review publication).
The second stream is divided into (a) development of the paradigm and the behavioural/psychophysics results and (b) the neural mechanisms underlying the behavioural effects. Critically, different types of multisensory congruency can be computed by the brain at different post-stimulus processing times. For instance, in perception of apparent audiovisual motion, the synchrony of audiovisual stimulus onsets will be immediately available, whilst the audiovisual motion congruency requires initial temporal integration of the two subsequent flashes/beeps. Dr Zumer has developed a complex paradigm involving apparent motion in an audiovisual context, titrated very carefully to achieve the desired percepts of audiovisual motion and audiovisual correspondence. She built in top-down prior expectations for each presentation of the audiovisual signals, again titrated carefully so that the manipulations on expectations produced the desired manipulation final percept. Now that the precise physical details of the stimulation paradigm have been established through extensive behavioural testing amongst many participants, this paradigm now may be used both by Dr Zumer and Prof. Noppeney for future work, as well as by the research field at large. In addition to being useful for the multisensory field, it also contributes to the sub-field of psychology focussed on apparent motion. Participants most of the time bound the auditory and visual information even when motion was incongruent; this ‘mistake’ was counteracted when participants expected the audiovisual signals to move incongruently. This data has been presented orally at a conference during the last year of the Fellow’s grant and the full manuscript is under preparation.
Third, Dr Zumer utilised her expertise in the brain imaging technique MEG to query many facets of the neural dynamics that compute the transition from the complex input signals and task instructions to the final perceptual decision and behavioural response. The complex task design modulated neural responses in line with proposed hypotheses (e.g. in line with the hypothesis that alpha (10 Hz) power provides active, task-related inhibition of specific brain regions). The expectation cue (i.e. ‘top-down’ factor) manipulated alpha power in a spatially-specific manner in line with the hypothesis. Furthermore, the final behavioural output (perceptual decision) was correlated with beta (16-20 Hz) power on a trial-by-trial basis for the confusing ‘incongruent’ trials, highlighting the relevance of this neural marker for difficult decisions. This data has been presented orally at a conference during the last year of the Fellow’s grant and the full manuscript is under preparation.
Taking these studies altogether, Dr Zumer has advanced the field in understanding how distinct neural mechanisms, specific to stimulus-phase-locked and oscillatory power related changes. She has taken leading hypotheses in related fields (as mentioned in the proposal) and linked them to novel paradigms and novel contexts. This international fellowship has allowed Dr Zumer to boost her visibility in the EEG/MEG field and develop a reputation in the multisensory field.
Dr Zumer’s project has built on previous work in several ways. The project was divided primarily into two main streams: one stream first established the neural mechanisms associated with varying levels of temporal asynchrony, as one type of ‘bottom up’ cue for multisensory integration; the second stream elaborated by combining bottom-up and top-down cues, with an examination of both the behavioural effects and neural mechanisms associated.
Specific to the first stream, Dr Zumer revealed novel distinct neural mechanisms that process multisensory temporal correspondence (i.e. asynchrony) for multisensory integration. Importantly, the brain should integrate sensory signals only when they co-occur within a ‘temporal integration window’ and are hence likely to originate from a common source. This study examining both psychophysics and electroencephalography (EEG) data unravelled a multitude of neural interactions governed by different temporal constraints: interactions were confined to a behaviourally-relevant ‘temporal integration window’ for evoked response potentials, were specific for one particular asynchrony for inter-trial coherence, and extended beyond the behavioural ‘temporal integration window’ for induced low frequency (4-20 Hz) power. This diversity of temporal profiles demonstrates that distinct neural mechanisms mediate a cascade of multisensory integration processes. A part of these findings is in line with established hypotheses (e.g. phase-resetting as a means of enabling multisensory integration). These findings go beyond previous work to relate, across multisensory asynchronies and across neural mechanistic features (e.g. neural activity phase-locked and non-phase-locked to stimulus timing), the specific multisensory interactive responses in the brain. This work has been presented by Dr Zumer: as a poster at both national and international conferences, orally during an invited talk at another university, and as a detailed manuscript made public as a preprint on bioRxiv (and submitted for peer-review publication).
The second stream is divided into (a) development of the paradigm and the behavioural/psychophysics results and (b) the neural mechanisms underlying the behavioural effects. Critically, different types of multisensory congruency can be computed by the brain at different post-stimulus processing times. For instance, in perception of apparent audiovisual motion, the synchrony of audiovisual stimulus onsets will be immediately available, whilst the audiovisual motion congruency requires initial temporal integration of the two subsequent flashes/beeps. Dr Zumer has developed a complex paradigm involving apparent motion in an audiovisual context, titrated very carefully to achieve the desired percepts of audiovisual motion and audiovisual correspondence. She built in top-down prior expectations for each presentation of the audiovisual signals, again titrated carefully so that the manipulations on expectations produced the desired manipulation final percept. Now that the precise physical details of the stimulation paradigm have been established through extensive behavioural testing amongst many participants, this paradigm now may be used both by Dr Zumer and Prof. Noppeney for future work, as well as by the research field at large. In addition to being useful for the multisensory field, it also contributes to the sub-field of psychology focussed on apparent motion. Participants most of the time bound the auditory and visual information even when motion was incongruent; this ‘mistake’ was counteracted when participants expected the audiovisual signals to move incongruently. This data has been presented orally at a conference during the last year of the Fellow’s grant and the full manuscript is under preparation.
Third, Dr Zumer utilised her expertise in the brain imaging technique MEG to query many facets of the neural dynamics that compute the transition from the complex input signals and task instructions to the final perceptual decision and behavioural response. The complex task design modulated neural responses in line with proposed hypotheses (e.g. in line with the hypothesis that alpha (10 Hz) power provides active, task-related inhibition of specific brain regions). The expectation cue (i.e. ‘top-down’ factor) manipulated alpha power in a spatially-specific manner in line with the hypothesis. Furthermore, the final behavioural output (perceptual decision) was correlated with beta (16-20 Hz) power on a trial-by-trial basis for the confusing ‘incongruent’ trials, highlighting the relevance of this neural marker for difficult decisions. This data has been presented orally at a conference during the last year of the Fellow’s grant and the full manuscript is under preparation.
Taking these studies altogether, Dr Zumer has advanced the field in understanding how distinct neural mechanisms, specific to stimulus-phase-locked and oscillatory power related changes. She has taken leading hypotheses in related fields (as mentioned in the proposal) and linked them to novel paradigms and novel contexts. This international fellowship has allowed Dr Zumer to boost her visibility in the EEG/MEG field and develop a reputation in the multisensory field.