We performed two complementary studies investigating a) audio-visual and b) visuo-haptic interaction in the human brain.
First, I employed a well-established visual object categorization task, in which early sensory evidence and post-sensory decision evidence can be properly dissociated based on electroencephalography (EEG) recordings. Specifically, using a face-vs-car categorization task, we have previously profiled two temporally distinct neural components that discriminate between the two stimulus categories: an early component, appearing ~170–200 ms poststimulus onset, and a late component, seen after 300–400 ms following the stimulus presentation. We hypothesized that using AV information to discriminate complex object categories—rather than more primitive visual features—would lead primarily to enhancements in the Late, as opposed to the Early, component, consistent with a post-sensory account. Importantly, by combining single-trial modelling and EEG data, we exploited the trial-by-trial variability in the strength of the Early and Late neural components in a neurally informed DDM to derive mechanistic insights into the specific role of these representations in decision-making with AV information. In short, we demonstrated in this work that multisensory behavioral improvements in accuracy arise from enhancements in the quality of post-sensory, rather than early sensory, decision evidence, consistent with the emergence of multisensory information in higher-order brain networks.
Second, I employed an active sensing paradigm coupled with neuroimaging, multivariate analysis and computational modeling to probe how the human brain actively samples multisensory information to make perceptual judgments. Participants of both sexes actively sensed to discriminate two texture stimuli using visual (V) or haptic (H) information or the two sensory cues together (VH). We showed that the simultaneous exploration of different modalities (multi-sensing) enhances neural encoding of active sensing movements. To strengthen the mechanistic interpretation of this result we exploit an informed drift diffusion model to link single-trial perceptual choice with the neural encoding of active sensing, in a clearly interpretable mechanistic context of decision making. This modeling approach demonstrated that the neural encoding of active sensing modulates the decision evidence regardless of the sensing modality and that multi-sensing results in significantly faster evidence accumulation. Then, to identify crossmodal interactions in the human brain and characterise their functional roles in decision-making behavior, we implemented a novel information-theoretic analysis, namely Partial Information Decomposition (PID). This revealed an interaction of the two unisensory representations in the human brain, in particular over motor and somatosensory cortex. Crucially, this cross-modal representational interaction correlates with multisensory performance, thus constituting a putative neural mechanism for forging active multisensory perception.
The above findings have been presented in conferences:
Society for Neuroscience Meeting
Symposium on Biology of Decision-Making
as we all as invited talks and seminars:
Behavioural Data Analytics Seminar Series, Imperial College London, UK
Centre for Mathematical Neuroscience, City University, London, UK
Unit for visually-impaired children, Istituto Italiano di tecnologia, Genoa, Italy
MultiTime Lab, Panteion University, Athens, Greece
I have also been invited to present this work to the general audience in a a)TEDx event at Glasgow Caledonian University (October 2021) and b) a Pint of Science event at the University of Leeds (in 2022).
