To achieve our objectives we conducted two streams of investigation: at the intracranial scale to access the neuronal mechanism of multisensory integration and at the whole brain level to get a global understanding of the interplay between multisensory integration, memory and decision making processes.
Our first approach relied on the rare chance to record intracranial neuronal oscillatory activity in epileptic patients while they were undergoing a strict and independent clinical procedure. During the experiment, participants were exposed to illusory contours accompanied or not by a sound. Illusory contours are visual illusions that evoke the perception of a shape without the presence of edges (e.g. Kanizsa triangle). Our analysis was directed at oscillatory activity to investigate the modulation of functional connectivity between brain regions. To precisely localize the intracranial electrodes in the brain, we developed a semi-automatized analysis pipeline utilizing multimodal imaging (i.e. CT-scan and MRI). We report two main findings. First, the phase of ongoing oscillations in auditory cortex is strongly re-aligned by illusory contours as to compare to non-illusory contours. That is, auditory region was informed about the visual binding of illusory contours into a shape. Second, when the illusory contours was associated with a sound, ongoing oscillations in auditory and visual regions were greatly synchronized. This communication between distant neuronal populations represents the encoding of a multimodal object: the sound was associated with the shape formed by the illusory contours.
Our second approach consisted in performing a study in healthy participants using EEG. While their brain activity was recorded, the participants were exposed to dynamic sequence of stimuli. Each sequence consisted in a cacophony of audio-visual noise, where they had to detect, or to categorize, an unpredictable target cue presented either in the auditory domain, in the visual domain or both. To investigate the behavioral benefit of multisensory integration, we utilized computational models of decision making. The idea of a computational model is to map different cognitive processes to different psychologically meaningful parameters. This approach revealed that to explain the faster multisensory responses, there were two key parameters: one corresponding to sensory encoding process, the other to the process of decision formation. Next, to analyze EEG signal, we utilized a machine learning method. That is, we trained an algorithm to decode experimental conditions based on the recorded brain activity. First, using the decoding from unisensory conditions, we mapped cognitive processes over time. Then we performed a cross-condition decoding, generalized in time, to decode the multisensory condition. The results demonstrated that multisensory integration accelerated the brain dynamic during both the encoding of sensory information and when the decision was formed before participant’s response. In the two tasks, the result from electrophysiological data was coherent with our behavioral modeling.
In summary, our research indicated that multisensory integration is pervasive in human brain and completes different processes along the cortical hierarchy. First, we showed that to build multimodal association, sensory regions exchange information through the synchronization of their oscillatory activity. Second, we evidenced that after sensory encoding; multisensory integration implies associative regions to mediate the formation of a decision, in link with representations stored in memory.
