Final Report Summary - MODULAREXPERIENCE (How the modularization of the mind unfolds in the brain)
This project aimed to understand how the structure of the mind is grounded in the underlying organization of the brain in terms of focal modules and more widespread maps. We demonstrated and characterized the relationships between neural maps and modules. In addition, we showed how a combination of multiple neural maps plus experience might underlie the formation of modules from pre-existing maps. Finally, we showed how neural selectivity in sensory systems can still develop despite the absence of normal sensory input. These findings help us understand how the human brain achieves to support the amazing capacities and structure of the human mind.
More specifically, in Work Package 1 we analyzed properties of module-like areas and the systems they belong to. We characterized the visual word representations in the visual word form area, face representations in and beyond the fusiform face area, and magnitude representations in the intraparietal sulcus (Baeck et al., 2015) (Goesaert et al., 2013) (Bulthé et al., 2015). We combined the multi-voxel pattern analysis (MVPA) methods which are central in this proposal with methods to analyze the functional connectivity between regions. An important test case for this combination was Boets, Op de Beeck, et al. (2013), in which we showed that in the context or reading difficulties there is evidence for intact neural representations, as assessed through MVPA, combined with impaired connectivity.
In Work Package 2, we analyzed non-modular maps in sighted individuals and their relationship to module-like structure in both brain and behavior. This work package hypothesized that there might be a large-scale map of shape selectivity which might be an anchor point for the stronger category selectivity. We implemented a study (Bracci & Op de Beeck, 2016) in which we dissociated the two factors, shape and category. We also provided a computational analysis through deep-learning convolutional networks that gave insights into how shape and category selectivity might be related (Kubilius et al., 2016). Finally, we brought all the evidence together, and presented a theoretical analysis of the available data, arguing in favor of a feature-based categorical coding (Bracci et al., 2017, Neuropsychologia). This explains the role of features in the representation of categories.
In Work Package 3, we characterized non-modular maps in congenitally blind participants and category selectivity. After first investigating the properties of non-modular maps and modular category selectivity in the ventral occipitotemporal cortex of low-vision patients (Goesaert et al., 2014), we set up a large study to obtain high-quality data in congenitally blind participants about how their ventral occipitotemporal cortex responds to auditory stimulation related to particular object categories (van den Hurk et al., 2017, PNAS). We showed that the activity pattern in the blind for sounds of faces, bodies, objects, and scenes could be used to predict which visual stimulus was associated with an activity pattern from a sighted control participant. These findings show the extent to which the overall organization of ‘visual’ ventral occipitotemporal cortex can develop despite the absence of any visual input.
In Work Package 4, we piloted an experiment in which we determine the selectivity for number-related words and symbols in adults (Peters et al., 2015), and afterwards also in children (Peters et al. 2016). The results confirm the hierarchical processing of words and symbols in the brain, as well as the prediction that children generally activate similar brain networks as adults do.
In Work Package 5, we investigated how experience affects non-modular maps and the strength of modularity. At the neural level, we obtained evidence that the neural units with most selectivity for the novel objects before training are the units that strengthen their selectivity most during training, as predicted by the informativeness hypothesis (Brants et al., 2015). This effect was observed in lateral occipital cortex (LOC), which is consistent with the properties of object learning (e.g. Baeck et al., 2016) and with the causal involvement of LOC in object learning (Van Meel et al., 2016). In a study with real-life expertise, we observed that real-life expertise is related to a mixture of domain-general and domain-specific changes in neural processing. Part of the domain-specific changes fit with the predictions of the informativeness hypothesis.
In Work Package 6, we contrasted the contribution of perceptual factors (shape similarity), conceptual factors (semantic similarity), and functional factors (similarity in object manipulation) to object representations in ventral and dorsal visual cortex (Bracci et al., 2017). In addition, we manipulated task context: either semantic or functional similarity was task relevant. We found that ventral occipitotemporal cortex represents all factors, independent of task factors. In contrast, (dorsal) parietal areas only represented the task-relevant factor, either semantic similarity or functional similarity. These data suggest that task relevance of functionality does not strongly influence selectivity in ventral visual cortex, in contrast to the strong effect of task in parietal cortex.
More specifically, in Work Package 1 we analyzed properties of module-like areas and the systems they belong to. We characterized the visual word representations in the visual word form area, face representations in and beyond the fusiform face area, and magnitude representations in the intraparietal sulcus (Baeck et al., 2015) (Goesaert et al., 2013) (Bulthé et al., 2015). We combined the multi-voxel pattern analysis (MVPA) methods which are central in this proposal with methods to analyze the functional connectivity between regions. An important test case for this combination was Boets, Op de Beeck, et al. (2013), in which we showed that in the context or reading difficulties there is evidence for intact neural representations, as assessed through MVPA, combined with impaired connectivity.
In Work Package 2, we analyzed non-modular maps in sighted individuals and their relationship to module-like structure in both brain and behavior. This work package hypothesized that there might be a large-scale map of shape selectivity which might be an anchor point for the stronger category selectivity. We implemented a study (Bracci & Op de Beeck, 2016) in which we dissociated the two factors, shape and category. We also provided a computational analysis through deep-learning convolutional networks that gave insights into how shape and category selectivity might be related (Kubilius et al., 2016). Finally, we brought all the evidence together, and presented a theoretical analysis of the available data, arguing in favor of a feature-based categorical coding (Bracci et al., 2017, Neuropsychologia). This explains the role of features in the representation of categories.
In Work Package 3, we characterized non-modular maps in congenitally blind participants and category selectivity. After first investigating the properties of non-modular maps and modular category selectivity in the ventral occipitotemporal cortex of low-vision patients (Goesaert et al., 2014), we set up a large study to obtain high-quality data in congenitally blind participants about how their ventral occipitotemporal cortex responds to auditory stimulation related to particular object categories (van den Hurk et al., 2017, PNAS). We showed that the activity pattern in the blind for sounds of faces, bodies, objects, and scenes could be used to predict which visual stimulus was associated with an activity pattern from a sighted control participant. These findings show the extent to which the overall organization of ‘visual’ ventral occipitotemporal cortex can develop despite the absence of any visual input.
In Work Package 4, we piloted an experiment in which we determine the selectivity for number-related words and symbols in adults (Peters et al., 2015), and afterwards also in children (Peters et al. 2016). The results confirm the hierarchical processing of words and symbols in the brain, as well as the prediction that children generally activate similar brain networks as adults do.
In Work Package 5, we investigated how experience affects non-modular maps and the strength of modularity. At the neural level, we obtained evidence that the neural units with most selectivity for the novel objects before training are the units that strengthen their selectivity most during training, as predicted by the informativeness hypothesis (Brants et al., 2015). This effect was observed in lateral occipital cortex (LOC), which is consistent with the properties of object learning (e.g. Baeck et al., 2016) and with the causal involvement of LOC in object learning (Van Meel et al., 2016). In a study with real-life expertise, we observed that real-life expertise is related to a mixture of domain-general and domain-specific changes in neural processing. Part of the domain-specific changes fit with the predictions of the informativeness hypothesis.
In Work Package 6, we contrasted the contribution of perceptual factors (shape similarity), conceptual factors (semantic similarity), and functional factors (similarity in object manipulation) to object representations in ventral and dorsal visual cortex (Bracci et al., 2017). In addition, we manipulated task context: either semantic or functional similarity was task relevant. We found that ventral occipitotemporal cortex represents all factors, independent of task factors. In contrast, (dorsal) parietal areas only represented the task-relevant factor, either semantic similarity or functional similarity. These data suggest that task relevance of functionality does not strongly influence selectivity in ventral visual cortex, in contrast to the strong effect of task in parietal cortex.