Periodic Reporting for period 2 - RELEVANCE (How body relevance drives brain organization)
Periodo di rendicontazione: 2022-01-01 al 2023-06-30
Social species, and specifically human and nonhuman primates, rely heavily on conspecifics for
survival. The project RELEVANCE aims to understand how the brain evolved special structures to process highly relevant social stimuli like bodies and to reveal how social vision sustains adaptive behaviour. The project aims at developing a mechanistic and computational understanding of the visual processing of bodies and their interactions. It will show how this processing sustains higher abilities such as understanding intention, action and emotion. RELEVANCE tries to accomplish this by integrating advanced methods from multiple disciplines: psychophysics and high-field functional imaging in combination with virtual reality and neural stimulation in humans; electrophysiology and causal perturbation methods in nonhuman primates; computational modeling of neural data and development of computer-generated, well-controlled stimuli. We will develop novel deep neural network models that unify the data. These models will not only capture detailed mechanisms of neural processing of complex social stimuli and its dynamics, but also reproduce the modulation of brain activity during active behavior. RELEVANCE will reveal novel ways of understanding and diagnosing social communication deficits in neuropsychiatry, and suggest novel hypotheses about their genetic basis. It will motivate novel principles and architectures for processing of socially relevant information in computer and robotic systems.
survival. The project RELEVANCE aims to understand how the brain evolved special structures to process highly relevant social stimuli like bodies and to reveal how social vision sustains adaptive behaviour. The project aims at developing a mechanistic and computational understanding of the visual processing of bodies and their interactions. It will show how this processing sustains higher abilities such as understanding intention, action and emotion. RELEVANCE tries to accomplish this by integrating advanced methods from multiple disciplines: psychophysics and high-field functional imaging in combination with virtual reality and neural stimulation in humans; electrophysiology and causal perturbation methods in nonhuman primates; computational modeling of neural data and development of computer-generated, well-controlled stimuli. We will develop novel deep neural network models that unify the data. These models will not only capture detailed mechanisms of neural processing of complex social stimuli and its dynamics, but also reproduce the modulation of brain activity during active behavior. RELEVANCE will reveal novel ways of understanding and diagnosing social communication deficits in neuropsychiatry, and suggest novel hypotheses about their genetic basis. It will motivate novel principles and architectures for processing of socially relevant information in computer and robotic systems.
We made a photorealistic dynamic macaque face avatar which provided the necessary groundwork to produce an animation of a highly-realistic three-dimensional monkey avatar with veridical full-body motion data obtained by tracking of real animals in their home environment. We constructed a novel dynamic, naturalistic stimulus set of bodies of monkeys and humans. The stimuli were presented to the monkey and human subjects in fMRI studies to reveal regions in the brain that respond specifically to dynamic bodies (“body patches”) and to examine the body patch network. A 7T human fMRI study revealed hitherto unknown dynamic body patches and a network analysis revealed two networks selective to bodies. One of these networks showed a high species selectivity, being specifically tuned to human bodies. We performed a full mapping of dynamic body patches in monkeys with fMRI using realistic dynamic monkey body stimuli. Subsequently, we revealed the shape and motion features that drive the neural responses to bodies at the single-unit level in these patches. We generated a large set of monkey avatars, varying in pose and viewpoint, and examined the feature selectivity of single neurons in body patches. In parallel, in humans, we revealed the selectivity for body pose features using human avatars and modeling in a body-selective area. A neural model was developed that accounts for monkey single-unit recording data of responses to moving bodies behind a narrow slit. This model is based on a novel nonlinear mechanism that suppresses clutter generated by the slit. Models for body motion and pose recognition were implemented and tested with stimuli employed in the physiological experiments. One model combines a deep network architecture with a recurrent neural field-based mechanism for sequence recognition. We extended and modified deep neural network models by combination with neurodynamic models in order to account for the dynamics of the responses of body-selective neurons.
We established for the first time 3D video-based tracking of monkey full-body motion at a level of accuracy that is sufficient for 3D animation of avatars. Previous video-based tracking methods for monkeys did not reach sufficient accuracy for this application. To do this, we developed machine-learning-based methods for stylized real-time animation. In addition, a novel plugin for the software Maja has been developed that implements hierarchical deep Gaussian Process models for the blending of dynamic stimuli.
We performed the first ultra-high field fMRI study of naturalistic dynamic images. We fine-tuned fMRI measurements on the 7T scanner for optimal imaging of cortical and subcortical signals in response to video images. We developed a new encoding pipeline for fMRI data analysis. We developed a novel way of investigating social threat perception by combining Virtual Reality, EEG, and Skin Conductance Responses. We addressed the central issue of body features driving dynamic perception and developed a sparse Independent Component Analysis method for feature computation and brain data modeling. A theoretical framework was formulated to clarify the role of midlevel features of body perception and its contribution to genuine social perception. We developed a computational method to assess the contribution of motion and static features to the single-unit selectivity for dynamic bodies, which can be generalized to other video stimuli.
Next, we expect to reveal the information flow between different body patches and map bottom-up and top-down modulations in the body patch network. We aim to reveal task-dependent processing in body patches. We will show how the body patch network is able to represent multiple agents and their interaction. We expect to understand how the neural architecture of body patches feeds in and is modulated by higher functions attributed to the social brain. We will elaborate further on the computational modeling of dynamic body processing.
We performed the first ultra-high field fMRI study of naturalistic dynamic images. We fine-tuned fMRI measurements on the 7T scanner for optimal imaging of cortical and subcortical signals in response to video images. We developed a new encoding pipeline for fMRI data analysis. We developed a novel way of investigating social threat perception by combining Virtual Reality, EEG, and Skin Conductance Responses. We addressed the central issue of body features driving dynamic perception and developed a sparse Independent Component Analysis method for feature computation and brain data modeling. A theoretical framework was formulated to clarify the role of midlevel features of body perception and its contribution to genuine social perception. We developed a computational method to assess the contribution of motion and static features to the single-unit selectivity for dynamic bodies, which can be generalized to other video stimuli.
Next, we expect to reveal the information flow between different body patches and map bottom-up and top-down modulations in the body patch network. We aim to reveal task-dependent processing in body patches. We will show how the body patch network is able to represent multiple agents and their interaction. We expect to understand how the neural architecture of body patches feeds in and is modulated by higher functions attributed to the social brain. We will elaborate further on the computational modeling of dynamic body processing.