Periodic Reporting for period 1 - FOFC (Function and Organization of Feedback Connections)
Reporting period: 2019-09-01 to 2021-08-31
The brain has many recurrent loops---situations where neurons send connections back to populations of neurons that they got input from. In fact, less than 10% of synapses in sensory areas of cerebral cortex are from feedforward input; the rest stem from lateral and feedback connections. Much experimental and theoretical research into sensory processing, however, focuses on the feedforward pathway despite the significant impact recurrent connections have on activity. What role do these recurrent connections play, and how do they carry them out? This project combines computational modeling with in vivo imaging to explore the function and organization of recurrent connections in the mouse visual system. Specifically, it proposes training neural network models with recurrent connections on different challenging visual tasks inspired by a hypothesized role of feedback. Once these models are trained, their response to images can be compared to experimental data collected from multiple visual areas. These models also provide an ideal experimental setting wherein the anatomy of the learned connections can be fully characterized. These characterizations can inform hypotheses about what features should be present in the biology. To explore if these features are present, in vivo imaging will be used to characterize the information carried by different pathways in the mouse visual system. The impact of recurrence on neurons in V1 will also be determined by comparing activity under normal circumstances to when these recurrent pathways are silenced. In summary, this work will use modeling to relate properties of feedback connections to their function and test for these properties experimentally. It will also develop an approach that can be applied to many neural pathway. Understanding the role of recurrent processing will inform how the brain processes sensory information over time. Studying this process using computational models also makes it easy to transfer our understanding of the brain to artificial intelligence.
Thus far, work has included literature reviews and computational modeling and analysis. Two review articles on the use of artificial neural networks to study complex sensory processing were published. Modeling work has been carried out that explores different forms of recurrence in visual processing motivated by behavioral studies suggest that recurrence in the visual system is important for processing degraded stimuli.Specifically we added four different kinds of recurrence---two of each anatomical form of lateral and feedback---to a feedforward convolutional neural network and find all of them capable of increasing the ability of the network to classify noisy digit images. To do so, we take inspiration from findings in biology by adding predictive coding-based feedback and lateral surround suppression. To compare these forms of recurrence to anatomically-matched counterparts we also train feedback and lateral connections directly to classify degraded images. Counter-intuitively, we find that the anatomy of the recurrence is not related to its function: both forms of task-trained recurrence change neural activity and behavior similarly to each other and differently from their bio-inspired anatomical counterparts. By using several analysis tools frequently applied to neural data, we identified the distinct strategies used by the predictive coding versus task-trained networks. Specifically, predictive coding de-noises the representation of noisy images at the first layer of the network and decreases its dimensionality, leading to an expected increase in classification performance. Surprisingly, in the task-trained networks, representations are not de-noised over time at the first layer (in fact, they become `noiser' and dimensionality increases) yet these dynamics do lead to de-noising at later layers. The analyses used here can be applied to real neural recordings to identify the strategies at play in the brain. Our analysis of an fMRI dataset weakly supports the predictive coding model but points to a need for higher-resolution cross-regional data to understand recurrent visual processing. In a separate study, the style of training of networks was also explored in order to understand how objective functions impact visual representations. Specifically, artificial neural systems trained using reinforcement, supervised, and unsupervised learning all acquire internal representations of high dimensional input. To what extent these representations depend on the different learning objectives is largely unknown. Here we compare the representations learned by eight different convolutional neural networks, each with identical ResNet architectures and trained on the same family of egocentric images, but embedded within different learning systems. Specifically, the representations are trained to guide action in a compound reinforcement learning task; to predict one or a combination of three task-related targets with supervision; or using one of three different unsupervised objectives. Using representational similarity analysis, we find that the network trained with reinforcement learning differs most from the other networks. Through further analysis using metrics inspired by the neuroscience literature, we find that the model trained with reinforcement learning has a sparse and high-dimensional representation wherein individual images are represented with very different patterns of neural activity. Further analysis suggests these representations may arise in order to guide long-term behavior and goal-seeking in the RL agent. Our results provide insights into how the properties of neural representations are influenced by objective functions and can inform transfer learning approaches. These results are in the process of being disseminated in open-access venues and have been discussed at several conferences and seminars.
The project has expanded beyond its original conception by including many different forms of recurrence and putting explicit emphasize on developing the analysis tools needed to understand recurrent neural networks, both biological and artificial. This has implications for the use of artificial intelligence in the real world, where an understanding of how artificial neural networks work can increase trust in their use and ensure correct application. We intend to publish results on recurrent neural network models and present these results to broad audiences. Furthermore, an explicit plan for developing tools of neural network analysis will be written and published as a perspective piece to lay out the need for this work and offer guidance on best practices for carrying it out. This will push the field to more explicitly reflect on the tools it uses, and could have implications well beyond the study of sensory processing. An ongoing project focused on building better models of mouse visual processing will also be concluded. This study uses specific anatomical details of mouse visual cortex combined with biologically-motivated forms of training on ethologically relevant images to determine the best formula for creating a model that can predict mouse visual activity. Finally, plans for further experiments on mouse visual processing will be finalized based on the results of all of the above-mentioned modeling work.