Periodic Reporting for period 1 - MultiSense (System Identification of the Dynamics of Multisensory Integration)
Okres sprawozdawczy: 2021-04-01 do 2023-03-31
The central nervous system (CNS) processes sensory information obtained through various sensory structures in the body. These include signals with a variety of spatiotemporal features, such as different speed and propagation patterns. For instance, visual information moves at the speed of light, enabling a direct spatial representation of the external world, while acoustic signals are far slower, and spatial cues are indirectly computed from amplitude and phase information. In this stream of multisensory information, CNS must decide what information is reliable, what should be rejected and how it should integrate these signals to construct a unique representation of the environment. How CNS accomplishes this process is not known.
There are two main aspects of the importance of the project idea for society. First, the results of this work enabled us to increase our understanding of the dynamics of the multisensory integration process. Note that the deficits in multisensory integration in humans are common and often detrimental to the quality of life. Progress on this front will undoubtedly facilitate better diagnosis and treatment for various disorders associated with multisensory integration issues, such as autism spectrum disorder, schizophrenia, and mild cognitive impairment.
More importantly, understanding the mechanisms of dynamic sensory reweighting in relation to sensory salience is critical, and it will find implementations in robotics and artificial intelligence. Currently, engineers know how to build custom filters for optimal sensor fusion. However, what is not known is how to perform online and real-time sensory weighting under dynamically changing sensory information. In other words, what is the optimal strategy for tuning the weights associated with each sensory system when the quality of the sensory information changes?
Motivated by this problem and the expected impact, our project had four main objectives, involving a systematic combination of behavioral experiments to reveal the underlying dynamics of multisensory integration: (1) Design and build an augmented reality setup that allows controlled and independent multisensory stimulation for the fish during their free-swimming behavior, (2) Identification of the tracking response of the fish during free swimming as a baseline for our multisensory conflict experiments, (3) Independent identification of sensory weights assigned to visual and electrosensory systems by the CNS during refuge tracking behavior, and (4) Monitoring how the CNS tunes its sensory weights in response to varying sensory salience.
During the implementation of the project, we developed a unique experimental setup that allows independently stimulating different sensory systems of weakly electric fish and zebrafish during their unconstrained, free behaviors. The setup allows online and real-time data collection under different experimental settings to support the use of control-theoretic analysis, such as data-driven system identification to reveal the dynamics of multisensory integration in freely behaving animals. The infrastructure we’ve developed in this project is modular and can be quickly adapted to use for different problems in neuroscience, where control-theoretic approaches would be a better fit for the analysis.
We succeed to estimate the weights assigned to each sensory system independently using control-theoretic approaches. Our analysis allowed us to identify the relative contribution of each sensory modality to the behavioral response of the fish through control-theoretic modeling of the behavior. More critically, we repeated this process under different sensory salience conditions to monitor how these weights are changing as the quality of the sensory information changes. This is critical to characterize the adaptive tuning strategy used by the animals to achieve optimal sensory integration under varying environmental conditions.
There are two main aspects of the importance of the project idea for society. First, the results of this work enabled us to increase our understanding of the dynamics of the multisensory integration process. Note that the deficits in multisensory integration in humans are common and often detrimental to the quality of life. Progress on this front will undoubtedly facilitate better diagnosis and treatment for various disorders associated with multisensory integration issues, such as autism spectrum disorder, schizophrenia, and mild cognitive impairment.
More importantly, understanding the mechanisms of dynamic sensory reweighting in relation to sensory salience is critical, and it will find implementations in robotics and artificial intelligence. Currently, engineers know how to build custom filters for optimal sensor fusion. However, what is not known is how to perform online and real-time sensory weighting under dynamically changing sensory information. In other words, what is the optimal strategy for tuning the weights associated with each sensory system when the quality of the sensory information changes?
Motivated by this problem and the expected impact, our project had four main objectives, involving a systematic combination of behavioral experiments to reveal the underlying dynamics of multisensory integration: (1) Design and build an augmented reality setup that allows controlled and independent multisensory stimulation for the fish during their free-swimming behavior, (2) Identification of the tracking response of the fish during free swimming as a baseline for our multisensory conflict experiments, (3) Independent identification of sensory weights assigned to visual and electrosensory systems by the CNS during refuge tracking behavior, and (4) Monitoring how the CNS tunes its sensory weights in response to varying sensory salience.
During the implementation of the project, we developed a unique experimental setup that allows independently stimulating different sensory systems of weakly electric fish and zebrafish during their unconstrained, free behaviors. The setup allows online and real-time data collection under different experimental settings to support the use of control-theoretic analysis, such as data-driven system identification to reveal the dynamics of multisensory integration in freely behaving animals. The infrastructure we’ve developed in this project is modular and can be quickly adapted to use for different problems in neuroscience, where control-theoretic approaches would be a better fit for the analysis.
We succeed to estimate the weights assigned to each sensory system independently using control-theoretic approaches. Our analysis allowed us to identify the relative contribution of each sensory modality to the behavioral response of the fish through control-theoretic modeling of the behavior. More critically, we repeated this process under different sensory salience conditions to monitor how these weights are changing as the quality of the sensory information changes. This is critical to characterize the adaptive tuning strategy used by the animals to achieve optimal sensory integration under varying environmental conditions.
The first part of the project focused on design and production of the experimental setup that we’ve planned to build during the implementation of the project. We designed a swim tunnel that can mimic the natural habitats of these fish. This was especially critical for zebrafish experiments, since the flow is the main factor that motivates the fish to perform the target tracking behavior. We also designed state-of-the-art hardware and software solutions for online and real-time data collection and processing. The multisensory stimulator systems allow us to investigate the individual contributions of different sensory systems on the behavioral response of the fish.
In the second part of the project, we benefited from control-theoretic models and data-driven system identification to estimate the frequency response characteristics of the natural tracking response in these fish. This was critically important for us to understand the baseline characteristics of the natural responses of the fish. In the third part of the project, we used this baseline activity to separately estimate the weights assigned to different sensory systems by the CNS. To achieve this, we stimulated different sensory organs separately using sensory conflicts and estimated the individual frequency responses. This enabled us to estimate the individual sensory weights adopted by the CNS during multisensory behavioral control.
As the final part of the project, we worked on monitoring the changes in sensory weights in response to dynamically changing sensory salience. To this end, we manipulated the quality of the sensory information during subsequent experimental trials, and we continuously applied system identification to separately estimate the individual sensory weights as the sensory salience varies. This is crucial for modeling the online retuning mechanism adopted by the CNS to change the sensory weights in response to dynamically changing environmental conditions.
In the second part of the project, we benefited from control-theoretic models and data-driven system identification to estimate the frequency response characteristics of the natural tracking response in these fish. This was critically important for us to understand the baseline characteristics of the natural responses of the fish. In the third part of the project, we used this baseline activity to separately estimate the weights assigned to different sensory systems by the CNS. To achieve this, we stimulated different sensory organs separately using sensory conflicts and estimated the individual frequency responses. This enabled us to estimate the individual sensory weights adopted by the CNS during multisensory behavioral control.
As the final part of the project, we worked on monitoring the changes in sensory weights in response to dynamically changing sensory salience. To this end, we manipulated the quality of the sensory information during subsequent experimental trials, and we continuously applied system identification to separately estimate the individual sensory weights as the sensory salience varies. This is crucial for modeling the online retuning mechanism adopted by the CNS to change the sensory weights in response to dynamically changing environmental conditions.
There are two aspects of the project that extends beyond the state of the art. First of all, the augmented reality experimental setup that we’ve developed is unique in that it allows multisensory conflict experiments both for weakly electric fish and zebrafish to stimulate various characteristics of the associated sensory systems. This setup allows the researchers to examine the multisensory behavioral control in these species during their unconstrained, free swimming. This is critical for a systematic characterization of the multisensory behavioral control, since motor movements of the animals have a substantial effect on the spatiotemporal characteristics of the sensory information.
In addition, our results present key systematic results about the changes in sensory weights in relation to the salience of the sensory information. We continuously estimate the sensory weights assigned to each sensory modality as the quality of the sensory stimulus changes. This is critical to characterize the adaptive strategies adopted by the central nervous system to retune its sensory weights based on environmental conditions. Understanding the mechanisms that the CNS use to dynamically tune its sensory weights is essential to develop next-generation bioinspired sensor fusion solutions for robotic systems. Our system allows monitoring the changes in the sensory weights for subsequent data-driven identification of the amazing tuning mechanism implemented by the CNS.
In addition, our results present key systematic results about the changes in sensory weights in relation to the salience of the sensory information. We continuously estimate the sensory weights assigned to each sensory modality as the quality of the sensory stimulus changes. This is critical to characterize the adaptive strategies adopted by the central nervous system to retune its sensory weights based on environmental conditions. Understanding the mechanisms that the CNS use to dynamically tune its sensory weights is essential to develop next-generation bioinspired sensor fusion solutions for robotic systems. Our system allows monitoring the changes in the sensory weights for subsequent data-driven identification of the amazing tuning mechanism implemented by the CNS.