Final Report Summary - APPEARINACTION (Appearance in Action: The interplay of perception and action as revealed by attention-related changes in phenomenological appearance)
Summary description of Project background and objectives
Perception and motor control act in concert to suit our behavior to current needs. Most research on the interplay of perception and motor control has examined how sensory input is transformed into movement generation. Recent psychophysical and physiological evidence, however, shows that the generation of movements has considerable impact on perception. Most prominently, these interactions have been documented for vision and eye movements, whereas considerably less is known about the mutual dependencies of perception and other goal-directed movements, such as reaching. In the proposed project, we aim to find out whether and how perception and action interact by accessing the same visual representation, a saliency map, in which bottom-up stimulus characteristics are weighted by attentional mechanisms. Taking appearance to represent the readout of integrated perceptual saliency (Carrasco et al., Nat. Neurosci. 2004; Treue, Trends Cogn. Sci. 2004), we study its interactions with goal-directed eye and hand movements.
In a three-fold project, we set out to investigate influences of (1) attention on perception (appearance and/or performance), (2) action on perception, and (3) perception on action. Our research sheds light on whether and how perception and motor control share visual information to coordinate behavior in space and time.
The project is based on a close collaboration between Dr. Martin Rolfs (the researcher), Prof. Marisa Carrasco (the outgoing host at the Department of Psychology and Center for Neural Science at New York University, USA), and Dr. Eric Castet (the scientist in charge; Laboratoire de Psychologie Cognitive, CNRS Aix-Marseille Universit?, France).comprehensive training at NYU deepens and broadens the researcher's theoretical knowledge, methodological competency, and other scientific skills. The project thus contributes to a strong European position in the burgeoning field of Mind and Brain sciences.
Work performed and main results achieved
The project had a total duration of 31 months, including 24 months in the outgoing phase and 7 months in the return phase. While the return phase had initially been planned for a duration of 12 months, the researcher received an attractive offer to become a Junior Research Group Leader in Germany and the termination of the project has been advanced by 5 months. Nevertheless, the main objectives of the project have been reached. We conducted a number of studies exploring the link between action (in terms of eye and reach movements) and perception (in terms of visual performance and appearance, which will be summarised in the following.
In a first series of experiments, we determined the dynamic changes of perceived contrast and visual performance at the target of saccadic eye movements, revealing simultaneous progressive enhancement in both orientation discrimination performance and perceived contrast as time approached saccade onset. These results link the dynamics of saccade preparation, visual performance, and subjective experience and show that upcoming eye movements alter visual processing by increasing the signal strength. This study has now been published in a high-impact, peer-reviewed journal (Journal of Neuroscience).
In a second study, we adapted the eye-movement paradigm for use with goal-directed reach movements of the hands. This project has been done in close collaboration with Bonnie Lawrence, a visiting scholar at NYU and an expert on the neural control of goal-directed movements. We show that reaches, akin to saccadic eye movements, influence the visual representation of their targets, but at a different time scale. These results have been published at the Meeting of the Vision Sciences Society (and as an abstract in the Journal of Vision).
In a third project, we investigated the dependence of the perceptual consequences of movement preparation on the broader temporal context of the task. We show that what you see depends not only on where you are going to look next, but also on where you looked a moment ago. Specifically, preparing an eye movement not only re-distributes attention to particular locations in the visual periphery, but also to particular features, and both depend implicitly on recent experience. These results have important theoretical consequences as they show that the link between perception and action is plastic, flexibly adapting to current task demands. This study has now been published in a top journal in the field (Vision Research).
In a fourth project, we discovered that presaccadic remapping, a process thought to aid perceptual continuity across saccades, causes perceptual learning at the location a presaccadic stimulus is predicted to have after the saccade, contributing to our understanding of how the brain keeps track of relevant locations in the world despite the frequent, large image shifts that saccadic eye movements cause on the retina. Data collection is now complete and a manuscript is in preparation for a Special Issue of the (to be published either in Philosophical Transactions of the Royal Society B or in Journal of Vision).
A fifth project that we started during the return phase in Dr. Eric Castet's lab in Marseille set the stage for a long-term collaboration between the researcher and the return host, Dr. Castet. We studied the dynamics of visual sensitivity around the time of saccadic eye movements. We find reliable decrements in human's ability to see stimuli defined by color contrastcontrary to long-standing assumptions in the field. An abstract describing our results is planned for one of the two main international meetings in the Vision Sciences.
In summary, we have uncovered pronounced and bidirectional interactions between perception and action. Our results have led to revisions of current theories in the interdisciplinary field of active vision. The bulge of this research has been communicated to a broader scientific audience. These communications included more than 10 presentations at professional meetings and 8 invited presentations at research institutes. Two manuscripts have been accepted for publication, two are under review, and two more are currently in preparation. Through these conferences and talks, as well as through short and productive research visits to labs in the US and in Germany, Martin Rolfs extended his international professional network, which now includes a number of labs both in America and in Europe. In collaborations during the Marie Curie Fellowship, the researcher was also able to publish several other key findings that have received considerable attention from a broad audience and will have a lasting impact on his research program for the years to come. One piece of evidence for the outstanding scientific impact of these findings is that the article describing the main results resulting from these efforts has been recommended two times by the Faculty of 1000, an expert panel identifying and evaluating the most significant articles in the field of biomedical research.
The researcher also achieved the other main objectives of the project, including the acquisition of research skills and techniques (e. g., advanced training in psychophysics, hand-movement recordings, brain imaging using Magnetoencephalography), and a broader and deeper understanding of perception, attention, and action through training and participation in relevant meetings. Intensive workshops fostered the researchers'skills in research management and science communication. To build up more experience in teaching, the researcher also lectured an intensive undergraduate summer course on Perception and substituted Prof. Carrasco in some of her lectures. Finally he had many opportunities to mentor young students in collaborative research projects, preparing him for a group leader position in science and academia.
Perception and motor control act in concert to suit our behavior to current needs. Most research on the interplay of perception and motor control has examined how sensory input is transformed into movement generation. Recent psychophysical and physiological evidence, however, shows that the generation of movements has considerable impact on perception. Most prominently, these interactions have been documented for vision and eye movements, whereas considerably less is known about the mutual dependencies of perception and other goal-directed movements, such as reaching. In the proposed project, we aim to find out whether and how perception and action interact by accessing the same visual representation, a saliency map, in which bottom-up stimulus characteristics are weighted by attentional mechanisms. Taking appearance to represent the readout of integrated perceptual saliency (Carrasco et al., Nat. Neurosci. 2004; Treue, Trends Cogn. Sci. 2004), we study its interactions with goal-directed eye and hand movements.
In a three-fold project, we set out to investigate influences of (1) attention on perception (appearance and/or performance), (2) action on perception, and (3) perception on action. Our research sheds light on whether and how perception and motor control share visual information to coordinate behavior in space and time.
The project is based on a close collaboration between Dr. Martin Rolfs (the researcher), Prof. Marisa Carrasco (the outgoing host at the Department of Psychology and Center for Neural Science at New York University, USA), and Dr. Eric Castet (the scientist in charge; Laboratoire de Psychologie Cognitive, CNRS Aix-Marseille Universit?, France).comprehensive training at NYU deepens and broadens the researcher's theoretical knowledge, methodological competency, and other scientific skills. The project thus contributes to a strong European position in the burgeoning field of Mind and Brain sciences.
Work performed and main results achieved
The project had a total duration of 31 months, including 24 months in the outgoing phase and 7 months in the return phase. While the return phase had initially been planned for a duration of 12 months, the researcher received an attractive offer to become a Junior Research Group Leader in Germany and the termination of the project has been advanced by 5 months. Nevertheless, the main objectives of the project have been reached. We conducted a number of studies exploring the link between action (in terms of eye and reach movements) and perception (in terms of visual performance and appearance, which will be summarised in the following.
In a first series of experiments, we determined the dynamic changes of perceived contrast and visual performance at the target of saccadic eye movements, revealing simultaneous progressive enhancement in both orientation discrimination performance and perceived contrast as time approached saccade onset. These results link the dynamics of saccade preparation, visual performance, and subjective experience and show that upcoming eye movements alter visual processing by increasing the signal strength. This study has now been published in a high-impact, peer-reviewed journal (Journal of Neuroscience).
In a second study, we adapted the eye-movement paradigm for use with goal-directed reach movements of the hands. This project has been done in close collaboration with Bonnie Lawrence, a visiting scholar at NYU and an expert on the neural control of goal-directed movements. We show that reaches, akin to saccadic eye movements, influence the visual representation of their targets, but at a different time scale. These results have been published at the Meeting of the Vision Sciences Society (and as an abstract in the Journal of Vision).
In a third project, we investigated the dependence of the perceptual consequences of movement preparation on the broader temporal context of the task. We show that what you see depends not only on where you are going to look next, but also on where you looked a moment ago. Specifically, preparing an eye movement not only re-distributes attention to particular locations in the visual periphery, but also to particular features, and both depend implicitly on recent experience. These results have important theoretical consequences as they show that the link between perception and action is plastic, flexibly adapting to current task demands. This study has now been published in a top journal in the field (Vision Research).
In a fourth project, we discovered that presaccadic remapping, a process thought to aid perceptual continuity across saccades, causes perceptual learning at the location a presaccadic stimulus is predicted to have after the saccade, contributing to our understanding of how the brain keeps track of relevant locations in the world despite the frequent, large image shifts that saccadic eye movements cause on the retina. Data collection is now complete and a manuscript is in preparation for a Special Issue of the (to be published either in Philosophical Transactions of the Royal Society B or in Journal of Vision).
A fifth project that we started during the return phase in Dr. Eric Castet's lab in Marseille set the stage for a long-term collaboration between the researcher and the return host, Dr. Castet. We studied the dynamics of visual sensitivity around the time of saccadic eye movements. We find reliable decrements in human's ability to see stimuli defined by color contrastcontrary to long-standing assumptions in the field. An abstract describing our results is planned for one of the two main international meetings in the Vision Sciences.
In summary, we have uncovered pronounced and bidirectional interactions between perception and action. Our results have led to revisions of current theories in the interdisciplinary field of active vision. The bulge of this research has been communicated to a broader scientific audience. These communications included more than 10 presentations at professional meetings and 8 invited presentations at research institutes. Two manuscripts have been accepted for publication, two are under review, and two more are currently in preparation. Through these conferences and talks, as well as through short and productive research visits to labs in the US and in Germany, Martin Rolfs extended his international professional network, which now includes a number of labs both in America and in Europe. In collaborations during the Marie Curie Fellowship, the researcher was also able to publish several other key findings that have received considerable attention from a broad audience and will have a lasting impact on his research program for the years to come. One piece of evidence for the outstanding scientific impact of these findings is that the article describing the main results resulting from these efforts has been recommended two times by the Faculty of 1000, an expert panel identifying and evaluating the most significant articles in the field of biomedical research.
The researcher also achieved the other main objectives of the project, including the acquisition of research skills and techniques (e. g., advanced training in psychophysics, hand-movement recordings, brain imaging using Magnetoencephalography), and a broader and deeper understanding of perception, attention, and action through training and participation in relevant meetings. Intensive workshops fostered the researchers'skills in research management and science communication. To build up more experience in teaching, the researcher also lectured an intensive undergraduate summer course on Perception and substituted Prof. Carrasco in some of her lectures. Finally he had many opportunities to mentor young students in collaborative research projects, preparing him for a group leader position in science and academia.