Periodic Reporting for period 2 - VIS-A-VIS (How visual action shapes active vision)
Reporting period: 2022-07-01 to 2023-12-31
A key component of visual perception is our ability to move: In a flick of the eye, we see the time on the watch, and we quickly turn around if we hear our name in a crowd of people. These visual actions shift our eyes, heads, and bodies to bring relevant parts of the world into the fovea—the small central portion of the retina that provides high-acuity vision.
Although visual actions vitally extend the scope of high-acuity vision, their immediate sensory consequences have challenged scientists for centuries: How do we not experience the brisk motion of the entire scene on the retina every time the eyes move (perceptual omission)? How does the brain keep track of objects’ changing retinal locations across consecutive glances (object continuity). And how do we routinely attribute retinal motion to our own movements rather than to motion in the world (sense of agency). Understanding these phenomena provides insights into how the mind works in health and disease, how to synthesize computer vision systems and build robots that have similar capabilities, and how to improve technology that relies on human perception and interaction.
VIS-À-VIS pursues a radically new perspective at these issues, based on a key insight: We move our body and its sensors in very reliable ways. Visual actions, in particular, follow distinct kinematic rules, and because every visual action translates directly into a movement of the world on the retinal image, these rules also directly govern the sensory input. The sensory consequences of visual actions can thus be distinguished from motion in the world based on the basic kinematic rules they follow. VIS-À-VIS pursues the idea that the immediate sensory consequences of visual actions—rather than being a nuisance to sensory processing—support core functions in active vision.
VIS-À-VIS leverages innovative technology, state-of-the-art psychophysical tools, and robust experimental protocols to find out if and how the active visual system learns and exploits the lawful relation between visual actions and their sensory consequences to establish perceptual omission, object continuity, and the sense of agency.
Although visual actions vitally extend the scope of high-acuity vision, their immediate sensory consequences have challenged scientists for centuries: How do we not experience the brisk motion of the entire scene on the retina every time the eyes move (perceptual omission)? How does the brain keep track of objects’ changing retinal locations across consecutive glances (object continuity). And how do we routinely attribute retinal motion to our own movements rather than to motion in the world (sense of agency). Understanding these phenomena provides insights into how the mind works in health and disease, how to synthesize computer vision systems and build robots that have similar capabilities, and how to improve technology that relies on human perception and interaction.
VIS-À-VIS pursues a radically new perspective at these issues, based on a key insight: We move our body and its sensors in very reliable ways. Visual actions, in particular, follow distinct kinematic rules, and because every visual action translates directly into a movement of the world on the retinal image, these rules also directly govern the sensory input. The sensory consequences of visual actions can thus be distinguished from motion in the world based on the basic kinematic rules they follow. VIS-À-VIS pursues the idea that the immediate sensory consequences of visual actions—rather than being a nuisance to sensory processing—support core functions in active vision.
VIS-À-VIS leverages innovative technology, state-of-the-art psychophysical tools, and robust experimental protocols to find out if and how the active visual system learns and exploits the lawful relation between visual actions and their sensory consequences to establish perceptual omission, object continuity, and the sense of agency.
VIS-À-VIS consists of 6 work packages (WPs), each addressing a particular challenge:
WP1 aims at understanding if the kinematics of eye movements are related to why the rapid motion that they cause on the retina is not seen. Using exceptionally high-speed visual displays, we uncovered a tight coupling between saccadic eye movements (the fastest movements of the retinal surface) and a fundamental perceptual process in human vision (seeing objects moving at high speeds): The speed limit of perceiving a stimulus’ rapid motion from one location to the next is directly proportional to the speed of saccadic eye movements over the same distance. We showed further that the visual statistics of natural scenes strongly impact perceptual omission of visual motion smear during saccades. In both cases, a parsimonious visual mechanisms based on the interplay of the retinal consequences of eye movements and processes in early visual areas can explain the results.
WP2 explores the origin of lawful mutual relations between visual action and perception. We have found ways to reliably manipulate the lawful mutual relations between visual action and perception, relying on pace-based or reward-based manipulations of motor behavior (and, thus, its visual consequences) and gaze-contingent image motion. These allow us to test hypotheses about the consequences of such manipulations for the visual system and its coupling to motor control. For instance, we have already shown people with fast saccades can perceive objects moving at higher speeds than people with slow saccades. Moreover, we established a link between eye-movement kinematics and storage of visual information in short-term memory and showed that saccades engage processes that enable spatial generalization of perceptual skills.
In WP3, we ask if the high-speed motion that saccades impose on the retina may help the visual system to keep track of objects over time. We have now demonstrated specific visual consequences of saccadic eye movements (motion smear across the retina) indeed aid establishing object correspondence across successive fixations of the eyes. We continue to explore the role of these sensory consequences in object continuity and feature integration in trans-retinal movements during both saccades and passive fixation.
WP4 links lawful sensory consequences of visual actions to causality and agency. Initial results show that visual signals alone can cause temporal recalibration of the association between saccadic eye movements and their visual consequences. Moreover, we have used high-speed displays to render visual information visible during microsaccades or catch-up saccades during smooth pursuit eye movements and discovered that human observers have little-to-none awareness of their own oculomotor behavior both in the presence and absence of saccade-contingent retinal stimulation.
In WP5, we integrate the empirical knowledge gained to synthesize a new theory of active vision. We have developed a framework that distinguishes three types of sensory consequences of actions: intended consequences, intrinsic consequences, and incidental consequences, integrated what we currently know about incidental consequences of visual actions (e.g. the massive motion signals introduced saccades or the transients imposed by eye blinks), and developed hallmarks for different degrees of coupling between perception and action.
Finally, in WP6, we are setting up a novel 360-degree screen that will extend the scope of our experimental tools to enable the study of large-scale visual actions (combined eye-head-body movements) rather than just saccadic eye movements.
WP1 aims at understanding if the kinematics of eye movements are related to why the rapid motion that they cause on the retina is not seen. Using exceptionally high-speed visual displays, we uncovered a tight coupling between saccadic eye movements (the fastest movements of the retinal surface) and a fundamental perceptual process in human vision (seeing objects moving at high speeds): The speed limit of perceiving a stimulus’ rapid motion from one location to the next is directly proportional to the speed of saccadic eye movements over the same distance. We showed further that the visual statistics of natural scenes strongly impact perceptual omission of visual motion smear during saccades. In both cases, a parsimonious visual mechanisms based on the interplay of the retinal consequences of eye movements and processes in early visual areas can explain the results.
WP2 explores the origin of lawful mutual relations between visual action and perception. We have found ways to reliably manipulate the lawful mutual relations between visual action and perception, relying on pace-based or reward-based manipulations of motor behavior (and, thus, its visual consequences) and gaze-contingent image motion. These allow us to test hypotheses about the consequences of such manipulations for the visual system and its coupling to motor control. For instance, we have already shown people with fast saccades can perceive objects moving at higher speeds than people with slow saccades. Moreover, we established a link between eye-movement kinematics and storage of visual information in short-term memory and showed that saccades engage processes that enable spatial generalization of perceptual skills.
In WP3, we ask if the high-speed motion that saccades impose on the retina may help the visual system to keep track of objects over time. We have now demonstrated specific visual consequences of saccadic eye movements (motion smear across the retina) indeed aid establishing object correspondence across successive fixations of the eyes. We continue to explore the role of these sensory consequences in object continuity and feature integration in trans-retinal movements during both saccades and passive fixation.
WP4 links lawful sensory consequences of visual actions to causality and agency. Initial results show that visual signals alone can cause temporal recalibration of the association between saccadic eye movements and their visual consequences. Moreover, we have used high-speed displays to render visual information visible during microsaccades or catch-up saccades during smooth pursuit eye movements and discovered that human observers have little-to-none awareness of their own oculomotor behavior both in the presence and absence of saccade-contingent retinal stimulation.
In WP5, we integrate the empirical knowledge gained to synthesize a new theory of active vision. We have developed a framework that distinguishes three types of sensory consequences of actions: intended consequences, intrinsic consequences, and incidental consequences, integrated what we currently know about incidental consequences of visual actions (e.g. the massive motion signals introduced saccades or the transients imposed by eye blinks), and developed hallmarks for different degrees of coupling between perception and action.
Finally, in WP6, we are setting up a novel 360-degree screen that will extend the scope of our experimental tools to enable the study of large-scale visual actions (combined eye-head-body movements) rather than just saccadic eye movements.
In VIS-À-VIS's second half, we expect to make further progress in each WP. In WP1, we will leverage our protocols that allow for precise characterization of, and experimental control over, the visuomotor contingencies that shape perceptual phenomenology. In WP2, we aim to obtain empirical evidence for the causal direction of the tight relation between visual actions and perceptual omission. In this domain, we will continue to extend our efforts from saccadic eye movements to other types of eye movements (e.g. pursuit, optokinetic nystagmus, and blinks). WP3 has revealed functional consequences of immediate sensory contingencies of saccades and established that they are sufficient to establish object correspondence. We have little evidence, however, that these sensory consequences by themselves lead to the integration of feature information across successive glances at the same object. In WP4, we have initial evidence that the consequences of visual actions contribute to the temporal calibration between actions and their visual consequences. Further research will elucidate this relation and establish the boundary conditions that allow visual consequences to function as a cue to sensorimotor calibration. WP5 has made progress on a novel account of active vision that embeds our findings in the larger research landscape and will continue to integrate established findings with new results in a common theoretical framework.. WP6 will provide a powerful setup to study the interactions between visual actions and active vision, breaking ground for new research directions.