Periodic Reporting for period 1 - VisualGrasping (Visually guided grasping and its effects on visual representations)
Période du rapport: 2018-04-02 au 2020-04-01
Vision helps us select, plan, and execute actions, including grasping, that allow us to interact with our environment. To do so, the visual system needs to reconstruct the 3D shape and layout of objects in our surroundings from ambiguous 2D retinal images—a mathematically under-constrained task. Understanding how we use vision to pick up and interact with objects effectively is thus one of the most important challenges in behavioral science. Even state-of-the-art robotic AIs can fail to visually identify effective grasps nearly 20% of the time [Levine et al, 2018].
The main objectives of the project were thus to understand how humans use vision to plan grasping, and to investigate the visual representations along the human dorsal visual stream that are thought to play an important role in grasp planning. We found that humans combine visual information about object 3D shape, orientation and material composition to identify optimal grasping locations across different objects. Furthermore, we were able to characterize the neural computations that the brain employs to reconstruct the 3D shape of objects.
Klein LK, Maiello G, Paulun VC, Fleming RW (2020) Predicting precision grip grasp locations on three-dimensional objects. PLOS Computational Biology 16(8): e1008081. https://doi.org/10.1371/journal.pcbi.1008081
In parallel to this line of work, we investigated the visual computations that our brain performs to reconstruct the 3D shape and layout of the objects we plan to grasp. To do this, we showed human participants 3D images on stereoscopic monitors, and asked them to report on the perceived depth structure of the stimuli. Then, we constructed an image-computable model of depth processing. This model takes into account how our brain organizes the binocular visual input at our retinae, but operates directly in cortical image space. We tested the model with the same stimuli and procedures used with human participants, and found that the model reported the same depth structure of the environment as humans. This body of work was also published in the high-ranking journal PLOS Computational Biology.
To begin with, the project is pushing technological advancements in disparate engineering fields. For example, our observations on how human grasp selection is affected by object mass and mass distribution have already been implemented in a recent robotics paper [Veres et al, 2020]. In ongoing collaboration with computer science engineers instead, we are investigating how visually guided grasping differs in real and virtual environments, with the goal of designing more immersive and user-friendly virtual and augmented reality technologies.
The work carried out provides important benefits to society in terms of translation to clinical practice. As the population of Western countries ages, neurological and eye diseases causing low vision and visuomotor deficits are becoming increasingly prevalent. Understanding visual function in healthy populations is the first step in understanding the mechanisms of pathological visual loss. Therefore, in clinical collaborations stemmed from the current project, we have already published papers on retinal structure, visual function, and depth processing in glaucoma and age related macular degeneration, two potentially blinding eye diseases. Additionally, our computational account of human grasp selection has been the basis of an imaging study in which we are employing functional magnetic resonance imaging to identify the areas of the brain responsible for distinct aspects of grasp planning. This work does not only expand our knowledge of brain function, it could help identify and treat neurological disorders such as stroke. Similarly, in other ongoing work we are studying how humans use motor imagery, action observation, and sensorimotor feedback to learn how to grasp objects. Since motor imagery and action observation have shown promise in aiding and strengthening motor rehabilitation techniques in a variety of neurological conditions, our model-driven approach could be used to guide and strengthen these neurorehabilitation techniques.