If the sun appears from behind a cloud, some physical properties (light, shadow) of the scene drastically change, but our understanding of its structure does not. The first step of visual scene understanding is segmentation, in which our brain tries to infer which parts of the scene belong to which objects. Adults can do this in photographs – but photographs are not how we learned to see. We learned to see by moving around in a 3D world. The way that scenes project into our eyes, how light is affected by the optics of our eyes, how our photoreceptors sample the light, and how we move our eyes all provide rich information about our environment. However, we do not know how adults combine all this information to perceive segmented scenes, nor how important these different information sources are for learning. The goals of SEGMENT are to understand how adults use the rich information present in active 3D vision to perform segmentation, and to understand how this might be learned. We are developing a new display device and experimental methods to study how adults segment scenes when realistic visual information is available. In addition, we are developing new technologies using advanced computer graphics and machine learning to simulate the inputs to the visual system from early development to adulthood. We are conducting in silico experiments in artificial neural networks to understand segmentation learning, by systematically restricting or manipulating different factors.
