Final Report Summary - PRISM (Perceptual Representation of Illumination, Shape and Material)
PRISM: Perceptual Representation of Illumination, Shape and Material
http://prism-network.eu/
Project Coordinator:
Prof. Roland W. Fleming (roland.w.fleming@psychol.uni-giessen.de)
Network Manager:
Dr. Matteo Valsecchi (matteo.valsecchi@gmail.com)
About the PRISM network:
PRISM was an EU-funded research and training network that brings together ten leading academic and industrial partners from across Europe to understand how the brain represents the physical properties of objects, surfaces and lighting in the surrounding world. It was a Marie-Curie Initial Training Network, funded by the FP7 PEOPLE program, providing research and training opportunities for 14 Early Stage Researchers (ESRs) as part of their doctoral training.
Background and Objectives:
The visual system provides us with a richly detailed representation of our surrounding world, specifying the material properties and 3D shapes of objects, and the way light is distributed throughout the scene. These three factors (illumination, shape and material) interact in complex ways to determine the intensity of any surface patch in a retinal (or digital) image. This poses a considerable challenge: how does the brain separate images into their distinct physical causes to recover a richly detailed representation of the world? This problem is vital in guiding actions and making biologically important decisions (e.g. judging if food is edible). Moreover, visual ‘look and feel’ plays a critical role in industrial design, computer graphics, architecture and other industries. Despite this, we had a poor understanding of how the brain estimates the properties of objects and illumination. Previous work had typically focussed on one or two of the 3 causal factors independently, and from the viewpoint of a specific discipline. By contrast, PRISM took an integrative approach, to understand how the brain separates the image into distinct physical causes and creates a richly detailed representation of the world, by looking at how all three factors interact simultaneously. PRISM was radically interdisciplinary, uniting experts from psychology, neuroscience, computer science, engineering and physics to understand both the synthesis and analysis of shape, shading and materials. PRISM spanned sectors by bringing together researchers and developers from eight leading Universities and two industrial partners, enabling impact in basic research, technology and the creative industries. By delivering early-career training embedded in a cutting-edge research programme, PRISM aimed to 1) springboard the next generation of interdisciplinary researchers on perceptual representations of 3D scenes 2) cement long-term collaborations between sectors to enhance European perception research and its applications. PRISM’s research programme targeted the following research objectives:
1. To measure how shape, material properties and illumination are represented in the primate brain using behavioural, physiological and brain imaging techniques.
2. To identify image cues that can be used for inferring shape, materials and illumination from both natural and digital images.
3. To determine perceptual parameterizations of shape, illumination and materials, which describe their natural degrees of variation and interactions in psychological—as opposed to physical—terms, with applications in basic and applied research.
4. To use insights from perception to guide the design of shape, illumination and materials in both real- world and digital simulation scenarios.
Work Conducted and Main Results:
The fourteen ESRs conducted a wide variety of research projects ranging from behavioural experiments on the perception of the properties of objects and materials (e.g. colour, gloss, viscosity); to mathematical analyses of the physical behaviour of light-surface interactions, with applications in computer graphics; to advanced brain imaging techniques for uncovering the anatomical and physiological mechanisms involved in surface perception. The network’s research programme was divided into interdisciplinary “Work Packages” (WPs) that pooled expertise from across network to address large-scale research goals and expose the ESRs to a wide gamut of techniques and methodologies. WP1 dealt with low-level sensory cues for estimating scene properties. We successfully investigated the role of colour, motion and shape in the perception of surface material properties, shape and illumination. Some of the main results include: (1) experiments that explain how we attribute colours to objects—like autumn leaves—that consists of multiple different colours in spatial patterns across their surface. (2) Experiments showing how the reflectance properties of a surface (whether it is mirror-like or matte) affect the perception of its 3D shape when the object is in motion. (3) Mathematical and statistical analysis of the bidirectional distribution function, which provides a complete physical description of the way that surfaces reflect light, re-expressed in terms of its influence on the properties of images of surfaces. WP2 dealt with mid-level processes that enable the brain to separate images into distinct physical causes and how these different factors interact. The main results include: (1) experiments showing how the visual system combines information from shape, motion and the optical properties of materials to work out the physical properties of viscous liquids; (2) Experiments using advanced functional and structural brain imaging techniques to identify the mechanisms than combine information across different visual brain areas and (3) experiments explaining how illumination and shape interact to influence the perception of surface glossiness. WP3 investigated high-level representations of illumination, shape and material for perception and action. The main results include: (1) experiments that identified the perceptual dimensions of surface reflectance based on participants adjusting the reflectance of one object to match another; (2) experiments that identified the perceptual dimensions of complex illumination in patterns in real and depicted scenes and (3) electrophysiological recordings from shape-sensitive areas in cortex, which have identified basic elements in a visual code for shape.
Impact:
By combining state-of-the-art theoretical and experimental techniques from perceptual psychology, neuroscience, computer science and industrial design, the long-term result of the project will be to redefine the scientific concepts of objects, materials and lighting from a human-centred perspective. Through this, we believe the results have made a significant impact on European competitiveness in all areas for which the visual appearance is important (e.g. industrial design, architecture, manufacture, computer imagery). By bridging the gap between real and digital artefacts, we have provided researchers with new ways to formulate problems in creation and manufacture. In the short-term PRISM provided a mechanism for structuring collaboration and training across Europe through shared research projects and network-wide training events. In the long term, PRISM has also formed a community of researchers with cross-discipline expertise and exposure to the R&D needs of European industry, who can change the way key challenges are formulated in both academia and industry.
http://prism-network.eu/
Project Coordinator:
Prof. Roland W. Fleming (roland.w.fleming@psychol.uni-giessen.de)
Network Manager:
Dr. Matteo Valsecchi (matteo.valsecchi@gmail.com)
About the PRISM network:
PRISM was an EU-funded research and training network that brings together ten leading academic and industrial partners from across Europe to understand how the brain represents the physical properties of objects, surfaces and lighting in the surrounding world. It was a Marie-Curie Initial Training Network, funded by the FP7 PEOPLE program, providing research and training opportunities for 14 Early Stage Researchers (ESRs) as part of their doctoral training.
Background and Objectives:
The visual system provides us with a richly detailed representation of our surrounding world, specifying the material properties and 3D shapes of objects, and the way light is distributed throughout the scene. These three factors (illumination, shape and material) interact in complex ways to determine the intensity of any surface patch in a retinal (or digital) image. This poses a considerable challenge: how does the brain separate images into their distinct physical causes to recover a richly detailed representation of the world? This problem is vital in guiding actions and making biologically important decisions (e.g. judging if food is edible). Moreover, visual ‘look and feel’ plays a critical role in industrial design, computer graphics, architecture and other industries. Despite this, we had a poor understanding of how the brain estimates the properties of objects and illumination. Previous work had typically focussed on one or two of the 3 causal factors independently, and from the viewpoint of a specific discipline. By contrast, PRISM took an integrative approach, to understand how the brain separates the image into distinct physical causes and creates a richly detailed representation of the world, by looking at how all three factors interact simultaneously. PRISM was radically interdisciplinary, uniting experts from psychology, neuroscience, computer science, engineering and physics to understand both the synthesis and analysis of shape, shading and materials. PRISM spanned sectors by bringing together researchers and developers from eight leading Universities and two industrial partners, enabling impact in basic research, technology and the creative industries. By delivering early-career training embedded in a cutting-edge research programme, PRISM aimed to 1) springboard the next generation of interdisciplinary researchers on perceptual representations of 3D scenes 2) cement long-term collaborations between sectors to enhance European perception research and its applications. PRISM’s research programme targeted the following research objectives:
1. To measure how shape, material properties and illumination are represented in the primate brain using behavioural, physiological and brain imaging techniques.
2. To identify image cues that can be used for inferring shape, materials and illumination from both natural and digital images.
3. To determine perceptual parameterizations of shape, illumination and materials, which describe their natural degrees of variation and interactions in psychological—as opposed to physical—terms, with applications in basic and applied research.
4. To use insights from perception to guide the design of shape, illumination and materials in both real- world and digital simulation scenarios.
Work Conducted and Main Results:
The fourteen ESRs conducted a wide variety of research projects ranging from behavioural experiments on the perception of the properties of objects and materials (e.g. colour, gloss, viscosity); to mathematical analyses of the physical behaviour of light-surface interactions, with applications in computer graphics; to advanced brain imaging techniques for uncovering the anatomical and physiological mechanisms involved in surface perception. The network’s research programme was divided into interdisciplinary “Work Packages” (WPs) that pooled expertise from across network to address large-scale research goals and expose the ESRs to a wide gamut of techniques and methodologies. WP1 dealt with low-level sensory cues for estimating scene properties. We successfully investigated the role of colour, motion and shape in the perception of surface material properties, shape and illumination. Some of the main results include: (1) experiments that explain how we attribute colours to objects—like autumn leaves—that consists of multiple different colours in spatial patterns across their surface. (2) Experiments showing how the reflectance properties of a surface (whether it is mirror-like or matte) affect the perception of its 3D shape when the object is in motion. (3) Mathematical and statistical analysis of the bidirectional distribution function, which provides a complete physical description of the way that surfaces reflect light, re-expressed in terms of its influence on the properties of images of surfaces. WP2 dealt with mid-level processes that enable the brain to separate images into distinct physical causes and how these different factors interact. The main results include: (1) experiments showing how the visual system combines information from shape, motion and the optical properties of materials to work out the physical properties of viscous liquids; (2) Experiments using advanced functional and structural brain imaging techniques to identify the mechanisms than combine information across different visual brain areas and (3) experiments explaining how illumination and shape interact to influence the perception of surface glossiness. WP3 investigated high-level representations of illumination, shape and material for perception and action. The main results include: (1) experiments that identified the perceptual dimensions of surface reflectance based on participants adjusting the reflectance of one object to match another; (2) experiments that identified the perceptual dimensions of complex illumination in patterns in real and depicted scenes and (3) electrophysiological recordings from shape-sensitive areas in cortex, which have identified basic elements in a visual code for shape.
Impact:
By combining state-of-the-art theoretical and experimental techniques from perceptual psychology, neuroscience, computer science and industrial design, the long-term result of the project will be to redefine the scientific concepts of objects, materials and lighting from a human-centred perspective. Through this, we believe the results have made a significant impact on European competitiveness in all areas for which the visual appearance is important (e.g. industrial design, architecture, manufacture, computer imagery). By bridging the gap between real and digital artefacts, we have provided researchers with new ways to formulate problems in creation and manufacture. In the short-term PRISM provided a mechanism for structuring collaboration and training across Europe through shared research projects and network-wide training events. In the long term, PRISM has also formed a community of researchers with cross-discipline expertise and exposure to the R&D needs of European industry, who can change the way key challenges are formulated in both academia and industry.