Seeing Stuff: Perceiving Materials and their Properties

Project Information

STUFF

Grant agreement ID: 101098225

DOI

10.3030/101098225

EC signature date 29 June 2023

Start date 1 October 2023

End date 30 September 2028

Funded under

• European Research Council (ERC)

Total cost

€ 2 499 711,00

EU contribution

€ 2 499 711,00

Coordinated by

JUSTUS-LIEBIG-UNIVERSITAET GIESSEN
Germany

Periodic Reporting for period 1 - STUFF (Seeing Stuff: Perceiving Materials and their Properties)

Reporting period: 2023-10-01 to 2026-03-31

Summary of the context and overall objectives of the project

Practically every task that we perform involves interacting with our physical surroundings. Walking, cooking, shopping, even just picking something up: all these tasks require us to perceive and understand the physical properties of the objects, surfaces and materials around us. Yet despite this, how the human mind infers and represents the physical properties of materials remains scientifically poorly understood. We know that it’s a major computational challenge because the raw sensory signals about the outside world are highly ambiguous, and even state-of-the-art robots struggle to achieve the fluidity, robustness and dexterity of human behaviour.

The overall objective of this project is measure and model the fundamental mind and brain processes that allow us to recognise materials, infer their physical properties, mentally simulate or visualise how we expect them to behave, and physically interact with them using our hands. The project draws on many different branches of research: experimental and cognitive psychology, neuroscience, computational modelling—including the latest machine learning and AI techniques—and even the arts, as part of the project involves investigating how sculptors depict materials.

The main outcomes will be discoveries in basic research, especially in how the perceptual, cognitive and motor systems function. Yet in the long run, these discoveries may also facilitate the development of better robots and even facilitate the diagnosis and ultimately treatment of patients suffering from sensory or motor deficits as a result of brain injury or stroke.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The project is divided into five main work packages with dedicated subgoals. The first deals with the visual estimation of the optical properties of materials like glossiness or translucency, which provide important first indications of an object’s properties even before we reach out and touch them. So far, we have measured and modelled how we use local colour gradients in images to infer several different physical properties of surfaces, including glossiness, 3D shape, and surface colour.

The second work package deals with how we infer the mechanical properties of materials, like how hard or soft they are, especially from the way they move or change shape over time. So far, we have developed models that can predict the perceptual properties of materials from images and movies.

The third work package investigates how the physical properties of objects and materials impact the way that we grasp and interact with them. Previous work on grasping has typically focussed on extremely limited objects like artificial cubes and cylinders, and has often restricted grasping to the so-called “precision grip”—the thumb-and-index finger technique that humans use when holding small items, like a needle. In this work package, we have developed novel methods to measure exactly how we grasp everyday objects using our whole hands, and we are currently in the process of collecting a large dataset of such interactions.

The fourth work package investigates how sculptors make one material (e.g. marble) look like other materials (e.g. skin, cloth). This can give us unique insights into the 3D cues that the brain uses to work out physical properties. So far, we have performed studies showing how even tiny local features, like a little ripple can allow the brain to infer the presence of cloth over extended surface regions, implying the brain has sophisticated mechanisms for interpolating local cues.

The last work package is trying to work out how we imagine, visualise and predict physical events as they unfold over time. Prompted by a momentary snapshot of a scene—like a precariously balanced stack of plates—we can usually play out a “mental movie” of what will happen next so we can work out which direction and speed we expect objects to move, whether they will break, how far they will scatter and so on. To investigate this, so far, we have developed a computational model that can predict how we mentally rotate objects in our “mind’s eye”.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

As material perception is a relatively understudied field, practically every result we generate in this project is going beyond the state of the art in current understanding.

One important result that is worth highlighting is our recent work creating the world’s most comprehensive characterisation of the representation of materials in the human mind to date. While previous studies tended to focus on extremely limited ranges of materials and properties, we went through all words in the English language to put together a dataset of images of 200 different categories of material (the “STUFF” dataset). We then used an online experiment to gather about 1.8 million perceptual comparisons between these images. Using data-driven computational modelling we were able derive from these judgments a set of 36 “core dimensions” of material representation in the human mind. Importantly, we found that these dimensions were intuitively understandable, revealing specific physical characteristics, like graininess, minerality or swirliness that human observers use to compare and contrast materials with one another. This work was published in 2025 in the prestigious peer-reviewed journal PNAS and is available via open access for anyone to read.

Since then, we have used functional magnetic resonance imaging (fMRI) to record brain responses the images, so we can work out how the “core dimensions” are mapped out across the human cortex. This is the most extensive mapping of the brain’s representation of materials and their properties to date. We are currently analysing the data and preparing the work for publication.

The primary impact of this work is on basic research into brain function. It advances the state of the art in our scientific understanding of how networks of neurons and brain regions extract and represent information about materials. It helps us to understand how raw sensory signals that are highly ambiguous and uninformative are organised and transformed into useful representations for cognition, multi-sensory integration and ultimately behaviour. However, in the long run the discoveries may also be helpful in understanding the particular patterns of deficits that people suffer from when they get localised brain damage, for example due to brain trauma or stroke. This could one day potentially help in the design of rehabilitation strategies for patients.