From Meaning to Perception: How narratives shape visual processing

Información del proyecto

VisualNarrative

Identificador del acuerdo de subvención: 101106569

DOI

10.3030/101106569

Proyecto finalizado el 30 Septiembre 2025

Fecha de la firma de la CE 25 Abril 2023

Fecha de inicio 1 Noviembre 2023

Fecha de finalización 31 Octubre 2025

Financiado con arreglo a

• Marie Skłodowska-Curie Actions (MSCA)

Coste total

Sin datos

Aportación de la UE

€ 187 624,32

Coordinado por

STICHTING RADBOUD UNIVERSITEIT
Netherlands

Periodic Reporting for period 1 - VisualNarrative (From Meaning to Perception: How narratives shape visual processing)

Período documentado: 2023-11-01 hasta 2025-10-31

Resumen del contexto y de los objetivos generales del proyecto

Humans constantly generate expectations of what will happen next. This allows us to process overwhelming sensory input more efficiently. However, to date most of studies examining the influence of expectations on sensory processing have used rather oversimplified paradigms which lack higher-order meaning, like simple stimuli associations. To overcome this barrier, this project aimed to assess how more cognitive, meaning-based expectations influence visual processing. To this aim, I used picture books as a more ecologically valid paradigm and to experimentally still manipulate the meaningfulness of stimuli (whether subsequent pictures from the book were presented in a coherent, i.e. correct order, or a shuffled order where the meaning was lost). I used state-of-the-art large language models to quantify expectations in such stimulus and relate it to visual processing. Together, this project bridged from more higher-order cognition to lower-level visual processing.

Trabajo realizado desde el comienzo del proyecto hasta el final del período abarcado por el informe y los principales resultados hasta la fecha

I have performed an eye-tracking experiment where I presented the picture books to individuals, either in a coherent or shuffled order while measuring their eye movements during free viewing. Then I used GPT-2, a large language model to quantify, from separately obtained language narratives, what was semantically salient in images, in order to understand the narratives. In parallel, I used a state-of-the-art model of visual salience (DeepGaze-II) to assess what was salient from a visual standpoint and then assessed how this related to participants gaze behaviour. I found that my new, semantic salience model, could account for differences in eye movement behaviour depending on the condition in which images were presented - when they made up a coherent sequence, participants viewed both relatively more and relatively longer at objects that were semantically salient in images. Visual salience did not capture differences in gaze behaviour between conditions. This shows that the state-of-the-art models of visual salience lack some explanatory power when accounting for gaze behaviour, especially in more naturalistic, meaningful settings. In turn, language models could offer a useful avenue forward to in general capture cognitive influences on visual processing.

Avances que van más allá del estado de la técnica e impacto potencial esperado (incluida la repercusión socioeconómica y las implicaciones sociales más amplias del proyecto hasta la fecha)

The main result of the project was that semantic salience model, as derived from language, can account for differences in eye movement behaviour, and better explain gaze behaviour in more meaningful settings.
This could lead to much further research, especially extending the paradigm into even more naturalistic settings, such as movie watching or navigation. To ensure that, the results were presented at multiple conferences and right now, the manuscript is in review for publication.