From Meaning to Perception: How narratives shape visual processing

Informazioni relative al progetto

VisualNarrative

ID dell’accordo di sovvenzione: 101106569

DOI

10.3030/101106569

Progetto concluso il 30 Settembre 2025

Data della firma CE 25 Aprile 2023

Data di avvio 1 Novembre 2023

Data di completamento 31 Ottobre 2025

Finanziato da

• Marie Skłodowska-Curie Actions (MSCA)

Costo totale

Nessun dato

Contributo UE

€ 187 624,32

Coordinato da

STICHTING RADBOUD UNIVERSITEIT
Netherlands

Periodic Reporting for period 1 - VisualNarrative (From Meaning to Perception: How narratives shape visual processing)

Periodo di rendicontazione: 2023-11-01 al 2025-10-31

Sintesi del contesto e degli obiettivi generali del progetto

Humans constantly generate expectations of what will happen next. This allows us to process overwhelming sensory input more efficiently. However, to date most of studies examining the influence of expectations on sensory processing have used rather oversimplified paradigms which lack higher-order meaning, like simple stimuli associations. To overcome this barrier, this project aimed to assess how more cognitive, meaning-based expectations influence visual processing. To this aim, I used picture books as a more ecologically valid paradigm and to experimentally still manipulate the meaningfulness of stimuli (whether subsequent pictures from the book were presented in a coherent, i.e. correct order, or a shuffled order where the meaning was lost). I used state-of-the-art large language models to quantify expectations in such stimulus and relate it to visual processing. Together, this project bridged from more higher-order cognition to lower-level visual processing.

Lavoro eseguito dall’inizio del progetto fino alla fine del periodo coperto dalla relazione e principali risultati finora ottenuti

I have performed an eye-tracking experiment where I presented the picture books to individuals, either in a coherent or shuffled order while measuring their eye movements during free viewing. Then I used GPT-2, a large language model to quantify, from separately obtained language narratives, what was semantically salient in images, in order to understand the narratives. In parallel, I used a state-of-the-art model of visual salience (DeepGaze-II) to assess what was salient from a visual standpoint and then assessed how this related to participants gaze behaviour. I found that my new, semantic salience model, could account for differences in eye movement behaviour depending on the condition in which images were presented - when they made up a coherent sequence, participants viewed both relatively more and relatively longer at objects that were semantically salient in images. Visual salience did not capture differences in gaze behaviour between conditions. This shows that the state-of-the-art models of visual salience lack some explanatory power when accounting for gaze behaviour, especially in more naturalistic, meaningful settings. In turn, language models could offer a useful avenue forward to in general capture cognitive influences on visual processing.

Progressi oltre lo stato dell’arte e potenziale impatto previsto (incluso l’impatto socioeconomico e le implicazioni sociali più ampie del progetto fino ad ora)

The main result of the project was that semantic salience model, as derived from language, can account for differences in eye movement behaviour, and better explain gaze behaviour in more meaningful settings.
This could lead to much further research, especially extending the paradigm into even more naturalistic settings, such as movie watching or navigation. To ensure that, the results were presented at multiple conferences and right now, the manuscript is in review for publication.