From Meaning to Perception: How narratives shape visual processing

Project Information

VisualNarrative

Grant agreement ID: 101106569

DOI

10.3030/101106569

Project terminated on 30 September 2025

EC signature date 25 April 2023

Start date 1 November 2023

End date 31 October 2025

Funded under

• Marie Skłodowska-Curie Actions (MSCA)

Total cost

No data

EU contribution

€ 187 624,32

Coordinated by

STICHTING RADBOUD UNIVERSITEIT
Netherlands

Periodic Reporting for period 1 - VisualNarrative (From Meaning to Perception: How narratives shape visual processing)

Reporting period: 2023-11-01 to 2025-10-31

Summary of the context and overall objectives of the project

Humans constantly generate expectations of what will happen next. This allows us to process overwhelming sensory input more efficiently. However, to date most of studies examining the influence of expectations on sensory processing have used rather oversimplified paradigms which lack higher-order meaning, like simple stimuli associations. To overcome this barrier, this project aimed to assess how more cognitive, meaning-based expectations influence visual processing. To this aim, I used picture books as a more ecologically valid paradigm and to experimentally still manipulate the meaningfulness of stimuli (whether subsequent pictures from the book were presented in a coherent, i.e. correct order, or a shuffled order where the meaning was lost). I used state-of-the-art large language models to quantify expectations in such stimulus and relate it to visual processing. Together, this project bridged from more higher-order cognition to lower-level visual processing.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

I have performed an eye-tracking experiment where I presented the picture books to individuals, either in a coherent or shuffled order while measuring their eye movements during free viewing. Then I used GPT-2, a large language model to quantify, from separately obtained language narratives, what was semantically salient in images, in order to understand the narratives. In parallel, I used a state-of-the-art model of visual salience (DeepGaze-II) to assess what was salient from a visual standpoint and then assessed how this related to participants gaze behaviour. I found that my new, semantic salience model, could account for differences in eye movement behaviour depending on the condition in which images were presented - when they made up a coherent sequence, participants viewed both relatively more and relatively longer at objects that were semantically salient in images. Visual salience did not capture differences in gaze behaviour between conditions. This shows that the state-of-the-art models of visual salience lack some explanatory power when accounting for gaze behaviour, especially in more naturalistic, meaningful settings. In turn, language models could offer a useful avenue forward to in general capture cognitive influences on visual processing.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The main result of the project was that semantic salience model, as derived from language, can account for differences in eye movement behaviour, and better explain gaze behaviour in more meaningful settings.
This could lead to much further research, especially extending the paradigm into even more naturalistic settings, such as movie watching or navigation. To ensure that, the results were presented at multiple conferences and right now, the manuscript is in review for publication.