Combining electrophysiology and cognitive computational modeling in research on meaning in language

Project Information

Modeling ERPs

Grant agreement ID: 658999

DOI

10.3030/658999

Project closed

EC signature date 13 April 2015

Start date 1 August 2015

End date 31 July 2017

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 159 795,60

EU contribution

€ 159 795,60

Coordinated by

FREIE UNIVERSITAET BERLIN
Germany

Periodic Reporting for period 2 - Modeling ERPs (Combining electrophysiology and cognitive computational modeling in research on meaning in language)

Reporting period: 2016-08-01 to 2017-07-31

Summary of the context and overall objectives of the project

Language and meaning processing have been investigated with event-related brain potentials (ERPs), providing direct time-resolved measures of electrical brain activity, and with neural network models, providing mechanistic implementations of the assumed processes. However, there has been very little contact between these fields, even though a combination of both methods could be highly beneficial.
Specifically, a brain signal known as the N400 (a signal that was first described as a response to the presentation of a semantically unexpected word in a sentence) has aroused much interest for its promise to shed light on the brain basis of meaning processing. However, in spite of over 1000 studies using the N400 as a dependent variable, the representations and processes that underlie it remain incompletely understood.
The present project aims to provide an implemented theory of the N400’s functional basis and thus a theory of implicit meaning processing in the brain.
Concerning training, the main goal of the project was to provide the researcher with neural network modeling skills in order to link explicit computational models to neural signals.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

We linked the N400 to a deep neural network model that predates many current state of the art language processing models by over 20 years and has renewed relevance in the context of recent breakthroughs in the field of deep learning.
Specifically, we provide both support for and formalization of the view that the N400 reflects the stimulus-driven update of a representation of sentence meaning – one that implicitly and probabilistically represents all aspects of meaning as it evolves in real time during comprehension. We do so by presenting an explicit computational model of this process, showing that it can account for a broad range of empirically observed N400 effects which have been difficult to capture within a single theoretical account and have previously been taken to support diverse and sometimes conflicting N400 theories.
We also show that the model does not predict N400 effects in situations where such effects are not observed empirically (e.g. in response to syntactic irregularities), demonstrating the model's specificity. Furthermore, model comparison with a simple recurrent network model (SRN) trained to predict the next word based on the preceding context (which has also been proposed to account for the N400 component) shows that the SRN fails to capture the N400 data pattern in several instances where our model is in line with the empirical data. Thus, to date our model accounts for the widest range of different N400 effects reported in the literature.
With respect to training, the researcher acquired skills in neural network modeling enabling the described research.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

"The present work provides a computationally explicit and precise theoretical formulation of the N400’s functional basis. This theory of the N400 and thus the brain’s implicit representation of meaning can provide a solid foundation for future studies to further delineate the distinct roles of the N400 as well as other signals evoked during language processing, and can serve as the basis for examining the roles of experience and individual differences in deriving meaning from language. It also provides a computationally explicit account of a number of phenomena that have been taken to suggest that language comprehension does not always result in completely accurate representations of the linguistic input, but may sometimes give rise to representations that are plausibility-sensitive and just ""good enough"", as a result of the inherent uncertainty in real language processing situations.
The project thus provides the basis for extensive follow-up research that has resulted in continued funding and ongoing collaborations enabling the researcher to further develop and establish this line of research."

Simulating N400 amplitudes as change in an implicit and probabilistic representation of meaning