Community Research and Development Information Service - CORDIS


LANGDYN Report Summary

Project ID: 669820
Funded under: H2020-EU.1.1.

Periodic Reporting for period 1 - LANGDYN (Language dynamics: a neurocognitive approach to incremental interpretation)

Reporting period: 2015-10-01 to 2017-03-31

Summary of the context and overall objectives of the project

Understanding spoken language involves a complex set of processes that transform the auditory input into a structured and meaningful interpretation. The goal of our research is to understand the nature of the intermediate processes involved in the transition from early perceptual processing through different representational states to the development of a meaningfully structured utterance, the dynamic spatio-temporal relationship between these processes, and their evolution over time.

To achieve this we combine advanced techniques from neuroimaging, multivariate statistics and computational linguistics to probe directly the dynamic patterns of neural activity across the brain.
We carry out combined electroencephalography and magnetencephalography (EMEG) imaging to capture the real-time electrophysiological activity of the brain, and use Representational Similarity Analysis (RSA) and related multivariate techniques to probe the different types of neural computation that support these dynamic processes of incremental interpretation. Computational linguistic analyses of language corpora allow us to build quantifiable models of different dimensions of language interpretation – from phonetics and phonology to argument structure and anaphora -- and to test for their presence, using RSA, as the utterance unfolds in real time.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

We have initially focused on the processing of spoken words which involves complex processes that enable listeners to rapidly and seemingly effortlessly transform the auditory input into a meaningful interpretation. This effortless transition occurs on millisecond timescales, with remarkable speed and accuracy and without any awareness of the complex computations involved. Our research has revealed the real-time neural dynamics of these processes by collecting data about listeners’ brain activity as they hear spoken words. Using novel statistical models of different aspects of the recognition process, we have been able to locate directly which parts of the brain are accessing the stored form and meaning of words and how the competition between different word candidates is resolved neurally in real time. This gives us a uniquely differentiated picture of the neural substrate for the first 500 ms of spoken word recognition. This study, published in the Journal of Neuroscience (doi: 10.1523/JNEUROSCI.2858-16.2016) was a great advance in understanding the neural processes involved in how speech activates word meaning, and we are now extending this research to determine the neural mechanisms that support the transformation from speech input to meaning using neural oscillations and brain connectivity.
In further research we have investigated the neurocognitive processes which combine the meanings of individual words into larger phrases (e.g. 'yellow banana'). Once again, we combined electroencephalography and MEG (EMEG), enabling good spatiotemporal resolution of the brain signal. The data has been analysed using multivariate statistical methods similar to the single words. Preliminary results suggest that presence of the context word ('yellow') has unexpectedly strong predictive effects on the lexical and semantic competition related to the second word (“banana”) potentially enabling very early access to the second word's semantic competitors.

We have also been investigating computations involved in the incremental integration of words within spoken sentences in two studies. Both are EMEG experiments. One study was designed to investigate the way that the brain incrementally develops cognitive representations and computations for multiple linguistic aspects during speech comprehension. We explored how verb information constrains the integration of a subsequent complement phrase, using detailed statistical models of verb syntactic subcategorization and semantic selection preferences derived from language corpora. Relating the syntactic and semantic measures to the spatio-temporal dynamics of brain activity showed that the verb-constrained syntactic computations recruit a left fronto-temporal language network whereas verb semantic computations elicit activity in a more distributed bilateral semantic network. This suggests that the brain generates incremental syntactic and semantic predictions at the verb in the sentence in parallel via pre-activation of a likely syntactic frame. This work has been recently published in Journal of Neuroscience (doi: 10.1523/JNEUROSCI.2858-16.2016)

The second study investigates how lexically-driven expectations and syntactic complexity interact during the incremental interpretation of spoken sentences. We manipulated the complex long-distance grammatical structure-building that can occur in natural sentences, using embedded relative clauses. Our Representational Similarity analyses show how the lexical access of the first verb in the sentence is influenced by the likelihood of taking a direct object when it is encountered, and also subsequently in the effect of surprisal observed on encountering the main verb which comes later in the sentences. This project is ongoing.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

This project goes beyond the state-of-the-art in both its conceptualisation of how the dynamic richness and complexity of spoken language comprehension in the brain can be studied, and also in the methods we have developed and employed to quantify complex linguistic information and relate it to dynamic processes in the brain. In particular, we model phonological-lexical, lexical-syntactic, and lexical-semantic properties of our stimuli using novel and detailed probabilistic models, which we have derived from large-scale gating studies and from computational linguistic resources. Such an approach goes beyond the state-of-the-art in studies of the cognitive neuroscience of language, which have typically looked at broader contrasts between particular experimental conditions. For example, our use of detailed, multivariate probabilistic models of verb subcategorization information derived from corpus data allows us to investigate syntactic integration effects using naturalistic stimuli, without requiring sentences with syntactic violations or other kinds of manipulations that do not occur in normal speech comprehension. We anticipate that this fine-grained analysis will enable better detection of language deficits and their remediation.
Follow us on: RSS Facebook Twitter YouTube Managed by the EU Publications Office Top