BabyMindReader: a multivariate NIRS-EEG neural decoder to decipher newborns' inital representations of speech prosody

Project Information

BabyMindReader

Grant agreement ID: 101031716

DOI

10.3030/101031716

Project closed

EC signature date 23 April 2021

Start date 1 January 2022

End date 22 November 2024

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 183 473,28

EU contribution

€ 183 473,28

Coordinated by

UNIVERSITA DEGLI STUDI DI PADOVA
Italy

Periodic Reporting for period 1 - BabyMindReader (BabyMindReader: a multivariate NIRS-EEG neural decoder to decipher newborns' inital representations of speech prosody)

Reporting period: 2022-01-01 to 2023-12-31

Summary of the context and overall objectives of the project

When they are born, human infants are linguistic “citizens of the world” : they already possess such a wide range of broad and universal perceptual abilities that they can start learning any language(s), and they do it quickly and effortlessly during the first years of life.
How do they learn so easily? This question has been long investigated, first with behavioral methods and then with neuroimaging. Findings that emerged during the last decades have added an important piece to the picture: the acquisition of language already starts before birth. Despite a growing body of research, the neural underpinnings of this exquisitely human ability are still unclear. The overall objective of the BabyMindReader project is to shed light on this issue with an innovative interdisciplinary approach that brings together long-established experimental methods in infant language research with the most forefront technical advancements in the machine learning field. The value of the project is both theoretical and methodological, because it aims at advancing our understanding of the mechanisms underlying the very early stages of language acquisition while at the same time overcoming technical challenges behind infant multimodal neuroimaging: in particular, in this project, these questions will be investigated by recording newborns' concurrent functional near-infrared spectroscopy (fNIRS) and electroencephalograpy (EEG) signals. NIRS and EEG provide different information, in that NIRS yields information, with good spatial localization, about the functional activation of cortical areas in responses to stimuli elicited by the metabolic consumption of oxygen, while the EEG measures, with great temporal resolution, the neural activity that supports that activation. Therefore, the concurrent registration of the two modalities holds great potential for shedding light, in a all-round fashion, on the neuro-functional mechanisms underlying language acquisition. Nevertheless, it poses methodological challenges both for the experimental design and for the data analysis. Solving these challenges will be of great value for the infant neuroimaging communities, and it is one of the objectives of this project.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

To achieve the goals of the project, two experiments have been carried out on full-term newborns within the first few days of life, measuring their functional and neural responses to several classes of stimuli, including the native language (Italian), a non-native unfamiliar language (Japanese) and primate vocalizations, in particular baboon calls. While passively listening to these auditory stimuli, newborns' brain responses were measured using functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG). The experiments have been designed in a way to optimally measure both signals despite them developing over highly different timescales, i.e. over several seconds (fNIRS) and hundreds of milliseconds (EEG). In particular, long blocks of repeated stimuli were presented and used to measure the NIRS signal; within these long blocks, items were presented in a mismatch design fashion, useful to record the mismatch responses with the EEG. After recording data, data analysis has been set up through the development a multimodal and multivariate pattern analysis classifier, that pull together features from the concurrent NIRS and EEG signals. The combined classifier was first tested on a set of pilot data and showed good and statistically significant classification accuracies in more than half the tested sample, and for 80% of them classification accuracies that were larger than those achieved by the single-modalities classifiers. These routine is currently being applied on the full samples from the two experiments.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Preliminarly, the acquired NIRS-EEG data shows that babies are able to detect violations of the prosody in their native language but not in the other classes of stimuli, thus suggesting that indeed prosody is that feature of the language that is learnt prenatally over the last weeks of gestation. This knowldege holds value first of all for the general public, in particular mothers-to-be, in that it confirms that even language heard prenatally favours the postnatal stages of language acquisition. By acquiring concurrent NIRS-EEG data, it was possible to determine the functional architecture that supports this mechanism, and also the neural activity behind the functional activations, thus providing a fuller characterization compared to the state of the art, thus holding impact for developmental neuroscientists investigating these processes. Relatedly, the development of the framework for the concurrent NIRS-EEG analysis will have impact on other researchers's work, or on industrial developments in this area.

screenshot-2025-01-21-at-19-56-13.png