Crossmodal estimation of decisions, guesses and delays in decision making: a collaboration between psychology, physiology and artificial intelligence

Project Information

CrossEDGE

Grant agreement ID: 101066503

DOI

10.3030/101066503

Project terminated on 31 May 2025

EC signature date 11 July 2022

Start date 1 August 2022

End date 31 August 2026

Funded under

• Marie Skłodowska-Curie Actions (MSCA)

Total cost

No data

EU contribution

€ 203 464,32

Coordinated by

UNIVERSITEIT UTRECHT
Netherlands

Periodic Reporting for period 1 - CrossEDGE (Crossmodal estimation of decisions, guesses and delays in decision making: a collaboration between psychology, physiology and artificial intelligence)

Reporting period: 2022-08-01 to 2026-08-31

Summary of the context and overall objectives of the project

Decision‑making under time pressure is a fundamental cognitive function that shapes everyday behaviour, clinical diagnosis, education, and the design of autonomous systems. Yet, conventional behavioural analyses collapse thousands of milliseconds of neural activity into a single reaction‑time measure, obscuring the rapid succession of mental operations that actually drive a choice. The central goal of this was to combine trial-by-trial electroencephalographic (EEG) recordings with behavioral data and translate this into a computational framework capable of labeling each decision as a guess, an evidence-accumulation process, or a delayed response.

The project pursued three tightly coupled objectives:
1 - Derive a model integrating physiological and behavioral data for the characterization of cognitive events in the EEG at the trial level
2 - Use the detected EEG events to understand the cognitive processes in decision-making
3 - Combine the outputs of the first two objectives into a generative framework that predicts the proportion of each strategy used across different speed‑accuracy regimes, thereby offering a quantitative bridge between neural dynamics and behavioural policy

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

In the first phase, we developed the Hidden Multivariate Pattern (HMP) model—a method that reliably detects cognitive events on a single-trial basis. The framework combines multivariate sensor patterns with probabilistic timing models to infer latent cognitive events. HMP was validated on both simulated and open datasets. The results were published in a peer-reviewed article (Trial-by-trial detection of cognitive events in neural time-series, Weindel, van Maanen & Borst, 2025).

The second work package focused on decomposing decision-making into its constituent cognitive processes. Two experiments were conducted with around thirty participants each. The first study was published in eLife as a reviewed preprint (Decision-making components and times revealed by the single-trial electro-encephalogram, Weindel, Borst & van Maanen). This study showed that we can decompose the reaction time in a decision-making task at the trial level into different periods of decision processing. These periods were then used to fit evidence accumulation models, improving parameter estimation beyond behavioral data alone. The second dataset, in preparation, builds on this first study to extend leading decision models, the evidence accumulation models.

The third work package aimed to integrate decision models beyond evidence accumulation models, e.g. guessing, evidence accumulation, and delayed decision,. However, the EEG data did not yield sufficiently distinct neural signatures for these strategies. As a result, the project pivoted to focus on strengthening and generalizing the HMP framework. This pivot preserved the core scientific contribution and led to a tool already being adopted by several European research groups.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The project showed that the Hidden Multivariate Pattern (HMP) model markedly improves the interpretability of EEG data. In the second objective, HMP not only quantified the number of cognitive events underlying each decision but also pinpointed their timing on a trial‑by‑trial basis. By extracting reliable single‑event signatures despite low signal‑to‑noise ratios, the HMP model appears as a powerful for cognitive‑neuroscience research. Although the current work centered on decision‑making, future studies should explore its applicability to other domains—such as language processing, motor control, and clinical diagnostics—to fully evaluate its broader impact across the neuroscientific landscape.

Representation of the Hidden Multivariate Pattern model