Characterizing information integration in reinforcement learning: a neuro-computational investigation

Project Information

INFORL

Grant agreement ID: 948671

DOI

10.3030/948671

EC signature date 17 February 2021

Start date 1 September 2021

End date 31 August 2026

Funded under

• EXCELLENT SCIENCE - European Research Council (ERC)

Total cost

€ 1 500 000,00

EU contribution

€ 1 500 000,00

Coordinated by

CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE CNRS
France

Periodic Reporting for period 2 - INFORL (Characterizing information integration in reinforcement learning: a neuro-computational investigation)

Reporting period: 2023-03-01 to 2024-08-31

Summary of the context and overall objectives of the project

Reinforcement learning (RL) characterizes how we adaptively learn, by trial and errors, to select actions that maximize the occurrence of rewards, and minimize the occurrence of punishments. The behavioural, computational and neurobiological features of reinforcement-learning outcomes have been extensively studied in humans and other animals, but mostly in a standard context where the decision-maker only face one outcome (usually a reward) associated with the option they chose. As a consequence, little is known about how we prioritize, filter or value richer and more complex outcome information in RL, and how we subjectively evaluate the quality of information that supports our decision. This project proposes to address this gap, and hypothesizes that humans do learn from complex outcome information (multiple samples), but that computational limitations and affective biases curb information integration.
The prioritization, filtering and biased integration of the information carried by the outcomes of our decision may underpin critical (and undesirable) behavioral phenomena like confirmatory biases, overconfidence, and ultimately complex social phenomena like political polarization.
The objectives of our project are to investigate these cognitive processes in a well-controlled laboratory environment, to decipher the behavioral, computational and neurobiological aspects of information integration in reinforcement-learning and its biases and limitations.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

To start the project, we have comprehensively re-evaluated the impact of key extension of standard reinforcement-learning setups featured in our recent work: the manipulation of the valence of choice outcomes (reward vs punishment), the manipulation of the quantity of information (simple outcome vs revealing the outcome associated with the unchosen option), and the elicitation of confidence in choices. This conceptual and empirical investigation has established and consolidated the existence of critical information integration biases and limitations: namely, that 1) information-integration is context-dependent; 2) that information is integrated in a confirmatory way; and 3) that confidence in choice accuracy is dominated by the information related to the chosen option, leading to overconfidence and valence-dependent biases in confidence (i.e. the fact that confidence is higher in gain than loss frames). We have proposed computational models that explain those three observations, and account for a large amount of empirical data. Using neuroimaging, we have dissociated two brain network, that seem to integrate two kinds of information during reinforcement-learning: a dorsal parieto-frontal network, that seems to encode relatively unbiased information, akin to learning uncertainty, and a ventro-limbic prefrontal network, that integrates the biases in information integration.
Now, we are designing (even) more complex reinforcement-learning setups where information is even richer, which we will use to investigate the cost benefits tradeoffs associated with integrating (more) information in reinforcement-learning.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Biases and limitations in human reinforcement-learning had been surprisingly neglected, especially when compared to the vast literature documenting biases and heuristics in reasoning and decision-making. Our line of work constitute a clear progress toward better documenting human behavior in reinforcement-learning, beyond its standard, normative description .
With the development of reinforcement-learning paradigms that features richer and more complex information, we expect to identify and characterize key tradeoffs between the benefits of integrating more information to guide behavior, and the computational costs engaged in integrating and treating this information.