Periodic Reporting for period 1 - VINCI (The Value of Information and Choice to Improve Control.)
Reporting period: 2020-01-01 to 2021-12-31
Opportunities to influence the world through one’s own actions are a major predictor of high quality of life in humans and other animals. For example employees who can choose how to organize their daily tasks show lower levels of stress and longer life expectancy than those without this control. This improved quality of life may result directly from better outcomes due to choices made, but it also raises the possibility that choice opportunities may themselves have evolved to be desirable. Thus choice-seeking behavior may be intrinsically motivated by a kind of intrinsic reward (IR, also refer to as cognitive reward) distinct from extrinsic rewards (ER - e.g. food or money), with the former not necessarily leading immediately to the achievement of extrinsic goals. IRs may facilitate survival and well-being by improving long-run ER intake. One way this has been proposed to work is that behaviors such as choice-seeking promote the search for states that maximize the agent’s ability to survive and reproduce, thereby selecting individuals with an innate bias (or preference) for these types of behaviors. Experimental evidences also support the idea that choice opportunities are themselves rewarding. This choice-seeking preference may be related to the broader desire for a “sense of control”, which is mediated by the opportunity to choose and whose impairment has been linked to neuropsychiatric disorders. However none of these experiments clearly determines how learning and representation of ERs relate to choice-seeking, nor whether choice-seeking changes once ER associations are learned. Despite the importance of choice-seeking and desire for control in our daily lives, how IRs such as choice opportunities are encoded in the brain remains a major unanswered question. The VINCI project seeks to better characterize the behavioral and neural mechanisms involved in the encoding of IRs promoting intrinsic motivation.
Improving control over ER intake builds on a main strategy: choosing the most valuable option to maximize gains. It is therefore unclear whether IRs can be easily dissociated behaviorally from ERs. If they can be, understanding how these different reward types interact would be revealing about how extrinsic (triggered by ER) and intrinsic (triggered by IR) motivations are combined to guide real-world actions. Evidence suggests that IRs may be encoded by the dopamine (DA) system in a way similar to how DA neurons encode ERs. Midbrain DA neurons send massive projections to the prefrontal cortex (PFC) and the striatum and are central in coding and learning of ERs. These neurons encode a large range of rewarding experiences and generate “reward prediction errors” to signal unpredicted outcomes and changes from conditioned expected value. Under the neural common currency theory, the DA system may compare expected outcomes on a common value scale in order to evaluate which option is preferable. We hypothesize that IRs are encoded in the DA system in a manner consistent with neural common currency theory; that is DA activity should consistently reflect subjects’ preferences when IRs are pitted against different ERs. There is suggestive evidence that basal ganglia nuclei (receiving dense DA inputs), such as the striatum, are involved in encoding preference for choice and IRs.
However none of the previous studies systematically dissociated choice availability from ER intake or tested the dedicated neural networks involved in representing these different forms of reward and motivation. We developed a behavioral task that permits dissociating the value of choice opportunities and ER intake. We rigorously explored these different forms of rewards in primates (humans and monkeys, two species with a comparable DA system that differs from rodents). The goal was to test whether IRs can trigger a distortion of reward intake like a ‘DA bonus’ (i.e. a DA effect not explained by ER expectation), and how this recruits similar or different neural networks.
Improving control over ER intake builds on a main strategy: choosing the most valuable option to maximize gains. It is therefore unclear whether IRs can be easily dissociated behaviorally from ERs. If they can be, understanding how these different reward types interact would be revealing about how extrinsic (triggered by ER) and intrinsic (triggered by IR) motivations are combined to guide real-world actions. Evidence suggests that IRs may be encoded by the dopamine (DA) system in a way similar to how DA neurons encode ERs. Midbrain DA neurons send massive projections to the prefrontal cortex (PFC) and the striatum and are central in coding and learning of ERs. These neurons encode a large range of rewarding experiences and generate “reward prediction errors” to signal unpredicted outcomes and changes from conditioned expected value. Under the neural common currency theory, the DA system may compare expected outcomes on a common value scale in order to evaluate which option is preferable. We hypothesize that IRs are encoded in the DA system in a manner consistent with neural common currency theory; that is DA activity should consistently reflect subjects’ preferences when IRs are pitted against different ERs. There is suggestive evidence that basal ganglia nuclei (receiving dense DA inputs), such as the striatum, are involved in encoding preference for choice and IRs.
However none of the previous studies systematically dissociated choice availability from ER intake or tested the dedicated neural networks involved in representing these different forms of reward and motivation. We developed a behavioral task that permits dissociating the value of choice opportunities and ER intake. We rigorously explored these different forms of rewards in primates (humans and monkeys, two species with a comparable DA system that differs from rodents). The goal was to test whether IRs can trigger a distortion of reward intake like a ‘DA bonus’ (i.e. a DA effect not explained by ER expectation), and how this recruits similar or different neural networks.
The key idea was to use a two-stage task where we can manipulate IRs and ERs independently in order to characterize the conditions under which IRs influence behavior. Our main design involves to independently manipulate choice availability with subjects making sequential decisions within the two-stage structure. In the first stage, subjects make an initial decision (meta-choice) to accept (free trials) or reject (forced trials) the opportunity to freely choose between fractal images associated to ERs during a second-stage.
Our results show a clear preference for free over forced trials in humans since subjects selected in about 70% of the case free trials during the first-stage of the task independently from the ER accumulated. Modeling investigation using reinforcement learning models shows that it was necessary to incorporate overvaluation of extrinsic rewards obtained from free actions to account for choice-seeking behavior. Behavioral results in monkey, performing a similar task, suggested the same pattern.
To test whether DA system can be involved in the encoding of IRs we first asked how deficits of this neural system can affect preference for IR. We first investigated performance of Parkinson’s disease (PD) patients (who suffer from a loss of DA neurons) with (ON) and without (OFF) their treatment using the same task. These patients were subjects using DA medication to compensate their deficits or PD patients who underwent deep brain stimulation surgery of the subthalamic nucleus.
Data indicates that during the first-stage overall preference of the patients for free trials increase after reestablishment of their treatments (i.e. ON treatment > OFF treatment). This set of results postulate a key role of the DA system in intrinsic reward assessment and intrinsic motivation.
Finally we have developed an experimental set-up that will ultimately allow us, in non-human primates, to investigate specifically the calcium-activity of the midbrain DA neurons along with simultaneous electrophysiological recordings of neurons associated to the DA system. The goal is to gain insights about the correlation between the reward encoding by DA neurons and the associated modulation of downstream neurons involved in motivational and learning mechanisms.
Our results show a clear preference for free over forced trials in humans since subjects selected in about 70% of the case free trials during the first-stage of the task independently from the ER accumulated. Modeling investigation using reinforcement learning models shows that it was necessary to incorporate overvaluation of extrinsic rewards obtained from free actions to account for choice-seeking behavior. Behavioral results in monkey, performing a similar task, suggested the same pattern.
To test whether DA system can be involved in the encoding of IRs we first asked how deficits of this neural system can affect preference for IR. We first investigated performance of Parkinson’s disease (PD) patients (who suffer from a loss of DA neurons) with (ON) and without (OFF) their treatment using the same task. These patients were subjects using DA medication to compensate their deficits or PD patients who underwent deep brain stimulation surgery of the subthalamic nucleus.
Data indicates that during the first-stage overall preference of the patients for free trials increase after reestablishment of their treatments (i.e. ON treatment > OFF treatment). This set of results postulate a key role of the DA system in intrinsic reward assessment and intrinsic motivation.
Finally we have developed an experimental set-up that will ultimately allow us, in non-human primates, to investigate specifically the calcium-activity of the midbrain DA neurons along with simultaneous electrophysiological recordings of neurons associated to the DA system. The goal is to gain insights about the correlation between the reward encoding by DA neurons and the associated modulation of downstream neurons involved in motivational and learning mechanisms.
Gaining insights about intrinsic reward mechanisms through this project could have broad implications for better understanding of reward systems, of cognitive control, and of many cognitive dysfunctions such as motivational deficits linked with impaired dopaminergic system.
Investigating mechanisms of intrinsic reward and intrinsic motivation in primates is a crucial, but under-explored, question in the field of dopamine and reward systems and this multidisciplinary approach allows us to address it. Importantly these systems gather a very large scientific community (e.g. decision-making or learning mechanisms) and are also severely impaired in many diseases. It will provide new insights in primate neurophysiology but also for neurological patients, for whom we show impaired valuation of intrinsic rewards. This could allow for example deep brain stimulation neurosurgeons to improve electrode implantation for patients with several types of deficits such as motor but also motivational dysfunctions.
Investigating mechanisms of intrinsic reward and intrinsic motivation in primates is a crucial, but under-explored, question in the field of dopamine and reward systems and this multidisciplinary approach allows us to address it. Importantly these systems gather a very large scientific community (e.g. decision-making or learning mechanisms) and are also severely impaired in many diseases. It will provide new insights in primate neurophysiology but also for neurological patients, for whom we show impaired valuation of intrinsic rewards. This could allow for example deep brain stimulation neurosurgeons to improve electrode implantation for patients with several types of deficits such as motor but also motivational dysfunctions.