Final Report Summary - DOPAPREDICT (Functional Analysis of Dopamine Prediction Error Circuits)
A) BACKGROUND
One of the most common commonly observed forms of learning in humans and animals, relies on the reinforcing effect of rewarding or aversive experience. As a result of such learning, sensory cues, behavioral responses or specific actions become associated with positive or negative values, which is critical for seeking resources and avoiding danger. Previous studies have shown that the midbrain dopamine system plays a key role in reinforcement learning. When sensory cues (conditioned stimuli, CS) are associated with reward (unconditioned stimuli, US), they respond to unexpected reward with a phasic increase in their firing rate. When predicted reward is omitted, the firing rate of dopamine neurons is reduced. Thus, dopamine neurons encode the discrepancy between predicted and actual reward, i.e. reward prediction errors (RPE). RPEs play a central role in formal theories of reinforcement learning and experimental manipulations of dopamine with optogenetics, specifically during the US period have confirmed that dopamine RPE responses causally impact associative learning.
According to alternative theoretical frameworks, associative learning is also directly influenced by the CS through variations of its salience or associability. Consistent with an involvement in CS associability, novel, physically intense or otherwise salient sensory stimuli have long been recognized to evoke phasic dopamine responses which decrease with increasing experience. However, the contribution of dopamine CS salience responses to associative learning has not been addressed experimentally.
The concept of CS associability has also provided the cornerstone for models of latent inhibition, i.e. the common behavioral observation that conditioned responses to familiar cues establish much slower during associative learning than those to novel cues. Latent inhibition has been widely observed across different learning paradigms and mammalian species, suggesting it reflects an adaptive learning strategy. According to CS associability frameworks, experiencing stimuli which are not followed by an event or consequence reduces their associability, resulting in latent inhibition.
Pharmacological evidence has implicated the dopamine system in latent inhibition. Dopamine agonists, such as amphetamine attenuate latent inhibition in rodents, whereas dopamine antagonists such as the antipsychotic drugs haloperidol and chlorpromazine increase latent inhibition. Consistent with the effects of antipsychotic drugs on latent inhibition, schizophrenia has been associated with reduced latent inhibition. Despite this evidence for a role of dopamine in latent inhibition, the circuit mechanism by which the activity of dopamine neurons causes a learning bias towards novel stimuli remains unknown today.
B) SCIENTIFIC RESULTS
A behavioral paradigm for studying neural responses to stimulus novelty: To study responses of dopamine neurons to stimulus novelty, we have developed a behavioral paradigm, in which animals show spontaneous exploratory sniffing responses to novel but not familiar stimuli. To measure sniffing dynamics in head-restrained mice, we developed a novel non-invasive methodology (see below). Using this new approach, we discovered a spontaneous nasal orienting response towards novel stimuli, apparent already in the first sniff after odor onset. We show that interhemispheric communication is essential for orienting, but not novelty detection. This finding directly demonstrates that mice use internostril difference comparison for odor sources localization. Finally, we demonstrate the AON, a poorly understood olfactory region, is both necessary and sufficient for evoking orienting responses.
The role of dopamine cue salience responses in associative learning: Here, we characterized the response of dopamine neurons to stimulus novelty, the subjective stimulus quality of having never perceived the stimulus before, which is also implicated in latent inhibition. We measured dopamine transients with fiber photometry across the midbrain ventral tegmental area (VTA) and substantia nigra pars compacta (SNc) in our non-associative novelty exposure paradigm. We then performed olfactory conditioning experiments in head-restrained mice to study the role of dopamine CS salience responses in associative learning. We used stimulus novelty and familiarity to experimentally increase and decrease CS salience. Using bidirectional optogenetic manipulation of dopamine neurons selectively during the CS period, we confirmed our hypothesis that dopamine CS responses promote associative learning.
C) TECHNOLOGY DEVELOPMENT
Non-invasive, non-contact measurement of respiration using IR thermography: To overcome the need for invasive implantation of sensors for respiratory measurement, we developed a non-invasive, non-contact method involving infrared thermography and a novel algorithm for signal extraction.
Optobrain: We fabricated an integrated ultrathin neural interface with 12 optical outputs and 24 electrodes for simultaneous extracellular recording and optogenetic photostimulation. We established the new device by measuring the effect of highly localized stimulation in awake, behaving mice.
D) IMPACT
Scientific impact:
The role of these CS salience responses in associative learning has been less well studied than dopamine US responses, which signal prediction errors that determine how well an animal learns to associate a CS with a US. Our findings show for the first time that dopamine CS responses signal the associability of the CS, consistent with theoretical models of associative learning, in which the CS plays a critical role in forming the association between the CS and the US. The function of dopamine in associative learning thus involves a combination of stimulus salience signals and reward prediction error signals. This idea is consistent with dopamine response properties recorded previously in primates and with theoretical models combining stimulus associability and error terms, but it has lacked an experimental confirmation.
Our experimental finding that dopaminergic CS associability signals promote associative learning also has important implications for latent inhibition. Past research has focused on the idea that learning rates during conditioning are negatively affected by familiarity. Novelty, on the other hand, has received surprisingly little attention in the context of latent inhibition, even though it is known to enhance memory formation in a dopamine-dependent manner. This may in part be explained by the fact that the term latent inhibition implicitly implies a negative modulation of learning by familiarity, rather than a positive modulation by novelty, although this was not intended when the term was coined. Our results now provide a plausible explanation for the role of dopamine in latent inhibition, suggesting that the learning bias towards novel stimuli in latent inhibition is not caused by reduced CS associability of familiar stimuli but the increased associability of novel stimuli. Latent inhibition might thus merely result from the absence of dopamine CS salience responses to familiar stimuli.
Since disturbances of reinforcement learning and latent inhibition have been directly linked to human pathological conditions including schizophrenia and depression, as well as maladaptive behaviors in addiction, our results offer mechanistic insights into these pathological conditions which may ultimately support the development of novel interventional strategies.
Technology impact:
By removing the need for invasive surgery, IR thermography addresses the requirement for more refined animal experimentation according to the ‘3Rs’ principle (‘Refine, Reduce, Replace’),defined in the ethical framework for the use of animals. Moreover, IR thermography can be easily combined with other techniques to addresses the need for precise behavioral monitoring in head-restrained mice which is critical for linking neural dynamics to behavior.
We further established a novel tool combining spatially confined optical stimulation and extracellular recording. Our findings demonstrate the new optoelectrode is useful for characterizing microcircuits with spatially defined illumination in vivo. Finally, the fact that even very localized photostimulation has wide-ranging effects beyond the stimulation site, demonstrates the difficulty in predicting circuit-level effects of optogenetic manipulations and highlights the need for closely monitoring neural activity in optogenetic experiments. Further technological improvements will facilitate such experiments and enable dissection of neural circuits at high resolution in vivo.
One of the most common commonly observed forms of learning in humans and animals, relies on the reinforcing effect of rewarding or aversive experience. As a result of such learning, sensory cues, behavioral responses or specific actions become associated with positive or negative values, which is critical for seeking resources and avoiding danger. Previous studies have shown that the midbrain dopamine system plays a key role in reinforcement learning. When sensory cues (conditioned stimuli, CS) are associated with reward (unconditioned stimuli, US), they respond to unexpected reward with a phasic increase in their firing rate. When predicted reward is omitted, the firing rate of dopamine neurons is reduced. Thus, dopamine neurons encode the discrepancy between predicted and actual reward, i.e. reward prediction errors (RPE). RPEs play a central role in formal theories of reinforcement learning and experimental manipulations of dopamine with optogenetics, specifically during the US period have confirmed that dopamine RPE responses causally impact associative learning.
According to alternative theoretical frameworks, associative learning is also directly influenced by the CS through variations of its salience or associability. Consistent with an involvement in CS associability, novel, physically intense or otherwise salient sensory stimuli have long been recognized to evoke phasic dopamine responses which decrease with increasing experience. However, the contribution of dopamine CS salience responses to associative learning has not been addressed experimentally.
The concept of CS associability has also provided the cornerstone for models of latent inhibition, i.e. the common behavioral observation that conditioned responses to familiar cues establish much slower during associative learning than those to novel cues. Latent inhibition has been widely observed across different learning paradigms and mammalian species, suggesting it reflects an adaptive learning strategy. According to CS associability frameworks, experiencing stimuli which are not followed by an event or consequence reduces their associability, resulting in latent inhibition.
Pharmacological evidence has implicated the dopamine system in latent inhibition. Dopamine agonists, such as amphetamine attenuate latent inhibition in rodents, whereas dopamine antagonists such as the antipsychotic drugs haloperidol and chlorpromazine increase latent inhibition. Consistent with the effects of antipsychotic drugs on latent inhibition, schizophrenia has been associated with reduced latent inhibition. Despite this evidence for a role of dopamine in latent inhibition, the circuit mechanism by which the activity of dopamine neurons causes a learning bias towards novel stimuli remains unknown today.
B) SCIENTIFIC RESULTS
A behavioral paradigm for studying neural responses to stimulus novelty: To study responses of dopamine neurons to stimulus novelty, we have developed a behavioral paradigm, in which animals show spontaneous exploratory sniffing responses to novel but not familiar stimuli. To measure sniffing dynamics in head-restrained mice, we developed a novel non-invasive methodology (see below). Using this new approach, we discovered a spontaneous nasal orienting response towards novel stimuli, apparent already in the first sniff after odor onset. We show that interhemispheric communication is essential for orienting, but not novelty detection. This finding directly demonstrates that mice use internostril difference comparison for odor sources localization. Finally, we demonstrate the AON, a poorly understood olfactory region, is both necessary and sufficient for evoking orienting responses.
The role of dopamine cue salience responses in associative learning: Here, we characterized the response of dopamine neurons to stimulus novelty, the subjective stimulus quality of having never perceived the stimulus before, which is also implicated in latent inhibition. We measured dopamine transients with fiber photometry across the midbrain ventral tegmental area (VTA) and substantia nigra pars compacta (SNc) in our non-associative novelty exposure paradigm. We then performed olfactory conditioning experiments in head-restrained mice to study the role of dopamine CS salience responses in associative learning. We used stimulus novelty and familiarity to experimentally increase and decrease CS salience. Using bidirectional optogenetic manipulation of dopamine neurons selectively during the CS period, we confirmed our hypothesis that dopamine CS responses promote associative learning.
C) TECHNOLOGY DEVELOPMENT
Non-invasive, non-contact measurement of respiration using IR thermography: To overcome the need for invasive implantation of sensors for respiratory measurement, we developed a non-invasive, non-contact method involving infrared thermography and a novel algorithm for signal extraction.
Optobrain: We fabricated an integrated ultrathin neural interface with 12 optical outputs and 24 electrodes for simultaneous extracellular recording and optogenetic photostimulation. We established the new device by measuring the effect of highly localized stimulation in awake, behaving mice.
D) IMPACT
Scientific impact:
The role of these CS salience responses in associative learning has been less well studied than dopamine US responses, which signal prediction errors that determine how well an animal learns to associate a CS with a US. Our findings show for the first time that dopamine CS responses signal the associability of the CS, consistent with theoretical models of associative learning, in which the CS plays a critical role in forming the association between the CS and the US. The function of dopamine in associative learning thus involves a combination of stimulus salience signals and reward prediction error signals. This idea is consistent with dopamine response properties recorded previously in primates and with theoretical models combining stimulus associability and error terms, but it has lacked an experimental confirmation.
Our experimental finding that dopaminergic CS associability signals promote associative learning also has important implications for latent inhibition. Past research has focused on the idea that learning rates during conditioning are negatively affected by familiarity. Novelty, on the other hand, has received surprisingly little attention in the context of latent inhibition, even though it is known to enhance memory formation in a dopamine-dependent manner. This may in part be explained by the fact that the term latent inhibition implicitly implies a negative modulation of learning by familiarity, rather than a positive modulation by novelty, although this was not intended when the term was coined. Our results now provide a plausible explanation for the role of dopamine in latent inhibition, suggesting that the learning bias towards novel stimuli in latent inhibition is not caused by reduced CS associability of familiar stimuli but the increased associability of novel stimuli. Latent inhibition might thus merely result from the absence of dopamine CS salience responses to familiar stimuli.
Since disturbances of reinforcement learning and latent inhibition have been directly linked to human pathological conditions including schizophrenia and depression, as well as maladaptive behaviors in addiction, our results offer mechanistic insights into these pathological conditions which may ultimately support the development of novel interventional strategies.
Technology impact:
By removing the need for invasive surgery, IR thermography addresses the requirement for more refined animal experimentation according to the ‘3Rs’ principle (‘Refine, Reduce, Replace’),defined in the ethical framework for the use of animals. Moreover, IR thermography can be easily combined with other techniques to addresses the need for precise behavioral monitoring in head-restrained mice which is critical for linking neural dynamics to behavior.
We further established a novel tool combining spatially confined optical stimulation and extracellular recording. Our findings demonstrate the new optoelectrode is useful for characterizing microcircuits with spatially defined illumination in vivo. Finally, the fact that even very localized photostimulation has wide-ranging effects beyond the stimulation site, demonstrates the difficulty in predicting circuit-level effects of optogenetic manipulations and highlights the need for closely monitoring neural activity in optogenetic experiments. Further technological improvements will facilitate such experiments and enable dissection of neural circuits at high resolution in vivo.