511-reward

From conceptual category to brain circuitry

• Explicit vs. implicit

(Kringelbach & Berridge, 2009)

Figure 1. Measuring reward and hedonia. Reward and pleasure are multifaceted psychological concepts. Major processes within reward (first column) consist of motivation or wanting (white), learning (blue), and – most relevant to happiness – pleasure, liking or affect (light blue). Each of these contains explicit (top rows, light yellow) and implicit (bottom rows, yellow) psychological components (second column) that constantly interact and require careful scientific experimentation to tease apart. Explicit processes are consciously experienced (e.g. explicit pleasure and happiness, desire, or expectation), whereas implicit psychological processes are potentially unconscious in the sense that they can operate at a level not always directly accessible to conscious experience (implicit incentive salience, habits and ‘liking’ reactions), and must be further translated by other mechanisms into subjective feelings. Measurements or behavioral procedures that are especially sensitive markers of each of the processes are listed (third column). Examples of some of the brain regions and neurotransmitters are listed (fourth column), as well as specific examples of measurements (fifth column), such as an example of how highest subjective life satisfaction does not lead to the highest salaries (top) [93]. Another example shows the incentive-sensitization model of addiction and how ‘wanting’ to take drugs may grow over time independently of ‘liking’ and ‘learning’ drug pleasure as an individual becomes an addict (bottom) [94].

(Kringelbach & Berridge, 2009)

Figure 2. Hedonic brain circuitry. The schematic figure shows the brain regions for causing and coding fundamental pleasure in rodents and humans. (a) Facial ‘liking’ and ‘disliking’ expressions elicited by sweet and bitter taste are similar in rodents and human infants. (b, d) Pleasure causation has been identified in rodents as arising from interlinked subcortical hedonic hotspots, such as in nucleus accumbens and ventral pallidum, where neural activation may increase ‘liking’ expressions to sweetness. Similar pleasure coding and incentive salience networks have also been identified in humans. (c) The so-called ‘pleasure’ electrodes in rodents and humans are unlikely to have elicited true pleasure but perhaps only incentive salience or ‘wanting.’ (d) The cortical localization of pleasure coding might reach an apex in various regions of the orbitofrontal cortex, which differentiate subjective pleasantness from valence processing for aspects of the same stimulus, such as a pleasant food.

• Analogous circuits mediating facial expressions of “liking” and “disliking” across mammals

Studying ‘reward’

• A reward reinforces (makes more prevalent/probable) some behavior

Intracranial self-stimulation (ICSS)

• Chemical or electrical
• Milner and Olds (Milner, 1989) discovered ‘rewarding’ power of electrical self-stimulation
• (Heath, 1963) studied effects in human patients.
• 50+ sites elicit
  • Increase in self-stimulation or
  • Conditioned place preferences

• Electrical stimulation specific? Can activate cell bodies and/or fibers of passage.
• Optogenetic techniques more spatially precise

https://youtu.be/de_b7k9kQp0

(Nestler & Carlezon, 2006)

• Lateral Hypothalamus (Hyp)
• Medial forebrain bundle (MFB)
• Ventral tegmental area (VTA) in midbrain
• Nucleus accumbens (nAcc)
• Dorsal Raphe Nucleus/Locus Coeruleus (DR/LC)
• Amygdala (Amy)
• Hippocampus (HP)
• Prefrontal cortex (PFC)

(Kohls, Chevallier, Troiani, & Schultz, 2012)

Pharmacological signals

Norepinephrine (NE)

• NE potentiates ICSS; NE deploetion or NE-R antagonists block
• NE also increases sleepiness

Dopamine (DA)

• Blocking DA receptors or depleting DA reduces ICSS
• DA release accentuates responses to cocaine, amphetamine

DA projection pathways

https://en.wikipedia.org/wiki/Dopaminergic_pathways

• Lesions to Nucleus Accumbens alter cocaine self-administration

What does DA signal?

• Hedonia and anhedonia?
  • But DA depletion does not alter orofacial responses to sweet vs. bitter
  • “Reward” not unitary?
  • Liking ≠ wanting (Berridge & Kringelbach, 2015)
• Incentive salience?
  • Make salient potential “pleasure” from future experience of X…
• Reward prediction error (RPE)
  • DA neuron fire only to unexpected/unpredicted reward outcomes
  • \(D \leftarrow R_{occurred} - R_{predicted}\)

(Hu, 2016)

Figure 1. Characteristic response to reward and punishment by different neurons. (a) Ventral tegmental area (VTA) dopamine (DA) neurons encode reward prediction error (RPE) signals, showing excitatory responses only when the reward is not fully predicted (Cohen et al. 2012, Schultz 1998). (b) VTA GABA neurons encode reward expectation, contributing to RPE calculation by serving as a source of subtraction (Eshel et al. 2015). (c) Lateral habenula (LHb) neurons show mirror-inverted phasic responses to DA neurons, potentially providing a source of negative RPE signals (Matsumoto & Hikosaka 2009a). (d,e) Recordings from dorsal raphe nucleus (DRN) serotonergic (5-HT) neurons reveal diverse responses to reward and punishment, with a substantial subset showing excitatory responses to reward even when reward is predicted (Cohen et al. 2015, Li et al. 2016, Liu et al. 2014). (f) DRN GABA neurons are inhibited by reward seeking and activated by aversive stimuli. Green, red, and blue dashed lines indicate the timing of reward cue, reward delivery, and punishment delivery, respectively. Purple dashed lines indicate entry into a reward zone in a sucrose-foraging task. The responses of DRN 5-HT and GABA neurons to unexpected reward and punishment are derived from calcium-imaging fiber photometry experiments. All others are from single-unit electrophysiological recordings.

(Hu, 2016)

DA and GABA signaling

• GABA releasing neurons also play a role
• DA + GABA or DA + Glu co-release
• GABA neurons active during delays…

(Watabe-Uchida, Eshel, & Uchida, 2017)

Figure 1. Firing patterns of identified dopamine and GABA neurons in VTA. (a) VTA neurons were recorded while mice performed an odor-outcome association task in which different odors predicted different outcomes (see legend on right). Odors were presented for 1 s (gray shading), and outcomes were presented after a 1-s delay. Neuron types were identified based on their optogenetic responses. Dopamine neurons (left) showed phasic excitations to reward-predictive cues and reward. GABA neurons (right) showed sustained activation during the delay. Data from Cohen et al. (2012). (b) Reward expectation modulates dopamine neuron firing. The plot on the left shows when outcome was presented, and the right-hand plot shows when outcome was omitted. Different odors predicted reward with different probabilities. Higher reward probability increased cue responses but suppressed reward responses. Data from Tian & Uchida (2015). Also see Fiorillo et al. (2003) and Matsumoto & Hikosaka (2009a,b). (c) Reward context-dependent modulation of dopamine responses to air puff–predictive cues and air puff. The task conditions during recording differed only in the probability of reward. Dopamine neurons showed both excitation and inhibition in high-reward contexts (left) but only inhibition in low-reward contexts (right). The response in reward trials is not shown. Data from Matsumoto et al. (2016). Abbreviations: CS, conditioned stimulus; VTA, ventral tegmental area.

Expectation modulates DA signaling

(Watabe-Uchida, Eshel, & Uchida, 2017)

Figure 2. Subtractive computation in dopamine neurons. (a) In one task condition (no odor, black), different amounts of reward were presented without any predictive cue. In another condition (odor A, orange), the timing of reward was predicted by an odor. (b) Prediction. Division should change the slope of the curve, whereas subtraction should cause a downward shift. (c) Average response of 40 optogenetically identified dopamine neurons. Prediction caused a subtractive shift. Data from Eshel et al. (2015). (d) Three example neurons. Although individual neurons exhibited diversity with respect to response magnitudes, their response functions were scaled versions of one another. Data from Eshel et al. (2016).

DA network

(Watabe-Uchida, Eshel, & Uchida, 2017)

(Watabe-Uchida, Eshel, & Uchida, 2017)

Figure 4. Monosynaptic input to dopamine neurons. (a) Monosynaptic inputs to VTA and SNc dopamine neurons (blue and red, respectively). Inputs were labeled through transsynaptic retrograde tracing using rabies virus. Data from Watabe-Uchida et al. (2012). (b) Schematic summary of panel a. The thickness of each line indicates the extent of inputs from each area (percentage of total inputs). (c) Firing patterns of monosynaptic inputs in a classical conditioning paradigm. Monosynaptic inputs to dopamine neurons were labeled by channelrhodopsin-2 using rabies virus. Optogenetics were used to identify these inputs in seven brain areas while mice performed a task. Data from Tian et al. (2016). Abbreviations: Acb, nucleus accumbens; BNST, bed nucleus of stria terminalis; Ce, central amygdala; DA, dopamine; DB, diagonal band of Broca; DR, dorsal raphe; DS, dorsal striatum; EA, extended amygdala; EP, entopeduncular nucleus (internal segment of the globus pallidus); GP, globus pallidus (external segment of the globus pallidus); IPAC, interstitial nucleus of the posterior limb of the anterior commissure; LDTg, laterodorsal tegmental nucleus; LH, lateral hypothalamus; LO, lateral orbitofrontal cortex; LPO, lateral preoptic area; M1, primary motor cortex; M2, secondary motor cortex; MPA, medial preoptic area; mRt, reticular formation; Pa, paraventricular hypothalamic nucleus; PAG, periaqueductal gray; PB, parabrachial nucleus; PPTg, pedunculopontine tegmental nucleus; PSTh, parasubthalamic nucleus; RMTg, rostromedial tegmental nucleus; S1, primary somatosensory cortex; SC, superior colliculus; SNc, substantia nigra pars compacta; STh, subthalamic nucleus; Tu, olfactory tubercle; VP, ventral pallidum; VTA, ventral tegmental area; ZI, zona incerta.

(Watabe-Uchida, Eshel, & Uchida, 2017)

Reward & Aversion Networks

(Watabe-Uchida, Eshel, & Uchida, 2017)

Other psychopharmacological components

• Serotonin (5-HT)
  • But linked to behavioral inhibition, reward suppression
  • Influence on reward via network involving other NTs
• ACh

(Cock, Vidailhet, & Arnulf, 2008)

• Opioids, endogenous morphine-like NTs (endorphins)

(Clapp, Bhave, & Hoffman, n.d.)

• Cannabinoids = psychoactive compounds found in cannibis
• Endocannabinoids (endogenous cannabinoid system)
  • Cannabinoid CB1 receptors in CNS; CB2 in body, immune system

(Flores, Maldonado, & Berrendero, 2013)

Linking action & perception

• Action -> Perception -> \(\uparrow p(A)\) + Feelings
• But, where/how are actions and perceptions/outcomes stored and compared?
• Requires memory system to store
  • Previous actions
• Compare with predictions of future sensory states, including rewards
  • \(S(t+1) \vert A(t)\)
• Is successful sensory prediction intrinsically ‘rewarding?’
• Reinforcement learning vs. Supervised learning vs. Unsupervised learning

References

Berridge, K. C., & Kringelbach, M. L. (2015). Pleasure systems in the brain. Neuron, 86(3), 646–664. https://doi.org/10.1016/j.neuron.2015.02.018

Clapp, P., Bhave, S. V., & Hoffman, P. L. (n.d.). How Adaptation of the Brain to Alcohol Leads to Dependence. Retrieved from http://pubs.niaaa.nih.gov/publications/arh314/310-339.htm

Cock, V. C. D., Vidailhet, M., & Arnulf, I. (2008). Sleep disturbances in patients with parkinsonism. Nature Clinical Practice Neurology, 4(5), 254–266. https://doi.org/10.1038/ncpneuro0775

Flores, Á., Maldonado, R., & Berrendero, F. (2013). Cannabinoid-hypocretin cross-talk in the central nervous system: What we know so far. Neuropharmacology, 7, 256. https://doi.org/10.3389/fnins.2013.00256

Heath, R. G. (1963). Electrical self-stimulation of the brain in man. American Journal of Psychiatry, 120(6), 571–577. https://doi.org/10.1176/ajp.120.6.571

Hu, H. (2016). Reward and aversion. Annual Review of Neuroscience, 39, 297–324. https://doi.org/10.1146/annurev-neuro-070815-014106

Kohls, G., Chevallier, C., Troiani, V., & Schultz, R. T. (2012). Social ‘wanting’dysfunction in autism: Neurobiological underpinnings and treatment implications. Journal of Neurodevelopmental Disorders, 4(10), 1–20. https://doi.org/10.1186/1866-1955-4-10

Kringelbach, M. L., & Berridge, K. C. (2009). Towards a functional neuroanatomy of pleasure and happiness. Trends in Cognitive Sciences, 13(11), 479–487.

Milner, P. M. (1989). The discovery of self-stimulation and other stories. Neuroscience & Biobehavioral Reviews, 13(2–3), 61–67. https://doi.org/10.1016/S0149-7634(89)80013-2

Nestler, E. J., & Carlezon, W. A. (2006). The mesolimbic dopamine reward circuit in depression. Biological Psychiatry, 59(12), 1151–1159. https://doi.org/10.1016/j.biopsych.2005.09.018

Watabe-Uchida, M., Eshel, N., & Uchida, N. (2017). Neural circuitry of reward prediction error. Annual Review of Neuroscience, 40, 373–394. https://doi.org/10.1146/annurev-neuro-072116-031109