Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-78c5997874-g7gxr Total loading time: 0 Render date: 2024-11-15T04:46:31.422Z Has data issue: false hasContentIssue false

14 - Clinical Computational Neuroscience

from Part III - Experimental and Biological Approaches

Published online by Cambridge University Press:  23 March 2020

By
Michael N. Hallquist  and
Alexandre Y. Dombrovski
Edited by
Aidan G. C. Wright  and
Michael N. Hallquist
Aidan G. C. Wright
Affiliation:
University of Pittsburgh
Michael N. Hallquist
Affiliation:
Pennsylvania State University
Get access

Summary

This chapter discusses mathematical models of learning in neural circuits with a focus on reinforcement learning. Formal models of learning provide insights into how we adapt to a complex, changing environment, and how this adaptation may break down in psychopathology. Computational clinical neuroscience is motivated to use mathematical models of decision processes to bridge between brain and behavior, with a particular focus on understanding individual differences in decision making. The chapter reviews the basics of model specification, model inversion (parameter estimation), and model-based approaches to understanding individual differences in health and disease. It illustrates how models can be specified based on theory and empirical observations, how they can be fitted to human behavior, and how model-predicted signals from neural recordings can be decoded. A functional MRI (fMRI) study of social cooperation is used to illustrate the application of reinforcement learning (RL) to test hypotheses about neural underpinnings of human social behavior.

Keywords

reinforcement learningcomputational psychiatrymodel-based cognitive neurosciencecognitive modeling
Type
Chapter
Information
The Cambridge Handbook of Research Methods in Clinical Psychology , pp. 168 - 190
Publisher: Cambridge University Press
Print publication year: 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Further Reading

For an in-depth treatment of reinforcement learning, we recommend Sutton and Barto’s recently updated classic book, Reinforcement Learning: An Introduction (2018). The Oxford Handbook of Computational and Mathematical Psychology introduces the reader to cognitive modeling and contains Gureckis and Love’s superb chapter on reinforcement learning (2015). Excellent computational neuroscience texts include Miller’s Introductory Course in Computational Neuroscience (2018) and Dayan and Abbott’s Theoretical Neuroscience (2005). Miller covers useful preliminary material, including mathematics, circuit physics and even computing and MATLAB (much of existing code for reinforcement learning modeling is written in MATLAB, but R and Python are becoming increasingly popular). Dayan and Abbot treat conditioning and reinforcement learning in greater detail. A more detailed treatment of model-based cognitive neuroscience can be found in An Introduction to Model-Based Cognitive Neuroscience (Forstmann & Wagenmakers, 2015).

References

Bartra, O., McGuire, J. T., & Kable, J. W. (2013). The Valuation System: A Coordinate-Based Meta-Analysis of BOLD fMRI Experiments Examining Neural Correlates of Subjective Value. Neuroimage, 76, 412–427.Google Scholar
Bouret, S., & Richmond, B. J. (2010). Ventromedial and Orbital Prefrontal Neurons Differentially Encode Internally and Externally Driven Motivational Values in Monkeys. Journal of Neuroscience, 30(25), 8591–8601.Google Scholar
Bower, G. H. (1994). A Turning Point in Mathematical Learning Theory. Psychological Review, 101, 290–300.Google Scholar
Box, G. E. P. (1979). Robustness in the Strategy of Scientific Model Building. In Launer, R. L. & Wilkinson, G. N. (Eds.), Robustness in Statistics (pp. 201–236). New York: Academic Press.Google Scholar
Burnham, K. P., & Anderson, D. R. (2002). Model Selection and Multi-Model Inference: A Practical Information-Theoretic Approach (2nd edn.). New York: Springer.Google Scholar
Bush, R. R., & Mosteller, F. (1951). A Mathematical Model for Simple Learning. Psychological Review., 58, 313–323.Google Scholar
Cai, X., & Padoa-Schioppa, C. (2012). Neuronal Encoding of Subjective Value in Dorsal and Ventral Anterior Cingulate Cortex. Journal of Neuroscience, 32, 3791–3808.CrossRefGoogle ScholarPubMed
Cavanagh, J. F. (2015). Cortical Delta Activity Reflects Reward Prediction Error and Related Behavioral Adjustments, but at Different Times. NeuroImage, 110, 205–216.Google Scholar
Cavanagh, J. F., Eisenberg, I., Guitart-Masip, M., Huys, Q., & Frank, M. J. (2013). Frontal Theta Overrides Pavlovian Learning Biases. Journal of Neuroscience, 33, 8541–8548.Google Scholar
Chase, H. W., Kumar, P., Eickhoff, S. B., & Dombrovski, A. Y. (2015). Reinforcement Learning Models and Their Neural Correlates: An Activation Likelihood Estimation Meta-Analysis. Cognitive, Affective, & Behavioral Neuroscience, 15(2), 435‒459.Google Scholar
Craver, C. F. (2001). Role Functions, Mechanisms, and Hierarchy. Philosophy of Science, 68, 53–74.Google Scholar
Critchley, H. D., & Rolls, E. T. (1996). Hunger and Satiety Modify the Responses of Olfactory and Visual Neurons in the Primate Orbitofrontal Cortex. Journal of Neurophysiology, 75(4), 1673–1686.Google Scholar
D’Ardenne, K., McClure, S. M., Nystrom, L. E., & Cohen, J. D. (2008). BOLD Responses Reflecting Dopaminergic Signals in the human Ventral Tegmental Area. Science, 319, 1264–1267.Google Scholar
Dayan, P., & Abbott, L. F. (2005). Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems (Revised edn.). Cambridge, MA: MIT Press.Google Scholar
Delgado, M. R., Frank, R. H., & Phelps, E. A. (2005). Perceptions of Moral Character Modulate the Neural Systems of Reward during the Trust Game. Nature Neuroscience, 8, 1611–1618.Google Scholar
Dombrovski, A. Y., Szanto, K., Clark, L., Reynolds, C. F., & Siegle, G. J. (2013). Reward Signals, Attempted Suicide, and Impulsivity in Late-Life Depression. JAMA Psychiatry, 70, 1020–1030.CrossRefGoogle ScholarPubMed
Dombrovski, A. Y., Hallquist, M. N., Brown, V. M., Wilson, J., & Szanto, K. (2019). Value-Based Choice, Contingency Learning, and Suicidal Behavior in Mid- and Late-Life Depression. Biological Psychiatry, 85(6), 506–516.Google Scholar
Dreher, J.-C., & Tremblay, L. (Eds.) (2016). Decision Neuroscience: An Integrative Perspective (1st edn.). Amsterdam: Academic Press.Google Scholar
Elston, G. N. (2003). Cortex, Cognition and the Cell: New Insights into the Pyramidal Neuron and Prefrontal Function. Cerebral Cortex, 13(11), 1124–1138.CrossRefGoogle ScholarPubMed
Estes, W.K., 1950. Toward a Statistical Theory of Learning. Psychological Review, 57, 94‒107.CrossRefGoogle Scholar
Fiorillo, C. D., Newsome, W. T., & Schultz, W. (2008). The Temporal Precision of Reward Prediction in Dopamine Neurons. Nature Neuroscience, 11, 966–973.CrossRefGoogle ScholarPubMed
Forstmann, B. U., & Wagenmakers, E.-J. (Eds.) (2015). An Introduction to Model-Based Cognitive Neuroscience. New York: Springer.Google Scholar
Fouragnan, E., Retzler, C., & Philiastides, M. G., 2018. Separate Neural Representations of Prediction Error Valence and Surprise: Evidence from an fMRI Meta-Analysis. Human Brain Mapping, 39, 2887–2906.Google Scholar
Gershman, S. J. (2016). Empirical Priors for Reinforcement Learning Models. Journal of Mathematical Psychology, 71, 1–6.CrossRefGoogle Scholar
Glimcher, P. W. (2011). Understanding Dopamine and Reinforcement Learning: The Dopamine Reward Prediction Error Hypothesis. Proceedings of the National Academy of Sciences of the United States of America, 108(Suppl 3), 15647–15654.CrossRefGoogle ScholarPubMed
Gureckis, T., & Love, B. (2015). Computational Reinforcement Learning. In The Oxford Handbook of Computational and Mathematical Psychology (pp. 99‒117). Oxford: Oxford University PressGoogle Scholar
Hallquist, M. N., & Dombrovski, A. Y., 2019. Selective Maintenance of Value Information Helps Resolve the Exploration/Exploitation Dilemma. Cognition, 183, 226–243.CrossRefGoogle ScholarPubMed
Hertwig, R., & Erev, I., 2009. The Description-Experience Gap in Risky Choice. Trends in Cognitive Science, 13, 517–523.Google Scholar
Holroyd, C. B., & Krigolson, O. E., 2007. Reward Prediction Error Signals Associated with a Modified Time Estimation Task. Psychophysiology, 44, 913–917.CrossRefGoogle ScholarPubMed
Hursh, S. R., & Silberberg, A. (2008). Economic Demand and Essential Value. Psychological Review, 115, 186–198.Google Scholar
Jocham, G., Neumann, J., Klein, T. A., Danielmeier, C., & Ullsperger, M. (2009). Adaptive Coding of Action Values in the Human Rostral Cingulate Zone. Journal of Neuroscience, 29, 7489–7496.Google Scholar
Kaelbling, L. P., Littman, M. L., & Moore, A. W., 1996. Reinforcement Learning: A Survey. Journal of Artificial Intelligence Research, 4, 237–285.Google Scholar
Kamin, L. J. (1968). Predictability, Surprise, Attention, and Conditioning. In Campbell, B. A. & Church, R. M. (Eds.), Punishment and Aversive Behavior (pp. 279–296). New York: Appleton-Century-Crofts.Google Scholar
Katahira, K. (2016). How Hierarchical Models Improve Point Estimates of Model Parameters at the Individual Level. Journal of Mathematical Psychology, 73, 37–58.Google Scholar
Kennerley, S. W., & Wallis, J. D. (2009). Evaluating Choices by Single Neurons in the Frontal Lobe: Outcome Value Encoded across Multiple Decision Variables. European Journal of Neuroscience, 29, 2061–2073.Google Scholar
Kennerley, S. W., Dahmubed, A. F., Lara, A. H., & Wallis, J. D. (2009). Neurons in the Frontal Lobe Encode the Value of Multiple Decision Variables. Journal of Cognitive Neuroscience, 21, 1162–1178.Google Scholar
Kobayashi, S., Pinto de Carvalho, O., & Schultz, W. (2010). Adaptation of Reward Sensitivity in Orbitofrontal Neurons. Journal of Neuroscience, 30, 534–544.Google Scholar
Kriegeskorte, N., & Douglas, P. K. (2018). Cognitive Computational Neuroscience. Nature Neuroscience, 21, 1148–1160.CrossRefGoogle ScholarPubMed
Kruschke, J. (2014). Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan. Cambridge, MA: Academic Press.Google Scholar
Laird, A. R., Fox, P. M., Eickhoff, S. B., Turner, J. A., Ray, K. L., McKay, D. R., … Fox, P. T. (2011). Behavioral Interpretations of Intrinsic Connectivity Networks. Journal of Cognitive Neuroscience, 23, 4022–4037.Google Scholar
Li, J., Schiller, D., Schoenbaum, G., Phelps, E. A., & Daw, N. D. (2011). Differential Roles of Human Striatum and Amygdala in Associative Learning. Nature Neuroscience, 14, 1250‒1252.Google Scholar
Logothetis, N.K., Pfeuffer, J., 2004. On the Nature of the BOLD fMRI Contrast Mechanism. Magnetic Resonance Imaging, 22, 1517–1531.Google Scholar
Love, B. C. (2015). The Algorithmic Level Is the Bridge between Computation and Brain. Topics in Cognitive Science, 7, 230–242.CrossRefGoogle Scholar
Marr, D. (1982). Chapter 1: The Philosophy and the Approach. In Vision: A Computational Investigation into the Human Representation and Processing Visual Information (pp. 8–38). Cambridge, MA: MIT Press.Google Scholar
McClelland, J. L., & Rumelhart, D. E. (1986). Parallel Distributed Processing, Explorations in the Microstructure of Cognition: Foundations. Cambridge, MA: MIT Press.Google Scholar
McCulloch, W. S., & Pitts, W. (1943). A Logical Calculus of the Ideas Immanent in Nervous Activity. Bulletin of Mathematical Biophysics, 5, 115–133.Google Scholar
Miller, P. (2018). An Introductory Course in Computational Neuroscience (1st edn). Cambridge, MA: MIT Press.Google Scholar
Miller, R. R., Barnet, R. C., & Grahame, N. J. (1995). Assessment of the Rescorla-Wagner Model. Psychological Bulletin, 117, 363–386.CrossRefGoogle ScholarPubMed
Minsky, M. L. (1967). Computation: Finite and Infinite Machines. Upper Saddle River, NJ: Prentice-Hall.Google Scholar
Minsky, M. L., & Papert, S. A. (1969). Perceptrons: An Introduction to Computational Geometry. Cambridge, MA: MIT Press.Google Scholar
Mohebi, A., Pettibone, J., Hamid, A., Wong, J.-M., Kennedy, R., & Berke, J. (2018). Forebrain Dopamine Value Signals Arise Independently from Midbrain Dopamine Cell Firing. bioRxiv, 334060.Google Scholar
Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A Framework for Mesencephalic Dopamine Systems Based on Predictive Hebbian Learning. Journal of Neuroscience, 16, 1936–1947.Google Scholar
Montague, P. R., Dolan, R. J., Friston, K. J., & Dayan, P. (2012). Computational Psychiatry. Trends in Cognitive Science, 16, 72–80.Google Scholar
Moran, R. J., Symmonds, M., Stephan, K. E., Friston, K. J., & Dolan, R. J. (2011). An In Vivo Assay of Synaptic Function Mediating Human Cognition. Current Biology, 21, 1320–1325.Google Scholar
Nocedal, J., & Wright, S. (2006). Numerical Optimization (2nd edn.). New York: Springer.Google Scholar
Noonan, M. P., Chau, B. K. H., Rushworth, M. F. S., & Fellows, L. K. (2017). Contrasting Effects of Medial and Lateral Orbitofrontal Cortex Lesions on Credit Assignment and Decision-Making in Humans. Journal of Neuroscience, 37, 7023–7035.Google Scholar
O’Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003). Temporal Difference Models and Reward-Related Learning in the Human Brain. Neuron, 38, 329–337.Google Scholar
Padoa-Schioppa, C., & Assad, J. A. (2006). Neurons in the Orbitofrontal Cortex Encode Economic Value. Nature, 441, 223–226.Google Scholar
Parker, N. F., Cameron, C. M., Taliaferro, J. P., Lee, J., Choi, J. Y., Davidson, T. J., … Witten, I. B. (2016). Reward and Choice Encoding in Terminals of Midbrain Dopamine Neurons Depends on Striatal Target. Nature Neuroscience, 19, 845–854.Google Scholar
Paton, J. J., Belova, M. A., Morrison, S. E., & Salzman, C. D. (2006). The Primate Amygdala Represents the Positive and Negative Value of Visual Stimuli during Learning. Nature, 439, 865–870.Google Scholar
Pawlow, I. P. (1904). Nobel-Vortrag*. Nordiskt Medicinskt Arkiv, 37, 1–20.Google Scholar
Rescorla, R. A., & Wagner, A. R. (1972). A Theory of Pavlovian Conditioning: Variations in the Effectiveness of Reinforcement and Nonreinforcement. In: Black, A. H. & Prokasy, W. F. (Eds.), Classical Conditioning II (pp. 64–99). New York: Appleton-Century-Crofts.Google Scholar
Rigoux, L., & Daunizeau, J. (2015). Dynamic Causal Modelling of Brain-Behaviour Relationships. NeuroImage, 117, 202–221.Google Scholar
Rigoux, L., Stephan, K. E., Friston, K. J., & Daunizeau, J. (2014). Bayesian Model Selection for Group Studies ‒ Revisited. NeuroImage, 84, 971–985.CrossRefGoogle Scholar
Roesch, M. R., & Olson, C. R. (2005). Neuronal Activity in Primate Orbitofrontal Cortex Reflects the Value of Time. Journal of Neurophysiology, 94, 2457–2471.Google Scholar
Rosenblatt, F. (1958). The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review, 65, 386–408.Google Scholar
Ruff, C. C., & Fehr, E. (2014). The Neurobiology of Rewards and Values in Social Decision Making. Nature Reviews Neuroscience, 15, 549–562.Google Scholar
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005). Representation of Action-Specific Reward Values in the Striatum. Science, 310, 1337–1340.Google Scholar
Schoenbaum, G., Chiba, A. A., & Gallagher, M. (1999). Neural Encoding in Orbitofrontal Cortex and Basolateral Amygdala during Olfactory Discrimination Learning. Journal of Neuroscience, 19, 1876–1884.Google Scholar
Schuck, N. W., Cai, M. B., Wilson, R. C., & Niv, Y. (2016). Human Orbitofrontal Cortex Represents a Cognitive Map of State Space. Neuron, 91, 1402–1412.Google Scholar
Schultz, W., Dayan, P., & Montague, P. R. (1997). A Neural Substrate of Prediction and Reward. Science, 275, 1593–1599.Google Scholar
Sejnowski, T. J., Churchland, P. S., & Movshon, J. A. (2014). Putting Big Data to Good Use in Neuroscience. Nature Neuroscience, 17, 1440–1441.CrossRefGoogle ScholarPubMed
Simon, H. A. (1956). Rational Choice and the Structure of the Environment. Psychological Review., 63, 129–138.Google Scholar
Stalnaker, T. A., Cooch, N. K., & Schoenbaum, G. (2015). What the Orbitofrontal Cortex Does Not Do. Nature Neuroscience, 18, 620–627.Google Scholar
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009). Bayesian Model Selection for Group Studies. NeuroImage, 46, 1004–1017.Google Scholar
Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd edn.). Adaptive Computation and Machine Learning Series. Cambridge, MA: MIT Press.Google Scholar
Tobler, P. N., Dickinson, A., & Schultz, W. (2003). Coding of Predicted Reward Omission by Dopamine Neurons in a Conditioned Inhibition Paradigm. Journal of Neuroscience, 23, 10402–10410.Google Scholar
Tremblay, L., & Schultz, W. (1999). Relative Reward Preference in Primate Orbitofrontal Cortex. Nature, 398, 704–708.CrossRefGoogle ScholarPubMed
Turner, B. M., Forstmann, B. U., Love, B. C., Palmeri, T. J., & Van Maanen, L. (2017). Approaches to Analysis in Model-Based Cognitive Neuroscience. Journal of Mathematical Psychology, Model-Based Cognitive Neuroscience, 76, 65–79.Google Scholar
Uttal, W. R. (2001). The New Phrenology: The Limits of Localizing Cognitive Processes in the Brain, The New Phrenology: The Limits of Localizing Cognitive Processes in the Brain. Cambridge, MA: MIT Press.Google Scholar
Uttal, W. R. (2014). Psychomythics: Sources of Artifacts and Misconceptions in Scientific Psychology (1st edn.). Hove: Psychology Press.Google Scholar
Vanyukov, P. M., Hallquist, M. N., Delgado, M. R., Szanto, K., & Dombrovski, A. Y. (2019). Neurocomputational Mechanisms of Adaptive Learning in Social Exchanges. Cognitive, Affective, & Behavioral Neuroscience, 19, 1‒13.CrossRefGoogle ScholarPubMed
Vermunt, J. K. (2010). Latent Class Modeling with Covariates: Two Improved Three-Step Approaches. Political Analysis, 18, 450–469.Google Scholar
Waelti, P., Dickinson, A., & Schultz, W. (2001). Dopamine Responses Comply with Basic Assumptions of Formal Learning Theory. Nature, 412, 43–48.Google Scholar
Wagenmakers, E.-J., Morey, R. D., & Lee, M. D. (2016). Bayesian Benefits for the Pragmatic Researcher. Current Directions in Psychological Science, 25, 169–176.Google Scholar
Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing Model Mimicry Using the Parametric Bootstrap. Journal of Mathematical Psychology, 48, 28–50.Google Scholar
Wallis, J. D., & Kennerley, S. W. (2010). Heterogeneous Reward Signals in Prefrontal Cortex. Current Opinions in Neurobiology, 20, 191–198.Google Scholar
Walton, M. E., Behrens, T. E., Buckley, M. J., Rudebeck, P. H., Rushworth, M. F. (2010). Separable Learning Systems in the Macaque Brain and the Role of Orbitofrontal Cortex in Contingent Learning. Neuron, 65, 927–939.Google Scholar
Werbos, P. (1974). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Cambridge, MA: Harvard University Press.Google Scholar
Whittington, J. C. R., & Bogacz, R. (2019). Theories of Error Back-Propagation in the Brain. Trends in Cognitive Science, 23, 235‒250.Google Scholar
Wiecki, T. V., Poland, J., & Frank, M. J. (2015). Model-Based Cognitive Neuroscience Approaches to Computational Psychiatry Clustering and Classification. Clinical Psycholical Science, 3, 378–399.Google Scholar
Wimmer, G. E., Braun, E. K., Daw, N. D., & Shohamy, D. (2014). Episodic Memory Encoding Interferes with Reward Learning and Decreases Striatal Prediction Errors. Journal of Neuroscience, 34, 14901–14912.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×