Hostname: page-component-848d4c4894-m9kch Total loading time: 0 Render date: 2024-06-07T13:14:46.629Z Has data issue: false hasContentIssue false
  • English
  • Français

Dynamic diversity is the answer to proxy failure

Published online by Cambridge University Press:  13 May 2024

Zeb Kurth-Nelson Open the ORCID record for Zeb Kurth-Nelson [Opens in a new window] ,
Steve Sullivan ,
Joel Z. Leibo Open the ORCID record for Joel Z. Leibo [Opens in a new window]  and
Marc Guitart-Masip Open the ORCID record for Marc Guitart-Masip [Opens in a new window]
Zeb Kurth-Nelson*
Affiliation:
Google DeepMind, London, UK jzl@google.com Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, UK
Steve Sullivan
Affiliation:
Department of Anesthesiology and Perioperative Medicine, Oregon Health and Science University, Portland, OR, USA sulliste@ohsu.edu
Joel Z. Leibo
Affiliation:
Google DeepMind, London, UK jzl@google.com
Marc Guitart-Masip
Affiliation:
Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, UK Aging Research Center, Department of Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden marc.guitart78@gmail.com Center for Psychiatry Research, Region Stockholm, Stockholm, Sweden. Center for Cognitive Computational Neuropsychiatry (CCNP), Karolinska Institutet, Stockholm, Sweden
*
Corresponding author: Zeb Kurth-Nelson; Email: zebkurthnelson@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

We argue that a diverse and dynamic pool of agents mitigates proxy failure. Proxy modularity plays a key role in the ongoing production of diversity. We review examples from a range of scales.

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 47 , 2024 , e77
Check for updates
Copyright
Copyright © The Author(s), 2024. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

Equal contribution

References

Agren, J. A., Haig, D., & McCoy, D. E. (2022). Meiosis solved the problem of gerrymandering. Journal of Genetics, 101(2), 38.CrossRefGoogle ScholarPubMed
Balleine, B. W., Delgado, M. R., & Hikosaka, O. (2007). The role of the dorsal striatum in reward and decision-making. Journal of Neuroscience, 27(31), 8161–8165.CrossRefGoogle ScholarPubMed
Bernstein, E., Shore, J., & Lazer, D. (2018). How intermittent breaks in interaction improve collective intelligence. Proceedings of the National Academy of Sciences of the United States of America, 115(35), 8734–8739.CrossRefGoogle ScholarPubMed
Buckner, R. L. (2010). The role of the hippocampus in prediction and imagination. Annual Review of Psychology, 61, 27–48.CrossRefGoogle ScholarPubMed
Cardinal, R. N., Parkinson, J. A., Hall, J., & Everitt, B. J. (2002). Emotion and motivation: The role of the amygdala, ventral striatum, and prefrontal cortex. Neuroscience & Biobehavioral Reviews, 26(3), 321–352.CrossRefGoogle ScholarPubMed
Crepinšek, M., Liu, S.-H., & Mernik, M. (2013). Exploration and exploitation in evolutionary algorithms: A survey. ACM Computing Surveys (CSUR), 45(3), 1–33.CrossRefGoogle Scholar
Cui, G., Jun, S. B., Jin, X., Pham, M. D., Vogel, S. S., Lovinger, D. M., & Costa, R. M. (2013). Concurrent activation of striatal direct and indirect pathways during action initiation. Nature, 494(7436), 238–242.CrossRefGoogle ScholarPubMed
Czarnecki, W. M., Gidel, G., Tracey, B., Tuyls, K., Omidshafiei, S., Balduzzi, D., & Jaderberg, M. (2020). Real world games look like spinning tops. Advances in Neural Information Processing Systems, 33, 17443–17454.Google Scholar
Da Silva, J. A., Tecuapetla, F., Paixão, V., & Costa, R. M. (2018). Dopamine neuron activity before action initiation gates and invigorates future movements. Nature, 554(7691), 244–248.CrossRefGoogle ScholarPubMed
De Bono, E. (1970). Lateral thinking (Vol. 70). Harper & Row.Google Scholar
Diamond, J. (2005). Collapse: How societies choose to fail or succeed. Penguin Books. ISBN 9780241958681.Google Scholar
Duncker, K. (1945). On problem-solving. Psychological monographs, 58(5), 1–113.CrossRefGoogle Scholar
Feyerabend, P. K. (1975). Against method. Verso.Google Scholar
Frank, M. J., & Claus, E. D. (2006). Anatomy of a decision: Striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychological Review, 113(2), 300.CrossRefGoogle ScholarPubMed
Graybiel, A. M., & Grafton, S. T. (2015). The striatum: Where skills and habits meet. Cold Spring Harbor Perspectives in Biology, 7(8), a021691.CrossRefGoogle ScholarPubMed
Hills, T. T., Todd, P. M., Lazer, D., Redish, A. D., & Couzin, I. D. (2015). Exploration versus exploitation in space, mind, and society. Trends in Cognitive Sciences, 19(1), 46–54.CrossRefGoogle ScholarPubMed
Judson, O. P., & Normark, B. B. (1996). Ancient asexual scandals. Trends in Ecology & Evolution, 11(2), 41–46.CrossRefGoogle ScholarPubMed
Kazanina, N., & Poeppel, D. (2023). The neural ingredients for a language of thought are available. Trends in Cognitive Sciences, 27(11), 996–1007.CrossRefGoogle ScholarPubMed
Klaus, A., Martins, G. J., Paixao, V. B., Zhou, P., Paninski, L., & Costa, R. M. (2017). The spatiotemporal organization of the striatum encodes action space. Neuron, 95(5), 1171–1180.CrossRefGoogle ScholarPubMed
Kurth-Nelson, Z., Behrens, T., Wayne, G., Miller, K., Luettgau, L., Dolan, R., … Schwartenbeck, P. (2023). Replay and compositional computation. Neuron, 111(4), 454–469.CrossRefGoogle ScholarPubMed
Lakoff, G., & Johnson, M. (1980). Metaphors we live by. University of Chicago Press.Google Scholar
Lane, N., & Martin, W. (2010). The energetics of genome complexity. Nature, 467(7318), 929–934.CrossRefGoogle ScholarPubMed
Lehtonen, J., Jennions, M. D., & Kokko, H. (2012). The many costs of sex. Trends in Ecology & Evolution, 27(3), 172–178.CrossRefGoogle ScholarPubMed
Leibo, J. Z., Hughes, E., Lanctot, M., & Graepel, T. (2019). Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.00742.Google Scholar
Livnat, A., Papadimitriou, C., Dushoff, J., & Feldman, M. W. (2008). A mixability theory for the role of sex in evolution. Proceedings of the National Academy of Sciences of the United States of America, 105(50), 19803–19808.CrossRefGoogle ScholarPubMed
Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011). How social influence can undermine the wisdom of crowd effect. Proceedings of the National Academy of Sciences of the United States of America, 108(22), 9020–9025.CrossRefGoogle ScholarPubMed
Margulis, L. (1970). Origin of eukaryotic cells: Evidence and research implications for a theory of the origin and evolution of microbial, plant, and animal cells on the Precambrian Earth. Yale University Press.Google Scholar
Maynard Smith, J. (1978). The evolution of sex (Vol. 4). Cambridge University Press.Google Scholar
McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3), 419.CrossRefGoogle ScholarPubMed
Melo, D., Porto, A., Cheverud, J. M., & Marroig, G. (2016). Modularity: Genes, development, and evolution. Annual Review of Ecology, Evolution, and Systematics, 47, 463–486.CrossRefGoogle ScholarPubMed
Mohebi, A., Pettibone, J. R., Hamid, A. A., Wong, J.-M. T., Vinson, L. T., Patriarchi, T., … Berke, J. D. (2019). Dissociable dopamine dynamics for learning and motivation. Nature, 570(7759), 65–70.CrossRefGoogle ScholarPubMed
Ohlsson, S. (1992). Information-processing explanations of insight and related phenomena. Advances in the Psychology of Thinking, 1, 1–44.Google Scholar
O'Reilly, R. C., Hazy, T. E., Mollick, J., Mackie, P., & Herd, S. (2014). Goal-driven cognition in the brain: A computational framework. arXiv preprint arXiv:1404.7591.Google Scholar
O'Reilly, R. C., Ranganath, C., & Russin, J. L. (2022). The structure of systematicity in the brain. Current Directions in Psychological Science, 31(2), 124–130.CrossRefGoogle ScholarPubMed
Ostrom, E., Gardner, R., & Walker, J. (1994). Rules, games, and common-pool resources. University of Michigan Press.CrossRefGoogle Scholar
Page, S. E. (2017). The diversity bonus. Princeton University Press.CrossRefGoogle Scholar
Roger, A. J., Muñoz-Gómez, S. A., & Kamikawa, R. (2017). The origin and diversification of mitochondria. Current Biology, 27(21), R1177–R1192.CrossRefGoogle ScholarPubMed
Saffo, P. (2008). Strong opinions weakly held. https://saffo.com/02008/07/26/strong-opinions-weakly-held/Google Scholar
Saper, C. B., & Lowell, B. B. (2014). The hypothalamus. Current Biology, 24(23), R1111–R1116.CrossRefGoogle ScholarPubMed
Saunders, B. T., & Robinson, T. E. (2012). The role of dopamine in the accumbens core in the expression of Pavlovian-conditioned responses. European Journal of Neuroscience, 36(4), 2521–2532.CrossRefGoogle ScholarPubMed
Schank, R. C., & Abelson, R. P. (1977). Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology Press.Google Scholar
Schulkin, J., & Sterling, P. (2019). Allostasis: A brain-centered, predictive mode of physiological regulation. Trends in Neurosciences, 42(10), 740–752.CrossRefGoogle ScholarPubMed
Sewards, T. V., & Sewards, M. A. (2003). Representations of motivational drives in mesial cortex, medial thalamus, hypothalamus and midbrain. Brain Research Bulletin, 61(1), 25–49.CrossRefGoogle ScholarPubMed
Sohl-Dickstein, J. (2022). Too much efficiency makes everything worse: Overfitting and the strong version of Goodhart's law, November 2022. https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.htmlGoogle Scholar
Speijer, D., Lukeš, J., & Eliáš, M. (2015). Sex is a ubiquitous, ancient, and inherent attribute of eukaryotic life. Proceedings of the National Academy of Sciences of the United States of America, 112(29), 8827–8834.CrossRefGoogle ScholarPubMed
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929–1958.Google Scholar
Veller, C. (2022). Mendel's first law: Partisan interests and the parliament of genes. Heredity, 129(1), 48–55.CrossRefGoogle ScholarPubMed
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., … Silver, D. (2019). Grandmaster level in Starcraft II using multi-agent reinforcement learning. Nature, 575(7782), 350–354.CrossRefGoogle ScholarPubMed
Wittgenstein, L. (1953). Philosophical investigations. Basil Blackwell.Google Scholar
Wood, M. J., Douglas, K. M., & Sutton, R. M. (2012). Dead and alive: Beliefs in contradictory conspiracy theories. Social Psychological and Personality Science, 3(6), 767–773.CrossRefGoogle Scholar
Yassa, M. A., & Stark, C. E. (2011). Pattern separation in the hippocampus. Trends in Neurosciences, 34(10), 515–525.CrossRefGoogle ScholarPubMed