No CrossRef data available.
Article contents
Against naïve induction from experimental data
Published online by Cambridge University Press: 05 February 2024
Abstract
This commentary argues against the indictment of current experimental practices such as piecemeal testing, and the proposed integrated experiment design (IED) approach, which we see as yet another attempt at automating scientific thinking. We identify a number of undesirable features of IED that lead us to believe that its broad application will hinder scientific progress.
- Type
- Open Peer Commentary
- Information
- Copyright
- Copyright © The Author(s), 2024. Published by Cambridge University Press
References
Birnbaum, M. H. (2008). New paradoxes of risky decision making. Psychological Review, 115, 463–501.CrossRefGoogle ScholarPubMed
Chang, H. (2004). Inventing temperature: Measurement and scientific progress. Oxford University Press.CrossRefGoogle Scholar
Cox, G. E., & Shiffrin, R. M. (2017). A dynamic approach to recognition memory. Psychological Review, 124, 795–860.CrossRefGoogle ScholarPubMed
Dunn, J. C., & Rao, L. L. (2019). Models of risky choice: A state-trace and signed difference analysis. Journal of Mathematical Psychology, 90, 61–75.CrossRefGoogle Scholar
Garcia-Marques, L., & Ferreira, M. B. (2011). Friends and foes of theory construction in psychological science: Vague dichotomies, unified theories of cognition, and the new experimentalism. Perspectives on Psychological Science, 6, 192–201.CrossRefGoogle ScholarPubMed
Hacking, I. (1983). Representing and intervening: Introductory topics in the philosophy of natural science. Cambridge University Press.CrossRefGoogle Scholar
Hotaling, J. M., Donkin, C., Jarvstad, A., & Newell, B. R. (2022). MEM-EX: An exemplar memory model of decisions from experience. Cognitive Psychology, 138, 101517.CrossRefGoogle ScholarPubMed
Humphreys, M. S., Bain, J. D., & Pike, R. (1989). Different ways to cue a coherent memory system: A theory for episodic, semantic, and procedural tasks. Psychological Review, 96, 208–233.CrossRefGoogle Scholar
Kellen, D. (2019). A model hierarchy for psychological science. Computational Brain & Behavior, 2, 160–165.CrossRefGoogle Scholar
Kellen, D., Steiner, M. D., Davis-Stober, C. P., & Pappas, N. R. (2020). Modeling choice paradoxes under risk: From prospect theories to sampling-based accounts. Cognitive Psychology, 118, 101258.CrossRefGoogle ScholarPubMed
Lewandowsky, S., Oberauer, K., & Brown, G. D. (2009). No temporal decay in verbal short-term memory. Trends in Cognitive Sciences, 13, 120–126.CrossRefGoogle ScholarPubMed
Maxwell, J. C. (1860/1965). General considerations concerning scientific apparatus. In Niven, W. D. (Ed.), The scientific papers of James Clerk Maxwell (Vol. 2, pp. 505–522). Dover.Google Scholar
Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108–141.CrossRefGoogle Scholar
Oberauer, K., Lewandowsky, S., Awh, E., Brown, G. D., Conway, A., Cowan, N., … Ward, G. (2018). Benchmarks for models of short-term and working memory. Psychological Bulletin, 144, 885–958.CrossRefGoogle ScholarPubMed
Peterson, J. C., Bourgin, D. D., Agrawal, M., Reichman, D., & Griffiths, T. L. (2021). Using large-scale experiments and machine learning to discover theories of human decision-making. Science (New York, N.Y.), 372, 1209–1214.CrossRefGoogle ScholarPubMed
Proulx, T., & Morey, R. D. (2021). Beyond statistical ritual: Theory in psychological science. Perspectives on Psychological Science, 16, 671–681.CrossRefGoogle ScholarPubMed
Roediger, H. L. (2008). Relativity of remembering: Why the laws of memory vanished. Annual Review of Psychology, 59, 225–254.CrossRefGoogle ScholarPubMed
Roediger, H. L. III, & Blaxton, T. A. (1987). Retrieval modes produce dissociations in memory for surface information. In Gorfein, D. S. & Hoffman, R. R. (Eds.), Memory and learning: The Ebbinghaus Centennial conference (pp. 349–379). Erlbaum.Google Scholar
Seamon, J. G., Williams, P. C., Crowley, M. J., Kim, I. J., Langer, S. A., Orne, P. J., & Wishengrad, D. L. (1995). The mere exposure effect is based on implicit memory: Effects of stimulus type, encoding conditions, and number of exposures on recognition and affect judgments. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 711–721.Google Scholar
Shiffrin, R. M., Börner, K., & Stigler, S. M. (2018). Scientific progress despite irreproducibility: A seeming paradox. Proceedings of the National Academy of Sciences of the United States of America, 115, 2632–2639.CrossRefGoogle ScholarPubMed
Shiffrin, R. M., & Nobel, P. A. (1997). The art of model development and testing. Behavior Research Methods, Instruments, & Computers, 29, 6–14.CrossRefGoogle Scholar
Singmann, H., Kellen, D., Cox, G. E., Chandramouli, S. H., Davis-Stober, C. P., Dunn, J. C., … Shiffrin, R. M. (2023). Statistics in the service of science: Don't let the tail wag the dog. Computational Brain & Behavior, 6, 64–83.CrossRefGoogle Scholar
Trendler, G. (2009). Measurement theory, psychology and the revolution that cannot happen. Theory & Psychology, 19, 579–599.CrossRefGoogle Scholar
Turner, B. M. (2019). Toward a common representational framework for adaptation. Psychological Review, 126, 660–692.CrossRefGoogle Scholar
Vergauwe, E., & Cowan, N. (2015). Working memory units are all in your head: Factors that influence whether features or objects are the favored units. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 1404–1416.Google ScholarPubMed
Target article
Beyond playing 20 questions with nature: Integrative experiment design in the social and behavioral sciences
Related commentaries (31)
Against naïve induction from experimental data
Are language–cognition interactions bigger than a breadbox? Integrative modeling and design space thinking temper simplistic questions about causally dense phenomena
Assume a can opener
Beyond integrative experiment design: Systematic experimentation guided by causal discovery AI
Commensurability engineering is first and foremost a theoretical exercise
Confidence in research findings depends on theory
Consensus meetings will outperform integrative experiments
Dimensional versus conceptual incommensurability in the social and behavioral sciences
Discovering the unknown unknowns of research cartography with high-throughput natural description
Diversity of contributions is not efficient but is essential for science
Don't let perfect be the enemy of better: In defense of unparameterized megastudies
Eliminativist induction cannot be a solution to psychology's crisis
Experiment commensurability does not necessitate research consolidation
Explore your experimental designs and theories before you exploit them!
Getting lost in an infinite design space is no solution
Individual differences do matter
Integrative design for thought-experiments
Integrative experiments require a shared theoretical and methodological basis
Is generalization decay a fundamental law of psychology?
Measurement validity and the integrative approach
Neuroadaptive Bayesian optimisation can allow integrative design spaces at the individual level in the social and behavioural sciences… and beyond
Phenomena complexity, disciplinary consensus, and experimental versus correlational research in psychological science
Representative design: A realistic alternative to (systematic) integrative design
Sampling complex social and behavioral phenomena
Some problems with zooming out as scientific reform
Test many theories in many ways
The elephant's other legs: What some sciences actually do
The future of experimental design: Integrative, but is the sample diverse enough?
The miss of the framework
The social sciences needs more than integrative experimental designs: We need better theories
There are no shortcuts to theory
Author response
Replies to commentaries on beyond playing 20 questions with nature