Hostname: page-component-84b7d79bbc-dwq4g Total loading time: 0 Render date: 2024-07-29T05:24:20.825Z Has data issue: false hasContentIssue false
  • English
  • Français

The Reference Class Problem for Credit Valuation in Science

Published online by Cambridge University Press:  01 January 2022

Carole J. Lee
Get access Rights & Permissions [Opens in a new window]

Abstract

Scholars belong to multiple communities of credit simultaneously. When these communities disagree about a scholarly achievement’s credit assignment, this raises a puzzle for decision and game theoretic models of credit seeking in science. The reference class problem for credit valuation in science is the problem of determining to which of an agent’s communities—which reference class—credit determinations should be indexed for an act under some state of nature. Solving this problem requires developing rich, mutually informed theories of community and credit that are sensitive to the structure and status systems of complex, heterogeneous scholarly networks.

Type
Research Article
Information
Philosophy of Science , Volume 87 , Issue 5: PSA 2018 - Proceedings of the 2018 biennial meeting of the Philosophy of Science Association. Part II Symposia Papers , December 2020 , pp. 1026 - 1036
Copyright
Copyright © The Philosophy of Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

†

Many thanks to Christopher Adolph, Melinda Baldwin, Alex Csiszar, Aileen Fyfe, Sheridan Grant, Crystal Hall, Remco Heesen, Liam Kofi-Bright, Jessica Lundquist, Joseph Martin, Conor Mayo-Wilson, Ties Nijssen, Cailin O’Connor, and Kevin Zollman for illuminating conversations. This work is supported by the National Science Foundation under grant 1759825. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation. This research used statistical consulting resources provided by the Center for Statistics and the Social Sciences, University of Washington.

References

AAAS (American Association for the Advancement of Science). 2020. “Mission and Scope.” AAAS. http://sciencemag.org/about/mission-and-scope.Google Scholar
Aalbersberg, IJsbrand Jan, et al. 2017. “Making Science Transparent by Default: Introducing the TOP Statement.” OSF Preprints, February 15. https://doi.org/10.31219/osf.io/sm78t.CrossRefGoogle Scholar
Academic Ranking of World Universities. 2018. “ShanghaiRanking’s Academic Ranking of World Universities 2018 Press Release.” http://www.shanghairanking.com/Academic-Ranking-of-World-Universities-2018-Press-Release.html.Google Scholar
Alberts, Bruce, Kirschner, Marc W., Tilghman, Shirley, and Varmus, Harold. 2014. “Rescuing US Biomedical Research from Its Systematic Flaws.” Proceedings of the National Academy of Sciences 111:5773–77.CrossRefGoogle Scholar
Arrow, Kenneth J. 1950. “A Difficulty in the Concept of Social Welfare.” Journal of Political Economy 58:328–46.CrossRefGoogle Scholar
Baldwin, Melinda. 2015. Making Nature: The History of a Scientific Journal. Chicago: Cambridge University Press.CrossRefGoogle Scholar
Biagioli, Mario. 2002. “From Book Censorship to Academic Peer Review.” Emergences: Journal for the Study of Media and Composite Cultures 12:11–45.CrossRefGoogle Scholar
Blank, Rebecca, Daniels, Ronald J., Gilliland, Gary, Gutmann, Amy, Hawgood, Samuel, Hrabowski, Freeman A., Pollack, Martha E., Price, Vincent, Reif, L. Rafael, and Schlissel, Mark S.. 2017. “A New Data Effort to Inform Career Choices in Biomedicine.” Science 358:1388–89.CrossRefGoogle ScholarPubMed
Boyer, Ernest L. 1990. Scholarship Reconsidered. San Francisco: Carnegie Foundation for the Advancement of Teaching.Google Scholar
Bright, Liam Kofi. 2017. “On Fraud.” Philosophical Studies 174:291–310.CrossRefGoogle Scholar
Bruner, Justin, and O’Connor, Cailin. 2017. “Power, Bargaining, and Collaboration.” In Scientific Collaboration and Collective Knowledge, ed. Boyer-Kassem, Thomas, Mayo-Wilson, Conor, and Weisberg, Michael, 135–57. Oxford: Oxford University Press.Google Scholar
Centola, Damon, Willer, Robb, and Macy, Michael. 2005. “The Emperor's Dilemma: A Computational Model of Self-Enforcing Norms.” American Journal of Sociology 110:1009–40.CrossRefGoogle Scholar
Correll, Shelley J., Ridgeway, Cecilia L., Zuckerman, Ezra W., Jank, Sharon, Jordan-Bloch, Sara, and Nakagawa, Sandra. 2017. “It’s the Conventional Thought That Counts: How Third-Order Inference Produces Status Advantage.” American Sociological Review 82:297–327.CrossRefGoogle Scholar
Crenshaw, Kimberle. 1989. “Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics.” University of Chicago Legal Forum 139:139–68.Google Scholar
Csiszar, Alex. 2015. “Objectivities in Print.” In Objectivity in Science: New Perspectives from Science and Technology Studies, ed. Padovani, Flavia, Richardson, Alan, and Tsou, Jonathan Y., 145–69. Cham: Springer.Google Scholar
DORA (Declaration on Research Assessment). 2013. “The San Francisco Declaration on Research Assessment.” DORA. https://sfdora.org/read/.Google Scholar
Escrigas, Cristina, Granados Sánchez, Jesús, Hall, Budd, and Tandon, Rajesh. 2014. “Knowledge, Engagement and Higher Education: Contributing to Social Change.” In Report: Higher Education in the World, ed. Escrigas, Cristina, Granados Sánchez, Jesús, Hall, Budd, and Tandon, Rajesh. London: Palgrave Macmillan.Google Scholar
Espeland, Wendy Nelson, and Sauder, Michael. 2012. “The Dynamism of Indicators.” In Governance by Indicators: Global Power through Quantification and Rankings, ed. Davis, Kevin, Fisher, Angelina, Kingsbury, Benedict, and Merry, Sally Engle, 86–109. Oxford: Oxford University Press.CrossRefGoogle Scholar
Espeland, Wendy Nelson, and Sauder, Michael. 2016. Engines of Anxiety: Academic Rankings, Reputation, and Accountability. New York: Russell Sage.Google Scholar
Espeland, Wendy Nelson, and Stevens, Mitchell L.. 1998. “Commensuration as a Social Process.” Annual Review of Sociology 24:313–43.CrossRefGoogle Scholar
Fang, Ferric C., and Casadevall, Arturo. 2011. “Retracted Science and the Retraction Index.” Infection and Immunity 79:3855–59.CrossRefGoogle ScholarPubMed
Feldman, Richard. 1985. “Reliability and Justification.” Monist 68:159–74.CrossRefGoogle Scholar
Fraley, R. Chris, and Vazire, Simine. 2014. “The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power.” PLoS ONE 9:e109019. https://doi.org/10.1371/journal.pone.0109019.CrossRefGoogle ScholarPubMed
Heesen, Remco. 2017. “Communism and the Incentive to Share in Science.” Philosophy of Science 84:698–716.CrossRefGoogle Scholar
Hicks, Diana, and Wouters, Paul. 2015. “The Leiden Manifesto for Research Metrics.” Nature 520:429–31.CrossRefGoogle ScholarPubMed
Kitcher, Philip. 1990. “The Division of Cognitive Labor.” Journal of Philosophy 87:5–22.CrossRefGoogle Scholar
Lamont, Michèle. 2009. How Professors Think: Inside the Curious World of Academic Judgment. Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
Larivière, Vincent, Kiermar, Véronique, MacCallum, Catriona J., McNutt, Marcia, Patterson, Mark, Pulverer, Bernd, Swaminathan, Sowmya, Taylor, Stuart, and Curry, Stephen. 2016. “A Simple Proposal for the Publication of Journal Citation Distributions.” BioRxiv. https://doi.org/10.1101/062109.CrossRefGoogle Scholar
Larivière, Vincent, Lozano, George A., and Gingras, Yves. 2013. “Are Elite Journals Declining?” Journal of the Association for Information Science and Technology 65:649–55.Google Scholar
Lee, Carole J. 2012. “A Kuhnian Critique of Psychometric Research on Peer Review.” Philosophy of Science 79:859–70.CrossRefGoogle Scholar
Lee, Carole J.. 2013. “The Limited Effectiveness of Prestige as an Intervention on the Health of Medical Journal Publications.” Episteme 10:387–402.CrossRefGoogle Scholar
Lee, Carole J.. 2015. “Commensuration Bias in Peer Review.” Philosophy of Science 82:1272–83.CrossRefGoogle Scholar
Lee, Carole J., and Moher, David. 2017. “Promote Scientific Integrity via Journal Peer Review.” Science 357:256–57.CrossRefGoogle ScholarPubMed
Lozano, George A., Larivière, Vincent, and Gingras, Yves. 2012. “The Weakening Relationship between the Impact Factor and Papers’ Citations in the Digital Age.” Journal of the American Society for Information Science and Technology 63:2140–45.CrossRefGoogle Scholar
Macrae, C. Neil, Bodenhausen, Galen V., and Milne, Alan B.. 1995. “The Dissection of Selection in Person Perception: Inhibitory Processes in Social Stereotyping.” Journal of Personality and Social Psychology 69:397–407.CrossRefGoogle ScholarPubMed
Martin, Joseph D. 2018. Solid State Insurrection: How the Science of Substance Made American Physics Matter. Pittsburgh: University of Pittsburgh Press.CrossRefGoogle Scholar
Merton, Robert K. 1968. “The Matthew Effect in Science.” Science 159 (3810): 56–63.CrossRefGoogle Scholar
Merton, Robert K.. 1973. “The Normative Structure of Science.” In The Sociology of Science: Theoretical and Empirical Investigations, ed. Storer, Norman W., 267–78. Chicago: Cambridge University Press.Google Scholar
Moher, David, et al. 2019. “The Hong Kong Principles for Assessing Researchers: Fostering Research Integrity.” OSF Preprints, September 17. https://doi.org/10.31219/osf.io/m9abx.CrossRefGoogle Scholar
NASEM (National Academies of Sciences, Engineering, and Medicine). 2018. Open Science by Design: Realizing a Vision for 21st Century Research. Washington, DC: National Academies.Google Scholar
National Science Foundation. 2015. “Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science.” Report of the Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences.Google Scholar
Nature. 1869. “Nature: A Weekly Illustrated Journal of Science.” Nature 1 (2).Google Scholar
Nature Publishing Group. 2015. “Author Insights 2015 Survey.” figshare. https://doi.org/10.6084/m9.figshare.1425362.v7.CrossRefGoogle Scholar
Nosek, B. A., et al. 2015. “Promoting an Open Research Culture: Author Guidelines for Journals Could Help to Promote Transparency, Openness, and Reproducibility.” Science 348:1422–25.CrossRefGoogle Scholar
Okasha, Samir. 2016. “On the Interpretation of Decision Theory.” Economics and Philosophy 32:409–33.CrossRefGoogle Scholar
Ormans, Laurent. 2016. “50 Journals Used in Ft Research.” Financial Times, September 12. https://www.ft.com/content/3405a512-5cbb-11e1-8f1f-00144feabdc0.Google Scholar
Ridgeway, Cecilia L., and Correll, Shelley J.. 2006. “Consensus and the Creation and Status Beliefs.” Social Forces 85:431–53.CrossRefGoogle Scholar
Rubin, Hannah, and O’Connor, Cailin. 2018. “Discrimination and Collaboration in Science.” Philosophy of Science 85:380–402.CrossRefGoogle Scholar
Sauder, Michael, and Espeland, Wendy Nelson. 2006. “Strength in Numbers? The Advantages of Multiple Rankings.” Indiana Law Journal 81:205–27.Google Scholar
Schimmack, Ulrich. 2015. “Replicability Ranking of 26 Psychology Journals.” Replicability-Index, August 13. https://replicationindex.wordpress.com/2015/08/13/replicability-ranking-of-26-psychology-journals/.Google Scholar
Strevens, Michael. 2003. “The Role of the Priority Rule in Science.” Journal of Philosophy 100:55–79.CrossRefGoogle Scholar
Szell, Michael, Ma, Yifang, and Sinatra, Roberta. 2018. “A Nobel Opportunity for Interdisciplinarity.” Nature Physics 14:1075–78.CrossRefGoogle Scholar
West, Jevin D., Bergstrom, Theodore C., and Bergstrom, Carl T.. 2010. “The Eigenfactor Metrics: A Network Approach to Assessing Scholarly Journals.” College and Research Libraries 71:236–44.CrossRefGoogle Scholar
Willer, Robb, Kuwabara, Ko, and Macy, Michael W.. 2009. “The False Enforcement of Unpopular Norms.” American Journal of Sociology 115:451–90.CrossRefGoogle ScholarPubMed
Wilsdon, James, Bar-Ilan, Judit, Frodeman, Robert, Lex, Elisabeth, Peters, Isabella, and Wouters, Paul. 2017. “Next-Generation Metrics: Responsible Metrics and Evaluation for Open Science.” Report of the European Commission Expert Group on Altmetrics.CrossRefGoogle Scholar
Wilsdon, James, et al. 2015. The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. London: Sage.CrossRefGoogle Scholar
Zollman, Kevin J. S. 2018. “The Credit Economy and the Economic Rationality of Science.” Journal of Philosophy 115:5–33.CrossRefGoogle Scholar
Zuckerman, Harriet, and Merton, Robert K.. 1971. “Patterns of Evaluation in Science: Institutionalisation, Structure and Functions of the Referee System.” Minerva 9:66–100.CrossRefGoogle Scholar