Hostname: page-component-848d4c4894-ndmmz Total loading time: 0 Render date: 2024-05-31T07:31:22.763Z Has data issue: false hasContentIssue false
  • English
  • Français

The New Fruit Fly for Applied Psychological Research

Published online by Cambridge University Press:  28 July 2015

Scott Highhouse  and
Don Zhang
Scott Highhouse*
Affiliation:
Bowling Green State University
Don Zhang
Affiliation:
Bowling Green State University
*
Correspondence concerning this article should be addressed to Scott Highhouse, Department of Psychology, Bowling Green State University, Bowling Green, OH 43403. E-mail: shighho@bgsu.edu
Get access Rights & Permissions [Opens in a new window]

Extract

Landers and Behrend (2015) present yet another attempt to limit reviewer and editor reliance on surface characteristics when evaluating the generalizability of study results (see also Campbell, 1986; Dipboye & Flanagan, 1979; Greenberg, 1987; Highhouse, 2009; Highhouse & Gillespie, 2009). Most of the earlier treatments of sample generalizability, however, have focused on the use of college students in (mostly) laboratory studies. Many industrial–organizational (I-O) scholars have experienced the hostility with which studies using students as participants receives. For instance, Jen Gillespie and I observed, “Reviewers and editors commonly assert that students should not be used to study workplace phenomena as though such a declaration requires no further explanation” (Highhouse & Gillespie, 2009, p. 247). The difference this time, however, is that Landers and Behrend (2015) are reacting to dismissals of research using Mechanical Turk (MTurk) workers to make inferences about behavior in organizations. Landers and Behrend (2015) make the important point that any research population is likely to be atypical on some dimensions and that all samples are samples of convenience (see also Oakes, 1972). We agree. Furthermore, we make two observations about MTurk: (a) We believe that it should be met with less resistance than student samples have historically faced, and (b) we suggest that it provides a unique opportunity to bring back randomized experimentation in I-O psychology.

Type
Commentaries
Information
Industrial and Organizational Psychology , Volume 8 , Issue 2 , June 2015 , pp. 179 - 183
Copyright
Copyright © Society for Industrial and Organizational Psychology 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aguinis, H., & Bradley, K. J. (2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods.CrossRefGoogle Scholar
Brinberg, D., & McGrath, J. E. (1985). Validity and the research process. Newbury Park, CA: Sage.Google Scholar
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality data? Perspectives on Psychological Science, 6, 3–5.Google Scholar
Calder, B. J., Phillips, L. W., & Tybout, A. M. (1981). Designing research for application. Journal of Consumer Research, 8, 197–207.Google Scholar
Campbell, J. P. (1986). Labs, fields, and straw issues. In Locke, E. A. (Ed.), Generalizing from laboratory to field settings (pp. 269–279). Lexington, MA: Heath.Google Scholar
Colquitt, J. A. (2008). From the editors: Publishing laboratory research in AMJ: A question of when, not if. Academy of Management Journal, 51 (4), 616–620.Google Scholar
Dipboye, R. L., & Flanagan, M. F. (1979). Research settings in industrial and organizational psychology: Are findings in the field more generalizable than in the laboratory? American Psychologist, 34, 141–150.CrossRefGoogle Scholar
Farber, M. L. (1952). The college student as laboratory animal. American Psychologist, 7, 102.Google Scholar
Fromkin, H. L., & Streufert, S. (1976). Laboratory experimentation. In Dunnette, M. D. (Ed.), Handbook of industrial and organizational psychology (pp. 415–465). Chicago, IL: Rand McNally.Google Scholar
Gilovich, T., Griffin, D., & Kahneman, D. (2002). Heuristics and biases: The psychology of intuitive judgment. Cambridge, England: Cambridge University Press.Google Scholar
Greenberg, J. (1987). The college sophomore as guinea pig: Setting the record straight. Academy of Management Review, 12, 157–159.Google Scholar
Greenberg, J., & Tomlinson, E. C. (2004). Situated experiments in organizations: Transplanting the lab to the field. Journal of Management, 30 (5), 703–724.CrossRefGoogle Scholar
Highhouse, S. (2009). Designing experiments that generalize. Organizational Research Methods, 12, 554–566.Google Scholar
Highhouse, S., & Gillespie, J. Z. (2009). Do samples really matter that much? In Lance, C. E. & Vandenberg, R. J. (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 249–268). New York, NY: Routledge.Google Scholar
Kardes, F. R. (1996). In defense of experimental consumer psychology. Journal of Consumer Psychology, 5, 279–296.CrossRefGoogle Scholar
Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizations, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice.Google Scholar
Locke, E. A. (1986). Generalizing from laboratory to field: Ecological validity or abstraction of essential elements? In Locke, E. A. (Ed.), Generalizing from laboratory to field settings (pp. 257–267). Lexington, MA: Heath.Google Scholar
Mitchell, G. (2012). Revisiting truth or triviality the external validity of research in the psychological laboratory. Perspectives on Psychological Science, 7 (2), 109–117.CrossRefGoogle ScholarPubMed
Mook, D. (1983). In defense of external invalidity. American Psychologist, 38, 379–387.CrossRefGoogle Scholar
Oakes, W. (1972). External validity and the use of real people as subjects. American Psychologist, 27, 959–962.Google Scholar
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188.Google Scholar
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.CrossRefGoogle Scholar
Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Journal of Personality and Social Psychology, 51 (3), 515.CrossRefGoogle Scholar
Shen, W., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., & Ones, D. S. (2011). Samples in applied psychology: Over a decade of research in review. Journal of Applied Psychology, 96 (5), 1055–1064.CrossRefGoogle Scholar
Stone-Romero, E. F. (2002). The relative validity and usefulness of various empirical research designs. In Rogelberg, S. G. (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 77–98). Malden, MA: Blackwell.Google Scholar