Hostname: page-component-cd9895bd7-p9bg8 Total loading time: 0 Render date: 2024-12-28T15:21:46.142Z Has data issue: false hasContentIssue false
  • English
  • Français

Caution! MTurk Workers Ahead—Fines Doubled

Published online by Cambridge University Press:  28 July 2015

P. D. Harms  and
Justin A. DeSimone
P. D. Harms*
Affiliation:
University of Alabama
Justin A. DeSimone
Affiliation:
University of Cincinnati
*
Correspondence concerning this article should be addressed to P. D. Harms, University of Alabama, 101 Alston Hall, Box 870225, 361 Stadium Drive, Tuscaloosa, AL 35487. E-mail: pharms@gmail.com
Get access Rights & Permissions [Opens in a new window]

Extract

Landers and Behrend (2015) are the most recent in a long line of researchers who have suggested that online samples generated from sources such as Amazon's Mechanical Turk (MTurk) are as good as or potentially even better than the typical samples found in psychology studies. It is important that the authors caution that researchers and reviewers need to carefully reflect on the goals of research when evaluating the appropriateness of samples. However, although they argue that certain types of samples should not be dismissed out of hand, they note that there is only scant evidence demonstrating that online sources can provide usable data for organizational research and that there is a need for further research evaluating the validity of these new sources of data. Because the target article does not directly address the potential problems with such samples, we will review what is known about collecting online data (with a particular focus on MTurk) and illustrate some potential problems using data derived from such sources.

Type
Commentaries
Information
Industrial and Organizational Psychology , Volume 8 , Issue 2 , June 2015 , pp. 183 - 190
Copyright
Copyright © Society for Industrial and Organizational Psychology 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Berinsky, A., Huber, G., & Lenz, G. (2012). Using Mechanical Turk as a subject recruitment tool for experimental research. Political Analysis, 20, 351–368.Google Scholar
Bohannon, J. (2011). Social science for pennies. Science, 334, 307.Google Scholar
Buhrmester, M. D., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5.CrossRefGoogle Scholar
Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaiveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavioral Research, 46, 112–130.Google Scholar
Deneme. (2009, December 21). Deneme: A blog of experiments on Amazon Mechanical Turk. http://groups.csail.mit.edu/uid/deneme/?p=523Google Scholar
DeSimone, J. A., Harms, P. D., & DeSimone, A. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171–181.CrossRefGoogle Scholar
Feitosa, J., Joseph, D., & Newman, D. (2015). Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English. Personality and Individual Differences, 75, 47–52.Google Scholar
Henrich, J., Heine, S., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33, 61–135.Google Scholar
Landers, R., & Behrend, T. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice.Google Scholar
Mason, W., & Suri, S. (2011). Conducting behavioral research on Amazon's Mechanical Turk. Behavioral Research, 44, 1–23.CrossRefGoogle Scholar
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.Google Scholar
Rand, D. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179.Google Scholar
Ross, J., Irani, I., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010). Who are the crowdworkers? Shifting demographics in Amazon Mechanical Turk. In Edwards, K. & Rodden, T. (Eds.), Proceedings of the ACM Conference on Human Factors in Computing Systems (pp. 2863–2872). New York, NY: ACM.Google Scholar
Shapiro, D., Chandler, J., & Mueller, P. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213–220.Google Scholar