Too Anecdotal to Be True? Mechanical Turk Is Not All Bots and Bad Data: Response to Webb and Tangney (2022)
Melissa G. Keith, Alexander McKay
Abstract
In response to Webb and Tangney (2022) we call into question the conclusion that data collected on Amazon's Mechanical Turk (MTurk) was "at best-only 2.6% valid" (p. 1). We suggest that Webb and Tangney made certain choices during the study-design and data-collection process that adversely affected the quality of the data collected. As a result, the anecdotal experience of these authors provides weak evidence that MTurk provides low-quality data as implied. In our commentary we highlight best practice recommendations and make suggestions for more effectively collecting and screening online panel data.
Topics & Concepts
Data qualityData collectionPsychologyQuality (philosophy)Best practiceAmazon rainforestData scienceProcess (computing)Computer scienceSociologyEpistemologyEngineeringPolitical scienceOperations managementPhilosophyBiologyOperating systemLawMetric (unit)EcologySocial scienceMobile Crowdsensing and CrowdsourcingComputational and Text Analysis MethodsDigital Marketing and Social Media