Yes stormtrooper, these are the droids you are looking for: Identifying and preliminarily evaluating bot and fraud detection strategies in online psychological research.

Abstract

Bots (i.e., automated software programs that perform various tasks) and fraudulent responders pose a growing and costly threat to psychological research as well as affect data integrity. However, few studies have been published on this topic. (a) Describe our experience with bots and fraudulent responders using a case study, (b) present various bot and fraud detection tactics (BFDTs) and identify the number of suspected bot and fraudulent respondents removed, (c) propose a consensus confidence system for eliminating bots and fraudulent responders to determine the number of BFDTs researchers should use, and (d) examine the initial effectiveness of dynamic versus static BFDT protocols. This study is part of a larger 14-day experience sampling method study with trauma-exposed sexual minority cisgender women and transgender and/or nonbinary people. Faced with several bot and fraudulent responder infiltrations during data collection, we developed an evolving BFDT protocol to eliminate bots and fraudulent responders. Throughout this study, we received 24,053 responses on our baseline survey. After applying our BFDT protocols, we eliminated 99.75% of respondents that were likely bots or fraudulent responders. Some BFDTs seemed to be more effective and afford higher confidence than others, dynamic protocols seemed to be more effective than static protocols, and bots and fraudulent responders introduced significant bias in the results. This study advances online psychological research by curating one of the largest samples of bot and fraudulent respondents and pilot testing the largest number of BFDTs to date. Recommendations for future research are provided. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

Topics & Concepts