Detecting Fraudulent Survey Responses
General Information
Fraudulent survey data consists of an intentional deviation from the stated guidelines, instructions or sampling procedures by any member of the survey project, including interviewers, supervisors, data entry personnel, the project leaders or the principal investigator, that results in a contamination of the data (Robbins, 2018). Fraudulent or fabricated survey data could include some of the following: selecting the wrong respondent, misreading the question, duplicating survey responses, misrecording a response, and creating data.
It is important to note that fabricated/fraudulent survey data is different from survey error. For example, intentionally selecting a house that was not originally in the survey sampling plan just because people are home is intentional, and therefore, fraudulent survey data collection. However, accidentally selecting a house that was not originally in the survey sampling plan by miscounting the skip pattern is unintentional, and therefore, this would be considered a survey error.
There are many motivations for why someone would fabricate survey data or collect the wrong data on purpose. One might do this to save time and money, to cover up a mistake, lack of incentive to improve methodology, the questions are too sensitive to ask, etc. To detect fraudulent survey data, one could record portions of interviews, use GPS trackers to make sure data collectors are going to the correct locations (also known as CAPI or computer-assisted personal interviewing), use PercentMatch to prevent duplicates, have supervisors attend interviews, etc. In addition, providing survey collection training and providing a financial incentive for doing great work could also limit fraudulent or fabricated data. To learn more about fabricated/fraudulent survey data please see the sources below.
Here are some resources on the technique:
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014). Separating the shirkers from the workers? Making sure respondents pay attention on self‐administered surveys. American Journal of Political Science, 58(3), 739-753.
Berinsky, A. J., Margolis, M. F., & Sances, M.W. (2016). Methods to Identify Inattentive Respondents. American Journal of Political Science.
Ceci, S. (2009, January 30). How Do You Know if People are Lying on a Survey? ILR Survey Research Institute’s Annual Speaker Series, Ithaca, New York.
Chesney, T. & Penny, K. (2013). The Impact of Repeated Lying on Survey Results. SAGE Open, 3.
Clifford, S., & Jerit, J. (2016). Cheating on Political Knowledge Questions in Online Surveys. Public Opinion Quarterly, 80(4), 858–887.
Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19.
Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior research methods, 48(1), 400-407.
Kanazawa, S. (2005). Who Lies on Surveys, and What Can We Do About It? Journal of Social Political and Economic Studies, 3, 361.
Kennedy, C., Hatley, N., Lau, A., Mercer, A., Keeter, S., Ferno, J., & Asare-Marfo, D. (2020). Two common checks fail to catch most bogus cases. Assessing the Risks to Online Polls from Bogus Respondents. Pew Research Center.
Lopez, J. & Hillygus, D. S. (2018). Why So Serious? Survey Trolls and Misinformation. Social Science Research Network.
Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83.
Moattar, K. (2019). How to prevent cheating in your surveys, technologies, best practices, ideas, and examples. SurveyLegend.
Pei, W., Mayer, A., Tu, K., & Yue, C. 2020. Attention Please: Your Attention Check Questions in Survey Studies Can Be Automatically Answered. In Proceedings of The Web Conference 2020 (WWW ’20). Association for Computing Machinery, New York, NY, USA, 1182–1193.
Perkel, J. M. (2020). Mischief-making bots attacked my scientific survey. Nature 579, 461.
Teitcher, J. E. F., Bockting, W. O., Bauermeister, J. A., Hoefer, C. J., Miner, M. H., & Klitzman, R. L. (2015). Detecting, Preventing, and Responding to Fraudsters in Internet Research: Ethics and Tradeoffs. Journal of Law, Medicine and Ethics, 43(1), 114–131.
To Catch a Survey Cheater. (2009). Grey Matter Research and Consulting, Phoenix, Arizona.
Using attention checks as a measure of data quality. (2020). Prolific Researcher Help Centre.
Vannette, D. (2017). Using Attention Checks in Your Surveys May Harm Data Quality. Qualtrics Blog.
Versta. (2018). How to Find and Eliminate Cheaters, Liars, and Trolls in Your Survey. Versta Newsletter.
3 types of attention checks. (2015). MTurk for Academics.