Successful market researchers know that to tell a powerful story that is both credible and brings customer insights to life, they must back up their arguments with relevant statistics from a high-quality research study. But ensuring that a survey delivers high-quality data can be difficult, especially when respondents are sourced from broad general audience panels, aren’t required to have their identities verified (for privacy reasons), or are offered monetary incentives.

Even when data is collected from a reputable sample provider and you are practicing good survey design techniques, some survey respondents will still race through the answers. This jeopardizes survey quality and may even invalidate survey findings. These unqualified, inattentive, or even fraudulent respondents must be filtered out or they can add unnecessary noise to collected data.

This filtering process can be accomplished by using attention checks, also known as trap questions, at strategic points in the survey. In addition to filtering out inattentive respondents, they also can filter out bots, people who are being dishonest in their answers, or those who don’t even speak the language.

Benefits of trap questions
Based on data collected from GroupSolver’s own studies, implementing trap questions can identify and filter out the 15-30% of respondents that typically should be disqualified because they are not answering carefully – usually because they are simply not paying enough attention. The questions are not intended to trick respondents or test their knowledge; each should be easy and obvious to answer. Rather, they are inserted at strategic points in the survey to filter out respondents who aren’t taking the survey seriously.

Trap questions should be used in every survey – even when respondents come from a very specialized and highly managed communities. Although bad respondents typically come from general population panels, high-cost specialized panels can suffer from inattentive respondents, as well. For this reason, we always recommend independent validation of respondent quality inside of every survey through the use of trap questions.

Trap questions that screen for inattentive respondents can be asked using both open-end and multiple-choice questions, though multiple-choice questions have traditionally been a little bit easier to implement. This was because open-ended questions previously required researchers to manually review, code, and categorize answers provided in the traditional text box.

However, the latest survey platforms apply natural language processing (NLP) to eliminate this chore. NLP and other tech-enabled techniques can be employed to quickly process the free-text answers, remove duplicates and noise, and only keep those that are unique and meaningful. Using these techniques to ask open-ended trap questions makes it easier to flag, in real-time, those respondents who have provided gibberish or zero value statements and who therefore should be terminated from the survey, so they do not provide data of dubious quality.

But here, I’d like to focus on multiple-choice trap questions. There are various types that can be used in any survey, and they should be tailored to test different types of problematic respondent behaviors.

How to implement trap questions
Ideally, more than one trap question should be used in each survey. Reviewing data from a sample of GroupSolver’s recent studies, the first trap question identifies the nearly 12% of all survey takers, on average, who are inattentive. If a second trap is inserted, an additional 7% of the total respondent pool is generally caught. With a third trap question, another 5% of the respondents have been identified. These inattentive respondents must be identified as soon as possible and terminated from the study or they will contaminate the results.

Stop Speeders, Cheaters, and Bots From Putting Your Survey Data Quality at Risk

Rasto Ivanic | 21 Jun 2021

While no survey technique is 100 percent foolproof, using the quick and simple strategy of deploying trap questions will help improve the quality of your data – and your confidence that the derived insights will be valid and actionable.

Log in or Create an Informa Connect Account to Access this content