Recently we told you about the power of using red herring, or trick questions, to ensure data quality. In the spirit of Halloween, we’re now going to tell you how you can use these trick questions to treat your respondents to a better online survey experience.
People stop paying close attention when taking surveys for a variety of reasons, but most legitimately want to provide their honest feedback and opinions. They merely become distracted or disengaged as they complete the questionnaire.
As we’ve stated before, the longer the survey is, the more likely people will stop paying close attention to the questions. We hypothesized that if respondents know up front that there are trick questions in the survey, they will pay close attention throughout.
To test this theory, we decided to run an experiment. On one survey, we had two red herring questions – one in a matrix in the middle of the questionnaire, and one as a stand-alone question near the end. On the other survey, we added a third red herring question, near the beginning of the questionnaire with the screening criteria. Both studies were n=1000 and fielded across the state of Oklahoma.
The results were amazing! On the survey that only had red herring questions in the middle and end, we had to remove 20% of the responses for failing the quality control checks. But on the survey with a red herring question up front, only 7% had to be removed. This proved our theory that respondents are more alert and engaged throughout the survey when they know their responses will be checked for accuracy.
Adding a red herring question at the beginning of your study is a smart move. It results in higher quality data and a better survey experience for the respondent. A win-win for all!
More Like This
Stewart is passionate about all things research. Designing studies, analyzing data, and moderating interviews - Stew does it all. Outside of research, he enjoys spending time with his wife and kids and trying new recipes on the grill.
