Bad Design = Bad Data = Bad Decisions

It seems straight-forward, right? Unfortunately, the survey designer failed to consider the very likely possibility that some of their respondents may not participate in the activities listed. If, for example, the respondent was not a climber or road biker, how should they respond? Do they leave that row unanswered? Do they select "no change"? The instructions did not provide any guidance.

This oversight made the question confusing, and beyond the concerns mentioned above, calls into question the validity of the data gathered. Imagine if only 10% of respondents are climbers and they all choose "Way more", but the other 90% were confused about how to answer and chose the "no change" option. The retailer would miss an opportunity to better serve this audience.

Fortunately, there are two easy ways to address this issue.

Option One is to provide clear and specific directions regarding how to answer the question. In this case, simply instructing the respondent to skip that answer if they do not participate in the activity would have helped. The downside is that it adds more copy to read, which lengthens the survey. Another concern is that some respondents may think they know what to do and not read the directions. A good survey question should not require a lot of explanation. Unless you have a meaningful relationship with the respondent audience, assume people will zip through your survey. It should be clear what to do at a glance.

Option Two is to provide an N/A (Do not participate in this activity) option in the far right column. This option provides several benefits. First, it is simple and straightforward, does not require explanation, and won't slow down response times. Additionally, it provides more detailed and robust data. In this case, it would not only offer a view into the expected behaviors of those who participate in that activity but also provide the retailer with a snapshot of what percent of their customers actually do that activity. Two types of valuable information from one question!

Good survey design involves putting yourself into the shoes of the respondent and considering all the possible ways they might respond. It requires that you are clear about what you are asking them to do. Remove uncertainty around how they are expected to answer and you will also remove uncertainty around the validity and usefulness of the data. And in turn, you will make better, more confident decisions.