3 ways verbatim answers add quality to survey results

By bruce mars @brucemars at Unsplash

Open-ended questions in online surveys have in the past not produced great results. But with advances in online survey software, nowadays such questions can yield rich verbatim answers. 

In this article we look at how such answers add quality to survey results.

1. Better qualified respondents

Many survey respondents can recognise, and therefore navigate past, the screener or qualifying questions in questionnaires. 

For example, consider a planned survey with dog owners on behalf of a dog food brand. The survey respondents are going to be found by sending out email invitations to a cross section of the population, who may or may not own a dog. 

In order to identify dog owners from among those people who receive the email invitation and then click on the link to begin the survey, the questionnaire could have a question at or near the start asking: “Do you own a dog?” (Yes/No). 

The issue with this, however, is that people who are used to taking surveys (because they regularly receive invitations to do so), will realise that if they answer “No” to this question, they will be prevented from completing the survey, and therefore from receiving the reward. So some will answer “Yes” even if they do not own a dog.

Even if the question asks “Which of the following types of pets do you own?” many respondents will guess the survey is about one of the answers. 

So asking an open-ended question such as “What outdoor activities do you do on a typical weekend?” is better, because it gives no clue about the subject of the research, resulting in much-better qualified respondents taking part in the survey. Anyone whose verbatim answer does not include reference to “walking the dog” will be screened out, so that only genuine dog owners can go on to complete the questionnaire.

Of course, a respondent might answer “walking” in their response without mentioning a dog. But in this case we can have a follow-up probing question asking them to give more detail about their walking activity.

2. Results in respondents’ own words

Continuing with our example of a survey about dog food, the client might want to include a question asking respondents how they would describe the client’s brand or packaging, or those of their competitors. 

Very often such a question would be asked in a closed multi-response format, with a list of possible words that respondents could choose from. There are several problems with this approach. 

Firstly, if the list of words is quite short, many respondents will nevertheless feel obliged to select one or more of the answers. This would be the case even if there is a “None of these” option, because respondents are often reluctant to select this.

Secondly, if the list of words is quite long, respondents may not read all the words, and therefore give answers which do not reflect their true thoughts. 

But thirdly, and most importantly, is that the words will come from the client rather than respondents, and therefore the results will be potentially misleading and incomplete.

So the alternative to a closed question is to ask an open-ended question, giving the respondents the opportunity to use their own choices of words.

The issues above don’t just apply to questions about brand perceptions, by the way. They can apply to any question.

3. Indications of engagement

Survey respondents can often be asked an open-ended question to explain their reasons for their answer to an earlier question.

So, they might have been asked to choose which of three new product concepts was most appealing to them, and then asked an open-ended question to explain what they particularly liked about the concept they preferred, and why. 

The verbatim answers to such open-ended questions can be very revealing. Apart from the insight they contain, they can also give a clear indicator about the level of engagement in the survey by each respondent. 

Respondents who clearly have not written a “proper” answer can be removed from a survey, and replacement respondents recruited instead.