02 Dec 2014

Surveys: An ABC Guide, part 8

The art of designing the right questions

The art of designing the right questions

There are good questions and there are bad questions. The final two parts of this survey school will look at this important subject. What happens is that poor questions tend never to be answered, while good questions can even provide you with bonus information. Below we will show some examples of how to design questions that make it easier for the respondents and how to avoid irritation and incomplete surveys.

A good question entices the truth from the respondent. The questions must not be intrusive or feel threatening. If respondents are worried about what the consequences of their reply may be to them there is a risk that the answer will not be an honest one. Anonymous surveys tend to produce more honest answers. If the questions include sensitive subjects, ensure that you point out the privacy and confidentiality policy that applies.

A good question produces an unequivocal answer. The question should be worded so that it can only be answered in one way. If the question is ambiguous, you will get ambiguous answers. For example, let us say you ask the question: ‘Do you think our gym and service are good?’ If you get the answer ‘No’, you cannot be certain whether the customer dislikes the gym or dislikes the service. Or both. Had you asked a good question you would have avoided having to guess the answer.

A good question can handle all reply options.

Multiple choice questions are popular as they are easy for respondents to complete and easy for you as the survey designer to analyse. However, be careful to ensure that the reply options to the question apply in all situations. Let us look at the following example of an incorrect question:

Where have you lived in your life until now?

A)   In a detached house

B)   In a flat (apartment)

There are several problems with a question like this. For example, the respondent may have lived in both. The respondent may also have lived in a semi-detached house or terrace house, so how do they answer in that case? Suppose the respondent lives in two places, with a flat (apartment) in one place and a detached house in the other. There are ways of overcoming this.

One way is to make the question dichotomous (split into separate paths). You would then begin by asking:

Have you lived in a detached house?

( ) Yes

( ) No

Have you lived in a flat (apartment)?

( ) Yes

( ) No

The other way to overcome the issue with the question is to provide several different reply options, where respondents can enter more than one reply to cover all options:

Where have you lived during your life?

(Tick all that apply).

( ) In a detached house

( ) In a terrace house

( ) In a semi-detached house

( ) In a flat (apartment)

( ) I have been homeless

( ) Other

A good question produces nuanced answers.

If your question does not allow for nuances, you will wonder why you asked the question at all. Suppose that you ask the question:

What do you think of our pizza?

A) It’s the best I’ve tasted.

B) It’s the worst I’ve tasted.

C) Somewhere between the two.

Here, we may suspect that everyone will put a cross against ‘C’. If so, what have you gained by asking that question?

A good question has a good link back to the preceding one.

Writing questionnaires has a lot in common with writing text in general – it needs to fit together and there needs to be a common thread. There must also be a smooth transition from one part to the next. It is the same with questions. Grouping together questions relating to the same topic creates a cohesive impression, and the respondent feels more at ease. Questions that jump from one topic to another can confuse the respondent and cause them to give up.

A good question does not begin with a baseless assumption

One of the most common mistakes when designing questionnaires is where the questioner makes baseless assumptions. A clear example of this mistake is as follows:

Are you satisfied with your theft insurance?

( ) Yes

( ) No

This question starts with the assumption that the person actually has theft insurance. How should the person respond if they do not have theft insurance? You can easily overcome this by adding a further reply option: ‘Yes’, ‘No’ and ‘I do not have theft insurance’.

Another very common error is where the questioner thinks that the respondent knows about everything that is being asked about. Many surveys ask very specific questions, which the respondent is sometimes unable to answer.

Nor can you count on the respondent making the effort to find out the answer. For example:

What percentage of your waking hours do you spend listening to the radio?

Free text answer

Not everyone keeps tabs on this; probably, even fewer would take the time to calculate the exact percentage. Therefore many prefer to leave the question unanswered – or, worse, quit the entire survey.

If nonetheless you must have an answer to the question, be careful to point out that it need only be an estimate, or a rapid ‘finger in the air’ guess.

You should also be careful to ensure that everyone who takes part in the survey has the opportunity to answer every question. This is usually achieved by adding the reply option:

‘Don’t know’. For example:

Do you think that the report on municipal school health care is reliable?

( ) Yes

( ) No

( ) Don’t know

In other words, anyone who has not read the report can still answer the question. However, you should be sparing in your use of ‘don’t know’ options where they are not required; they can be misused by lazy respondents who want to finish the survey quickly.

A good question does not look for a desired answer. The actual choice of words is enormously important when designing a question. Even if you want the survey to produce a certain result, it is important for the survey instrument itself to be completely neutral and objective. Be very careful, therefore, not to lead the respondent on to reply in the way you wish them to reply. These are called ‘leading questions’. They are often easy to spot as they often include negatives:

Do you not think that the municipality should spend more on the care of the elderly?

Free text answer

A good question does not use value-loaded words.

Or vague words either, for that matter. This is a common mistake, even among experienced survey designers. Words such as ‘biggest’, ‘worst’, ‘most common’ and ‘best’ are common among survey designers. Remember that this is often a matter of interpretation, and the respondent is perhaps not always inclined to agree.

A good question does not contain any exotic words or abbreviations.

Keep your target group in mind when you design the questions. Do not use jargon or unwieldy, complex sentences. Keep it brief! Wherever possible, avoid abbreviations.

Contrary to what many believe, abbreviations reduce readability. Computer programmers may, perhaps, understand this sentence, but not many other people will:

What sort of API does your invoicing program have?

Free text answer

A good question is not dependent on the previous question.

Branching the questions depending on the replies received works well in verbal interviews but is best avoided in written questionnaires. An example of this is the following:

Do you have private pension savings? If ‘Yes’ go to Question 2.

( ) Yes

( ) No

How much did you spend on private pension savings last year?

Free text answer

As you can see, it is perfectly fine to exclude the first question and ask only question number two. It makes it easier for respondents to reply, as well as it being easier for you when you come to collate the results.

A good question does not ask the respondent to rank more than five items.

Asking people to rank items should be avoided. The point is that, the more items you have to rank, the harder it becomes. The problem becomes even greater when you ask for percentages to be assigned to a number of items. To do this, the respondent has to continually adjust their previous answers to ensure that the total adds up to 100 per cent. Restricting the ranking to five items makes it easier for respondents to reply, and the answers are then more reliable.






Get our smart tips on digital marketing. The newsletter will be published with about 10 numbers a year and of course we will not leave your address to anyone. Much pleasure!

Lämna följande fält tomt