Skip to content

Please answer the question in front of you

July 16, 2013

2012-12-22As a sole trader researcher I’m a jack of all trades, and I look after all the different bits of my research projects from strategy to practice to admin.

One thing that I’ve noticed doing this that I hadn’t noticed before is how respondents sometimes refer to previous answers in self-completion surveys.

Say I’d written a web survey and included an open ended question like ‘What motivated you to go the park?’  Some people might respond ‘for the fresh air’ or ‘to feed the ducks’ or ‘to meet friends’ – but others might respond:

  • See Q3
  • As above
  • Refer to previous answers

I can completely understand why respondents do this.  It is quicker for them, and they think they have already told me the answer so they don’t need to tell me again.

(And they probably have told me the answer, because open ended questions usually come at the end of sections where respondents have been ‘primed’ to think broadly about their behaviour and attitudes through lengthy sets of closed questions.  Their ‘preference’ or ‘motivation’ or ‘key influence’ is probably in there somewhere.  But we want to hear a summary in their own words…)

Unfortunately though, not directly answering the question is a huge pain for me as a researcher – both because it takes me extra time to go through and check out what they are trying to say, but also because the act of doing so makes the process a little less ethically pure.

I think the thing here is about the mindset within which we approach the research.  The respondent understandably starts with the questionnaire and looks up, and the researcher starts with the data and looks down.

The respondent fills in a questionnaire, and perhaps if pressed might visualise what comes next as a process akin to a researcher in an office reading through a big pile of questionnaires.

In actual fact it doesn’t work like that.

For the researcher, the process starts with looking at all of the data in its entirety – a big blob of collated numbers from the many hundreds of people who responded to the survey.

This is because quantitative research (surveys, questionnaires, numbers, stats) is all about looking for patterns in data.  We are interested in:

  • The distribution of responses to a particular question (i.e. What percentage like and dislike kittens)
  • The characteristics of those that gave particular responses (i.e. are men or women more likely to dislike kittens?)
  • How responses to one question are linked to responses to another question (i.e. do people who dislike kittens also dislike puppies?)

Although we care about and put substantial effort into allowing people to have the opportunity to say what they want to say, in analysis terms we are not interested in what individual people had to say.  Sorry, it sounds mean when someone has bothered to fill the questionnaire in and it isn’t supposed to be mean.  But it is key.  We don’t look at individual questionnaires, both because we don’t need to and because it ensures that we don’t have access to any inethical opportunities to identify individuals from their responses.  Instead we process the data, and look at a summary.  In case you’re curious, here’s the sort of thing we do look at.

So if a respondent is self referential in a survey, I’m forced to dig out their individual questionnaire and read through it to work out what they are on about.  I would then ‘clean’ the data – either by cutting and pasting an earlier answer into a later answer, or inserting a note about the general theme, or more likely just deleting the response if the meaning is in any way ambiguous.

So if a respondent doesn’t directly answer the question, I’m doing something with their questionnaire that I’m not doing with everyone elses, I’m learning more about them as an individual than I am about other respondents, and I might even be discarding some of their responses that they might actually care about.

I’m pretty sure that if they thought about it most respondents wouldn’t want me trawling through their questionnaires and making these sort of decisions on their behalf if I didn’t have to, so I’d like to urge you – as a future respondent – to answer the question in front of you, even if it does feel a bit repetitive.  It’s good for you, and it’s good for me.

Advertisements
3 Comments leave one →
  1. July 16, 2013 10:29 am

    Great post! I think it really helps to demonstrate the power of being able to look at data in aggregate as well as clarifying how you look at individual questionnaires.

    I wonder if, in your experience, there are ways of encouraging the respondent to “answer the question in front of you”. For example are there ways you can pose an open question such as starting with “in your own words” or “Please summarise in one sentence” that might help them to realise referring to earlier answers isn’t helpful?

    I also wonder if there are structural approaches – I remember doing this myself once (sorry!) in a survey that asked “why do you like product X” then later “why do you buy product X over its competitors” (or something like that). I had answered both questions in the first box, because I didn’t know the second one was coming! Could the researcher have structured that to avoid me jumping ahead in my mind?

Trackbacks

  1. Reasons to position your demographic section at the end of a survey | ruthlessresearch
  2. How to clean your own Surveymonkey data | ruthlessresearch

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: