Please answer the question in front of you
One thing that I’ve noticed doing this that I hadn’t noticed before is how respondents sometimes refer to previous answers in self-completion surveys.
Say I’d written a web survey and included an open ended question like ‘What motivated you to go the park?’ Some people might respond ‘for the fresh air’ or ‘to feed the ducks’ or ‘to meet friends’ – but others might respond:
- See Q3
- As above
- Refer to previous answers
I can completely understand why respondents do this. It is quicker for them, and they think they have already told me the answer so they don’t need to tell me again.
(And they probably have told me the answer, because open ended questions usually come at the end of sections where respondents have been ‘primed’ to think broadly about their behaviour and attitudes through lengthy sets of closed questions. Their ‘preference’ or ‘motivation’ or ‘key influence’ is probably in there somewhere. But we want to hear a summary in their own words…)
Unfortunately though, not directly answering the question is a huge pain for me as a researcher – both because it takes me extra time to go through and check out what they are trying to say, but also because the act of doing so makes the process a little less ethically pure.
I think the thing here is about the mindset within which we approach the research. The respondent understandably starts with the questionnaire and looks up, and the researcher starts with the data and looks down.
The respondent fills in a questionnaire, and perhaps if pressed might visualise what comes next as a process akin to a researcher in an office reading through a big pile of questionnaires.
In actual fact it doesn’t work like that.
For the researcher, the process starts with looking at all of the data in its entirety – a big blob of collated numbers from the many hundreds of people who responded to the survey.
This is because quantitative research (surveys, questionnaires, numbers, stats) is all about looking for patterns in data. We are interested in:
- The distribution of responses to a particular question (i.e. What percentage like and dislike kittens)
- The characteristics of those that gave particular responses (i.e. are men or women more likely to dislike kittens?)
- How responses to one question are linked to responses to another question (i.e. do people who dislike kittens also dislike puppies?)
Although we care about and put substantial effort into allowing people to have the opportunity to say what they want to say, in analysis terms we are not interested in what individual people had to say. Sorry, it sounds mean when someone has bothered to fill the questionnaire in and it isn’t supposed to be mean. But it is key. We don’t look at individual questionnaires, both because we don’t need to and because it ensures that we don’t have access to any inethical opportunities to identify individuals from their responses. Instead we process the data, and look at a summary. In case you’re curious, here’s the sort of thing we do look at.
So if a respondent is self referential in a survey, I’m forced to dig out their individual questionnaire and read through it to work out what they are on about. I would then ‘clean’ the data – either by cutting and pasting an earlier answer into a later answer, or inserting a note about the general theme, or more likely just deleting the response if the meaning is in any way ambiguous.
So if a respondent doesn’t directly answer the question, I’m doing something with their questionnaire that I’m not doing with everyone elses, I’m learning more about them as an individual than I am about other respondents, and I might even be discarding some of their responses that they might actually care about.
I’m pretty sure that if they thought about it most respondents wouldn’t want me trawling through their questionnaires and making these sort of decisions on their behalf if I didn’t have to, so I’d like to urge you – as a future respondent – to answer the question in front of you, even if it does feel a bit repetitive. It’s good for you, and it’s good for me.