Skip to content

Call the research police, she’s changed the questionnaire

February 12, 2013

2009-04-23Something everybody knows about research is that if you’ve done a survey once and you want to run it again (and again, and again) you have to keep the questions the same.  Consistency is vitally important and you cannot make comparisons between findings over time unless the questions are identical.  I know everybody knows this, because my clients tell me it all the time.  It seems it is the golden rule of research and it cannot be broken.

Now those who know me will confirm that I’m substantially more idealistic than the average person, and in an ideal world it is true that you have to keep your questions exactly the same if you want to know for sure how much something has changed between one wave of a survey and the next.  (Keep that in mind while you read this, I am absolutely agreeing that consistency is the best case scenario)  However, in practice, doing this is not as straightforward as you might imagine and like many things whether or not you should change your questionnaire between waves of a survey is a bit of a judgement call.

So first, can you really be sure that the second wave (or fifth wave, or tenth wave) of your survey is actually going to be identical to the first?  The thing is, if you’re going to keep a survey consistent you need to keep it all the same.  This is not simply a case of using the same questions.  Any element of inconsistency, no matter how small, has the potential to impact on the respondent experience and this can affect the way that questions are approached or interpreted.  The survey also needs to be consistent in terms of:

  • Not adding in or taking out any questions
  • Consistency of whether questions were single answer or ‘tick all that apply’
  • Consistency of all lists of response options
  • Consistency of the order of the questions
  • Consistency of the layout of the questions
  • Consistency of the wording of the introduction, close screen and any other non-question text used (or not used)
  • Consistency of any logos or images used (or not used)
  • Consistency of any incentive used to encourage response
  • Consistency of distribution method (i.e. paper, online, telephone)
  • Consistency of sampling method and profile of respondents
  • Avoidance of externally influencing factors

There are some circumstances in which you can be sure that your survey is adequately consistent.  If you used a research consultant or an off the peg methodology such as an omnibus for multiple waves, then you can be pretty sure it is adequately consistent.  If you plan to literally copy the same questionnaire as last time (in paper or online format) it will probably be adequately consistent.

The problem for anyone running a subsequent wave of a survey is, getting on top of all of this consistency isn’t always as straightforward as it looks because people do not always keep full records of these things.  ‘On file’ questionnaires that I am given typically do not include notes on layout.  Half of the time they do not even include lists of response options.  And where they do, you often find that the final version had a surprise ‘don’t know’ or ‘prefer not to say’ option that hadn’t been recorded.  This is because people draft something up, then someone proof reads it and changes it, then someone else typesets it or sets it up online and adds in something new – but no-one takes a copy of the final version.  So when you come to run it again it is surprisingly fiddly and time consuming to work backwards to exactly re-create the original.

So what am I saying?  Well I guess I’m just throwing it out there that if you are honest with yourself you may find that your survey isn’t going to be as consistent as you thought it was anyway.  So maybe making a few adjustments to the question wording isn’t the worst thing in the world.

And honestly, is making a statistically robust comparison of every last one of your questions really all that important?

<sharp intake of breath> 

I know, I know, controversial, but unless a hell of a lot of money is riding on the findings or unless you are setting something up with the sole intention of tracking an important change over time… well maybe it is better to make a few changes so that you are doing something usable rather than something technically perfect.

So this is just a preamble to my main point which is that you don’t want to be stuck with a terrible questionnaire.

Maybe your questionnaire is great, in which case on you go.  But maybe it isn’t.  I’m sure you didn’t intend to have a terrible questionnaire, but sometimes you do something once (or twice) and find that it could be improved, or shortened, or is not as relevant as it once was.  Or sometimes you inherit something truly awful from a colleague.  Should you stick with this questionnaire for reasons of consistency?  Many would say yes, but I’m not so sure.

I think in the big scheme of things it is worse to run a bad questionnaire or an irrelevant questionnaire than it is to sacrifice a bit of comparability in the short term.  If you are bothering to run a survey then you might as well do something that will be useful and usable for your organisation, rather than running the same old same old just because you always have.  Fair enough, don’t make major changes unless you really have to.  Consistency is good, as long as the questionnaire is good.  But if you do make revisions you can be systematic and strategic when creating your new version to go forward with, by including well thought out core questions that you expect to remain the same into the future, and by planning for places to consistently slot in something new each wave if you have a different event or product or service to ask about.

And as long as you are clear in your analysis and reporting that you are comparing like with not-quite-like, and you consider this when making decisions based on the data, this is perfectly acceptable.

There is no question that changing a questionnaire is a judgement call, and dare I say it a brave move.  But what if you don’t change it?  In ten years’ time you will still be running that same terrible questionnaire.  Urgh that makes me shudder.

So this is the crucial bit.  You will not have the research police banging on your door if you change your questionnaire as long as you understand the limitations of what you have done.  And that is good practice, the actual golden rule.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: