Skip to content

Holistic questionnaire design

July 11, 2017

I’ve whinged on before that every man and his dog thinks they can write a good questionnaire, and you’ll not be surprised to hear that in my opinion that’s not true.  I’ve been reflecting on this a bit, and was wondering what would happen if you stuck me and a non-professional researcher in adjoining rooms and asked us both to come up with a questionnaire on the same topic.  How would our final products differ?

I think if you asked me to write a survey about cat ownership, it would come out quite differently than a non-professional researcher’s survey about cat ownership.

To be fair I think most people do know a bit about questionnaire design and are able to have a go, but it is my view that the quality of the output would vary depending on the level of understanding and experience of quantitative research theory.

So here’s my observations on how that tends to pan out, based on the non-professional surveys that I see.

Survey Design 101: Write good questions 

I make sure that I write useful, concise and non-leading single-issue questions so that if I ask something I am getting a useful and unambiguous answer.  Most people know this, although they don’t always achieve it in practice.

I might ask: What breed of cat, if any, do you currently own?

 

A non-professional researcher might ask: What cat do you have?

Survey Design 102: Write full questions

When I am writing a question, I don’t just think about the way that I will word the question itself but I also think about what type of question it will be: how I will lay it out, whether it will be ‘tick all that apply’ or not, what the response options will be.  This is just common sense.  I genuinely have no idea why so many people draft up a questionnaire and don’t include response options, but I see that all the time.

I might ask:

What breed of cat, if any, do you currently own? (tick all that apply)

  • Bengal

  • Maine Coon

  • Shorthair

  • Siamese

  • Tabby

  • Mixed breed

  • Other (please specify)

  • Don’t know

  • I don’t currently own a cat

 

A non-professional researcher might ask:

What cat do you have?

Survey Design Honours: Be considerate to your respondent

I prioritise respondent comfort to minimise drop-outs and maximise response quality.

  • I keep the wording straightforward but not patronising.
  • I keep it as short and tick-boxy as possible.
  • I minimise technical stuff at the point of use.
  • I try not to overwhelm respondents, setting up paper surveys with attractive layouts and lots of white space and online surveys with plenty of page breaks rather than one huge long screed.
  • I order my questions in a way that feels logical to the respondent and utilises a bit of research-on-research theory to warm them up, starting with ‘easy’ questions and building up to more complicated ones.

I might ask:

What breed of cat, if any, do you currently own?

  • Bengal

  • Maine Coon

  • Shorthair

  • Siamese

  • Tabby

  • Mixed breed

  • Other (please specify)

  • Don’t know

  • I don’t currently own a cat

 

[new page, auto skip logic IF ‘I don’t currently own a cat’]

Why do you choose not to own a cat?

………………………………………..

 

[new page, auto skip logic IF NOT ‘I don’t currently own a cat’]

Why do you choose to own a cat?

………………………………………..

 

A non-professional researcher might ask:

Q1. What cat do you have?

Q2. If you have a cat, why do you have a cat?

Q3. If you don’t have a cat, why don’t you have a cat?

Survey Design Masters: Strategic survey design

Before I start writing a survey, I think strategically about what I hope to get out of the survey and I design every element of it with this focus in mind.

I might prepare:

  • What I hope to find out (e.g. Patterns in cat ownership across the UK, motivations behind cat ownership, perceptions of cat ownership, differences in response between pedigree and non pedigree cat ownership)
  • What I DON’T need to know (e.g. cat ownership abroad, ownership of other pets)
  • Any intended KPIs / headline statistics (e.g. % of UK adults who are cat owners, % of cat owners with a pedigree cat)

In my experience a non-professional researcher just tends to start with a list of questions (usually a very long list of questions) and then add more questions as they occur to them.

Survey Design PhD: Holistic survey design 

In my experience a non-professional researcher tends to write a questionnaire in isolation and sees it as the end in itself.  They don’t think about what comes next.

When I am writing a survey, I always have the full survey process in mind, so I am thinking about how I will analyse and report on each element of it.  This is influenced by the survey strategy, and in turn influences the way that I word, order and lay out my questions.  In practice this means the way that I word questions and the way that I analyse questions looks very different from the way that a non-professional researcher would do it.

Lets say that I had written a question:

To what extent do you agree or disagree with the following statements about cats?

  1. Dogs are as good as cats
  2. Cats are fun pets
  3. Cats care for their owners
  4. Cats are expensive pets
  5. Everyone should own a cat

(Strongly agree, Slightly agree, Neither agree nor disagree, Slightly disagree, Strongly disagree, Don’t know)

Typically I have seen non-professional researchers struggle to know what to do with the results from a question like that.  They might analyse and present it as such:

graph3Complicated.

Instead, I would have intended this question for comparison and rank ordering purposes, and I would have written it in such a way that I could analyse and present it like this:

graph 1

Also, remember that first question about breeds of cats?  Well I would not have intended that question to be used alone, instead intending it for cross analysis purposes like this:

graph 2

So any time I write a question, I can on demand give a whole host of reasons to justify its wording, layout, response options, order positioning, intended linkages with other questions, future use for data processing and analysis and reporting, and indeed its inclusion in the first place.

But why am I telling you this?

Well, to a certain extent I am reiterating what I’m sure I’ve said before ad nauseum that there’s an art to questionnaire design and it is more complicated than it looks.

But more so I find that time and time again I’ll write some questions and a research-inexperienced client will come back and say that isn’t how they would have done it.  Fair enough!  It is absolutely my job to lead my clients through the questionnaire design process and to justify my recommendations, pitched at an appropriate level.  I love my research-inexperienced clients, and I genuinely gain professional satisfaction from helping them to understand what they need.  That said, when finalising a questionnaire recently and contesting some minor points, a client said to me “I always worry my lay view may impact on your ability to analyse data, so I’m happy to defer to your experience” – and I loved them for that.  I loved them so much.  They had a sophisticated understanding of what they didn’t understand!

But please don’t get me wrong this isn’t about expecting everyone to know what I know, or insulting those that don’t.  Hey, if everyone knew this stuff I’d be out of a job.

I suppose I just wanted to set out some of the rationale behind why I do things as I do things… because I usually have a reason in mind that may not be obvious on first reading and maybe this article will help to explain why that is.

 

 

 

 

 

 

Advertisements
No comments yet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: