Screener questions: Keeping out the riffraff

Of late I have noticed that many inexperienced designers of questionnaires make a fundamental error.  They write a lovely interesting unbiased survey and then they let any old riffraff answer it.

It would be unusual to write a questionnaire that is genuinely open to anyone to respond (i.e. any member of the public) as usually researchers have a particular market or segment of the population whose views they are interested in.  So how is it possible to ensure that all responses received come from this market or segment?

Even when a researcher does everything they can to control this (detailed explanation in the intro, emailing directly to desired respondents) respondents sometimes get it wrong.  They skim read the intro and miss the carefully prepared requirements, they forward the invitation email to their friends, or they simply don’t care and get on and answer the survey anyway.

Screener questions offer a decent solution to this problem.  A screener question is a question that asks ‘Are you a relevant person to answer this survey’.  If yes, the respondent goes on and answers the survey.  If no, the respondent is ‘screened out’ by being thanked for their interest and informed that on this occasion their input is not required.  The screener is like a bouncer at a nightclub, checking ID and only letting those who are over 18 and are not wearing trainers inside.

Now I apologise sincerely if this sounds like I’m teaching my granny to suck eggs, but here are some examples of screener questions.  Yes they seem obvious, but they are the sort of questions that are routinely forgotten in questionnaire design.

Example screener questions:

• (For a survey of Scottish people) Do you live in Scotland?
• (For a survey of people with children) Do you personally have children aged under 16?
• (For customers) Have you purchased our product?
• (For members) Are you a member of our organisation?

Sticking a screener question or two in at the start of a survey ensures that a survey is answered by people for whom it is relevant which in turn ensures that findings are actually meaningful.  These should be the first questions in a survey, to make sure that relevant responses are obtained, and that the time of irrelevant respondents is not wasted.

Case study from my life:

I recently saw a request for survey respondents in a mainstream third sector e-bulletin, with a subject matter of attitudes towards trans-phobic hate crime.  I answered a series of questions about my personal experience of trans-phobic hate crime and how it made me feel, saying that I had never been the victim of trans-phobic hate crime.  I completed the survey expecting it to become relevant to me at some point. At no point did the survey ask me whether I was a trans-person or not.  I’m not. I genuinely wanted to help by answering the survey but inadvertently made their findings inaccurate.  My responses presumably artificially inflated the proportion of responses in the ‘no hate crime’ category and reduced the proportion in the ‘yes hate crime’ category, as a consequence potentially under reporting the actual amount of trans-phobic hate crime occuring.    

However be warned, sometimes screener questions can narrow the population of respondents too far and prevent a researcher from collecting potentially useful data.  Now I’m the first one to be pragmatic about questionnaire design, and to say there is no need to feel bad for screening people out of a survey.  But if a researcher approaches the process with a very narrow view (usually because they think they know what the answers will be) then they can end up excluding people who could give relevant responses.

Case study from my life:

I recently received a questionnaire from my residents association, asking if I supported their bid to change local school catchment areas.  The questionnaire asked if I had school aged children, and as I did not I was not required to answer any questions.  However, had they asked they may have found it helpful to know that I had opinions on the subject that support their bid relating to future children, house prices, and information provided to me when I bought my property. By narrowing their population of respondents they underestimated support for their cause.

Top tips on screener questions when writing a questionnaire:

• Spend time thinking about the issues in question, and the type of people who might be in a good position to comment on them.
• Test out your questionnaire by going through it and pretending to be different people – if you were a different gender / age / ethnicity or if you were not a customer / local would it still make sense?
• Ask a couple of friends or colleagues to go through the questionnaire and check it for relevance – sometimes an outside eye spots something you did not.
• Add in screener questions at the start of the questionnaire to ensure that the survey is answered by those for whom it is relevant.