Research vs the world
In a recent blog post about changing a questionnaire between waves of a survey I put it out there that genuine consistency was actually less often attained than you might imagine, and that ensuring you are using a high quality useful questionnaire is more important. I also said that in an ideal world a survey would be consistent in all sorts of ways, and that one of these ways was the avoidance of externally influencing factors. Problem is, this is a tough one to manage.
By externally influencing factors I am talking about the conditions in society that could have an impact on how people view the world and respond to questions in a survey. And specifically in the case of surveys that run over time, I am talking about situations where the conditions in society are different each time you ask your questions and thus introduce inconsistency between waves (and screw with your results). Yes, it is a massive and woolly issue. So I’ll give you a couple of practical examples.
- Imagine you wanted to run a survey about attitudes towards gun ownership. If you asked a set of questions in November 2012 it is likely that you would get a difference response to if you asked the same set of questions on 15th December 2012 – the day after 20 children and 6 adults were killed by a gunman at a primary school in Connecticut.
- Or imagine you wanted to run a survey about celebrities doing charity work. If you asked a set of questions in early 2012 it is likely that you would get a difference response to if you asked the same set of questions in early 2013 – following allegations of the abuse of more than 400 individuals, including children as young as 8, by a TV presenter who has raised an estimated £40 million for charities – Jimmy Savile, OBE.
Occurrences such as these are likely to cause a spike in your tracking data as public opinion changes in response to the external event. Most times, this change in attitude is transient. It generally takes a long time to change attitudes permanently so running a survey in these circumstances can cause anomalous results.
But simply being aware of external factors is not enough. Also throw in the fact that any external influence could impact on your findings by making your trends go up, down or stay the same (and it does not always go the way that you expect) – and that you can’t know how they would have been without the influence happening. Oh, and in some cases you might not even know that there has been an influence at all. Yes, it’s complicated. And it usually only comes up if there is a very very obvious linked public event (much like how they hold off film releases if the content would be considered tactless in the circumstances) or if you notice a spike in your data and you don’t know what has caused it.
It is also important to be aware that these nebulous external factors could creep in and artificially inflate or deflate the findings of a single wave research project too. Or even within a single wave. When I used to manage an omnibus survey we kept the fieldwork period for a thousand responses down to three days to minimise the chance for anything occurring in the news during the fieldwork period meaning that the later responses would come out different to the earlier responses. Yes, we took it that seriously.
So if you are running a research project and you want to keep on top of this you need to look out for externally influencing factors, and if you spot anything going on in the world that might influence your survey you need to think about what, if anything, you are going to do about that.
If your survey is aligned but not closely linked with the external event you could consider holding off your fieldwork for a bit until things quieten down. If the influence is likely to have caused permanent attitude change, or it is the attitude that you are tracking, well, you will need to keep this in mind when comparing your findings over time as a potential explanation for any variation in your data. As I have said before, the most important thing is to understand and acknowledge any limitations of a survey that you run and make your conclusions accordingly.