Soup, sampling and quotas
I’ve had two ‘market research’ callers recently whose first question is to ask me what age group I am part of and, when I tell them, they say ‘Oh, we already have enough people from that age group, we don’t need to talk to you now’. What are they up to?
The answer to this is ‘quotas’ but in order to talk about quotas first of all we need to talk about sampling.
Research is all about asking the right people the right questions, and the process of selecting the right people to participate in a research project is called sampling. The idea behind sampling is that you don’t need to talk to a whole population to find out what that whole population thinks. Which is handy, because speaking to ‘all women’ or ‘all coca cola customers’ (etc etc) would be very expensive / time consuming / impractical.
Someone wise once explained sampling to me using a soup-based analogy. If you get a bowl of soup, add salt, and stir it up, you do not need to drink the whole bowl of soup to know whether it tastes right. You can tell whether it is correctly seasoned by tasting a spoonful because all of the spoonfuls taste the same.
So, I’m saying that if you talk to a subset of a population you will more than likely get the same answers as if you asked everyone, as long as you are sure that the characteristics of the subset match the characteristics of the whole population.
When a researcher sets up a research project, they think about the population that they need to talk to and they find out a bit about the kind of people that make it up. In practice they might think about tens of different characteristics (gender, age, socio economic group, location, purchasing behaviour…) but I will give you a straightforward example.
If a survey was being carried out with members of a sports centre, maybe the researcher will find out that the membership totals 10,000 people of which 60% are male and 40% are female and of which 25% are aged under 30 and 75% are aged over 30.
Say the researcher decides to conduct interviews with 200 sports centre members. They will want to try to make that sample of 200 members as similar as possible to the original 10,000. So they will say I want 60% of the interviews to be with men, 40% with women, 25% aged under 30 75% aged over 30. They will work out how many individual interviews need to fall into each category:
- TOTAL INTERVIEWS: 200
- Male: 120 interviews
- Female: 80 interviews
- Aged under 30: 50 interviews
- Aged over 30: 150 interviews
These are the quotas. What this means is that as the interviews are being conducted (and that could be in person, by phone or online – or indeed for recruitment for focus groups) the characteristics of the respondents are recorded against these targets in real time. As the quotas ‘fill up’ the researcher will be keeping an eye on things and once one is full (i.e. if 50 interviews with under 30s have been completed) then it will be closed (i.e. no more under 30s will be interviewed). This keeps going until 200 interviews are completed and all of the quotas have been filled.
However, in practice, it can sometimes happen that you will be invited to participate in a research project but when they find out a bit about you it will turn out that you are ‘out of quota’ – that you do not fulfil the characteristics that they are looking for to ensure their sample is accurate. In the case of this example, perhaps they have completed 195 interviews and they are just looking to interview five women aged over 30. So they keep ringing round the list of members until they find some!
As an aside, there will be other cases where the survey will be fairly open but there will be a screener question to make sure that only certain types of person can fill it in. You will notice this when you go to fill in a survey and get knocked back at the first question, which happens because the researcher or client only has an interest in collecting responses from people with a particular characteristic and you do not fit that characteristic.
I recently conducted a web survey with the screener question:
Q1 Approximately how many dives have you done in the last twelve months?
1) 0 [CLOSE]
Close screen: At this time we are looking for active scuba divers to complete the survey, but thank you for your interest.
A few people were disgruntled that they could not participate because they had not dived in the last 12 months, but the questions were only relevant to recent divers and it was important to the legitimacy of my findings that only recent divers completed it.
Unfortunately when quotas and screener questions are used some people will end up being willing but ineligible to participate, and I sincerely hope that those people would be properly thanked for their time and their interest.