How can you be sure I’m not making up data?
I was once teaching a class on research methods, and a student asked me how they could be sure I wasn’t just making up the findings. People are cynical about research at the best of times so I suppose this question shouldn’t have surprised me. But it did.
I have never made up research findings. I hope I don’t need to say that.
My job is to be a research expert – to plan, conduct and report back on research and to use this information to help organisations to make evidence-based decisions. I’ve been doing this for almost twelve years now for a variety of organisations, and it has always been quite straightforward. There’s a process, and systems, and the industry is set up to collect data ethically and demonstrate that you’ve done so.
It has never even crossed my mind to make up data.
I’m an accredited member of the Market Research Society and if I was found to be fiddling the data I’d fully expect to be slung off the job, reported to my professional body, and have my accreditation taken off me. Damn sure I’d have no hesitation in sacking the ass of anyone I’d subcontracted that compromises the integrity of my research, and getting them hauled over the coals too. This is not a gentleman’s agreement. It is important that it is done properly.
Actually, as a research consultant I have absolutely nothing to gain in making up results. Putting aside the professional shame if caught, I’m afraid I simply don’t care what the results are to research I’m working on. In the nicest, most professional way, of course. I’m independent. I don’t work for my clients’ organisations and have no vested interest in the findings, either way. It doesn’t actually matter to me whether things come back positive or negative. That’s kind of the whole point of what I do.
Ah ha, you might say, but it is easier to make something up than do it properly. Actually it would be really quite hard and time consuming to make up a bunch of data that looks plausible. Faking questionnaires or transcripts in such a way that they aggregate to something interesting and usable? Making up quotes or stats? Formulating recommendations based on nothing? That would take a lot of creativity and planning. Who has the time and inclination to do that?
There are things that I do as standard practice that ensure that any data that I collect or use can be audited, and traced back to its legitimate original source. Basically, in all circumstances I know more about the range of individuals that provided the data I use than I tell my clients. It would be inethical (in most cases) to tell my clients who responded, but I usually know because I have gone out of my way to ensure that each source of data comes from an individual. So, if necessary, at no point would I have any problem demonstrating that all data I have collected came from separate individuals (although of course I’d have to trade off any requests to do this against the need to preserve anonymity).
- I visually appraise a pile of questionnaires, and am vigilant of looking out for the same handwriting or patterns in completion. If anything looked dodgy I would spot it, consider it, and potentially discard it. Questionnaires are kept in a secure archive box, and could be audited in this way at any future point if required.
- When I do data entry I number every questionnaire and put that number in the data entry form along with the data. This means that any item in the data entry form can be traced back to the original source.
- When conducting a web survey I use a function on my software that allows me to feed in email addresses and track which ones respond. Alternatively I could collect email addresses through the survey or track IP addresses. Using any of these methods helps to prove that returned questionnaires have been filled in by separate individuals, and this information is archived in a secure electronic file.
- If I was sending out a postal survey, I would put a separate code on each return envelope, which helps to prove that returned questionnaires have been filled in by separate (probably named) individuals. These envelopes would also be archived for future reference and the codes would be recorded and stored in a secure electronic file.
- I audio record any interviews or focus groups that I conduct, label them, and store them in a secure electronic file. These could be listened to for content, and to appraise whether the voices come from separate individuals.
- Any audio recordings are also made into a transcription or partial transcription which is numbered and kept on file, and in my draft reports all quotes used reference this number to link them to a transcript. Quotes in reports can therefore be traced back to the original source and context, both in transcript or audio format.
- When sub-contracting large-scale surveys I ensure that the agency does a back-check – where 10% of respondents are phoned back to check the validity of respondents and responses.
I have no idea if people who do what I do make up data. I’ve always assumed not. I hope not. I certainly don’t.
As a consumer of research services I would urge clients to expect the highest ethical standards, and if in any doubt ask the consultant about their systems and processes for ensuring high quality data is collected and used.