This week I’ve been doing cognitive interviews…

A fancy term for finding out what people are really thinking when they are answering quantitative questionnaires.

The idea is to improve the quality of questions by understanding how people react and respond to them – to help alleviate ambiguity, to help with analysis of results, to help make the questionnaire people-friendly.

The process was incredibly useful. In our cognitive interviews we learnt…

• people want to give you answers they feel are typical of their behaviour rather than being their actual behaviour. We were asking people about food they’d eaten yesterday, and all of our interviewees had the urge to amend and adjust their answers because yesterday was an unusual day. (And everyone we interviewed had an unusual yesterday!)

• respondents try to give helpful answers – and they do this by second guessing what the intention of the questionnaire is – not always rightly. Signposting the purpose of the research can help…

• similarly people hate to be boxed into one group. We asked people which statement BEST reflected them, and even though they understood the question, they couldn’t let go of the fact that in the past they’d been different, or that sometimes they didn’t fit the best fit option. Being asked to choose one option/ the best made people genuinely uncomfortable, and ultimately they wanted to present the ‘best’ version of themselves – their idealised self.

• people hated reading lots of words and dense text when filling in the questionnaire. (No sh*t Sherlock!). And yet, seeing ‘system one’ reactions in action really forced us to re-evaluate how to put together a question. There’s a role for copy-writers in questionnaire design, as well as graphic designers…You can’t be too concise.

What did we do with our insights?

We simplified language, we corrected ambiguities, and most importantly we added in copy that recognised how people wanted to fill in the questionnaire – we know yesterday might not have been typical, but don’t worry – we genuinely want to know what you did then…
__
And one more conclusion. It’s likely to be much more reliable to get people reporting on their very recent behaviour than asking them ‘on average’ because of the tendency to halo average behaviour – AND – people need to be nudged into knowing that this is the most helpful way they can fill in the questionnaire.

Cognitive interviewing felt like a useful investment in the drafting process of the quantitative questionnaire. Why not give it a go?