But is it robust? Five ways to respond when stakeholders question the value of qual
Is it robust? What does it represent? Why have you interviewed so few people? Most researchers and insights professionals instinctively understand the value of qualitative research. But often, end stakeholders in other roles such as finance, marketing or communications don’t have the same experience, meaning they have concerns and reservations we need to address. We’ve listed five questions we are often asked, together with our responses, to help your stakeholders make space for the qualitative research that complex projects often need.
1. Is it robust? Statistics, rightly or wrongly, have an air of certainty, large quantitative samples signify rigour and ‘nat rep’ is (theoretically) representative of the views of the nation; qualitative research employs much smaller samples and makes no such claims about representation, so it can make stakeholders feel nervous about what they are buying.
The best way to address this is head on. Qualitative research is not about quantifying the views of the target audience or proving a hypothesis but about exploring the topic under investigation, understanding how people think, feel and behave and explaining why they hold the views that they do. We believe that the more complex the topic, the more important it is to conduct qualitative research as quantitative research is, necessarily, reductive.
Some of our clients have told us that their stakeholders will only listen to hard numbers. We don’t dispute the value of quantitative research but there is so much more to be gained from qual – the language that people use around an issue, the context in which they interact with brands and services, the truth that they can’t vocalise but that their body language reveals.
We can’t generalise the findings of a qualitative study to a wider population because that isn’t what we are trying to do – and it shouldn’t matter. For example, if we are evaluating public service messaging and the first participant tells us that some of the wording has another meaning that is offensive in her culture, we don’t need to conduct further research to quantify how many other people will be aware of the issue – we know we need to change the copy.
2. Are four groups enough? This is the other side of the same coin. Sometimes stakeholders are bought-in to qualitative research but still approach it with a quantitative mindset, which makes them nervous about whether four groups of six to eight participants is going to be enough.
Again, this suggests a misunderstanding about what qual can do. If we’ve recruited the right people – and careful recruitment is the key to all good qual - we don’t need to keep adding more participants to ensure we haven’t missed anything. There is evidence that the majority of themes in any project can be uncovered in just a few groups or depths. Research by Emily Namey, an expert in qualitative methods suggests that two to three focus groups per sub-population of interest will likely capture about 80% of themes on a topic and three to six focus groups are likely enough to identify 90% of important themes. Similarly, most themes are identified in the first ten depth interviews. She even wrote a catchy poem to sum up her findings, which we were rather taken by and plan to recite next time someone asks us this question.
Sampling to reach saturation?
Here’s the magical equation:
For interviews, to do them well,
choose a sample from 6-12*;
If focus groups are in the mix,
aim to conduct 3-6*.
(Okay, equation it is not
But empirical guidance helps a lot!)
*per sub-population of interest
3. But none of the participants said that? When presenting the findings and key themes to stakeholders who have viewed qualitative fieldwork, we sometimes get asked how we have got to the findings because, “I didn’t hear any of the participants say that.” Qualitative research – and subsequent analysis - gets you beyond the words that people say in focus groups or interviews and into the passion with which they said those words, the things they didn’t say and their body language when then said them. It finds prevailing themes and digs for the implicit as well as the explicit meaning of what is said. Our analysis looks at the research as a whole and brings themes together beyond what is said in answer to individual questions.
For example, in healthcare research or when talking about sensitive subjects, people often find it hard to recognise and acknowledge difficult feelings such as shame or humiliation and may not talk about them directly – but these themes can lurk beneath the words that are actually said and be critical in understanding the issues.
4. What percentage of respondents said that? Again, this is the other side of the coin to the previous question. When stakeholders want to feel confident about the findings they revert to ‘safety in numbers’ and only want to consider the views expressed by the majority. However sometimes, just one participant will express something novel or different which we consider to be important and valid. It may be that this person is an outlier in the group, but it may also be that there are many other outliers just like her in the wider population. Part of the skill of the qualitative researcher is to look at who this person is, the strength and emotion with which they have expressed their view and the context within which they have spoken. What sort of a person are they, what is leading them to have this view and why are they so different from the others? Sometimes, the views of an outlier can take the research in an unexpected but valuable direction. Which brings us to the next question.
5. Why did you change the script? One of the great things about quantitative surveys is that they are consistent – once in the field it is very hard to change them, and it creates all sorts of problems if you try. But qualitative research is fluid. Moderators are free to follow the topic guide or abandon it and be led by participants to where the issues really are. This can make stakeholders nervous and lead to rigidity around the design of the discussion guide and discomfort in the viewing room (online or IRL) if the moderator veers from the script. But because of the exploratory nature of qual, if there is permission to pivot, it enables us all to get the best from the work.
For example, we conducted some qualitative research to evaluate and prioritise amongst some early-stage advertising concepts for a social issue. However, the participants wanted to talk about their lived experience of the issue, so we abandoned the script, and the attempt to choose the best concept, and let them do so. We discovered that the advertising creative was based on a preconception that didn’t resonate at all with participants. Our recommendations led our client to abandon the existing concepts and create a much more powerful campaign.
Overall, qualitative research is about lived experience and that is powerful. The impact that the recent TV drama Mr Bates vs. the Post Office has had in bringing to life an issue that was already well documented in the press shows how human stories move us in a way that numbers never can.
We’re happy to share our experiences of helping our clients and their stakeholders unlock the power of qualitative research. For more information you can contact us at josie@solutions-research.co.uk or michelle@solutions-research.co.uk