This is a qualitative make-over post.
I’m going to do a 'before and after' on a sentence I pulled from a qualitative research report I was asked to read recently.
Here’s the before sentence: Of the 4 groups, 83% of people said they liked the design.
Now some of you reading will, at this point, know exactly what I’m talking about: You can go and play. For those who don’t please keep reading.
There are many things wrong with the above sentence being part of a qualitative research report. Here are 3 points to start:
1. It’s not robust
Percentages, in a qualitative context, are pretty much meaningless. While at first blush, a grand 83% looks pretty good, what does it really mean? It means that, assuming 4 groups of 8 participants, 26.6 research participants, screened to fit a particular profile, and willing to attend a particular research group, said they liked the design. That’s a very small, skewed sample: hardly robust and hardly worth reporting. But that’s only the beginning…
2. It’s not controlled
To get a good, clean read on any particular issue in a quantitative survey, the way the questions are ordered and the way they are asked is key. For all intents and purposes, and as much as possible, the survey should be administered in a controlled environment. Even rotating the order of questions is controlled. This ensures reliability (being able to replicate the findings) and therefore, some confidence in the results.
In contrast, to get a good read in a qualitative study (we don’t necessarily go for clean in qual), we need to dance around a bit. Cover the floor. A good qualitative facilitator will bounce around, jump ahead, reverse, turn corners, step to the side…you might even see a grand jeté.
The point here is that the context within which the question is asked, ie the discussion group, will vary wildly for each group. In effect, it will be confounded by all sorts of, well, confounding variables; not least, the discussion itself. The fact that 83% of people said they liked the design means absolutely zip without understanding the discussion that came before.
3. It's open to misinterpretation
The third and most worrying point is the potential for misinterpretation. The most obvious here is making the assumption that 83% of people, per se, liked the design. And then using this ‘finding’ to make Big, Important and Expensive decisions, like changing the design.
Here’s the after sentence: Positive feedback for the design was based on factors X, Y and Z.
Note the glaring (and appropriate) lack of percentages?