Wednesday, October 8, 2008

83% of people agree


This is a qualitative make-over post. 

I’m going to do a 'before and after' on a sentence I pulled from a qualitative research report I was asked to read recently.

Here’s the before sentence: Of the 4 groups, 83% of people said they liked the design.

Now some of you reading will, at this point, know exactly what I’m talking about: You can go and play. For those who don’t please keep reading.

There are many things wrong with the above sentence being part of a qualitative research report. Here are 3 points to start:

1. It’s not robust

Percentages, in a qualitative context, are pretty much meaningless. While at first blush, a grand 83% looks pretty good, what does it really mean? It means that, assuming 4 groups of 8 participants, 26.6 research participants, screened to fit a particular profile, and willing to attend a particular research group, said they liked the design. That’s a very small, skewed sample: hardly robust and hardly worth reporting. But that’s only the beginning…

2. It’s not controlled

To get a good, clean read on any particular issue in a quantitative survey, the way the questions are ordered and the way they are asked is key. For all intents and purposes, and as much as possible, the survey should be administered in a controlled environment. Even rotating the order of questions is controlled. This ensures reliability (being able to replicate the findings) and therefore, some confidence in the results.

In contrast, to get a good read in a qualitative study (we don’t necessarily go for clean in qual), we need to dance around a bit. Cover the floor. A good qualitative facilitator will bounce around, jump ahead, reverse, turn corners, step to the side…you might even see a grand jeté.

The point here is that the context within which the question is asked, ie the discussion group, will vary wildly for each group. In effect, it will be confounded by all sorts of, well, confounding variables; not least, the discussion itself. The fact that 83% of people said they liked the design means absolutely zip without understanding the discussion that came before.

3. It's open to misinterpretation

The third and most worrying point is the potential for misinterpretation. The most obvious here is making the assumption that 83% of people, per se, liked the design. And then using this ‘finding’ to make Big, Important and Expensive decisions, like changing the design.

Here’s the after sentence: Positive feedback for the design was based on factors X, Y and Z.

Note the glaring (and appropriate) lack of percentages?

16 comments:

Penni said...

Plus it's just really bad grammar.

Meanwhile my word verification is almost knickers.

Katie Harris said...

LOL!

Good to have you back from Paris. Next trip, Sydney, right?

Penni said...

I hope so.
x

Kristen said...

This reminded me of a case study I once read where people were put in a pitch black room, with a pin point of light. Afterwards they were asked if the dot of light was moving or stationary. If they were immediately separated & questioned, they all said that the light was stationary. When they were allowed to discuss the question in a group environment (with one "mole" confidently declaring that he "knew" the light was moving), they all eventually agreed that the light was moving.
It's hard to control results sometimes.

GUGAW said...

eeek - qual and %'s dont go, surely that's the first rule you learn?? kirsten...those research pieces on group conformity are really interesting (some more here http://www.age-of-the-sage.org/psychology/social/asch_conformity.html)....i feel that a good quallie can help to counteract those effects though by encouraging debate and differences of opinion so that group members dont feel so much pressure to conform in their opinions. this can be the case even with really sensitive research matter. what do you think?

kuku fashion australia said...

WOW.. Sweet Fashion! Fantastic topics. Liked it., Thanks for the nice Blog. This is really Fantastic. Enjoy online fashion shopping with us! We are- Kuku fashion Australia, with Kuku, Kuku dress, Kuku dresses, Kuku fashion, Mad Love stockist, Mad Love fashion, Toi et Moi online, Toi et Moi jacket, Toi et Moi online. Thanks for staying with us.

Knowledge Plus Australia said...

Thanks for the Great Post. Knowledge Plus is a Registered Training Organisation. training provider australia, certificate iv in business australia, diploma qualifications australia, elearning australia diploma qualifications australia, training provider australia, elearning australia, management qualifications australia, diploma qualifications australia, certificate iv in business australia and many more Training and Supports.

office 365 said...

Thanks for the Great Post. Groupe Conseil Solinteg can support -
office 365
cloud services
sharepoint online
exchange online
lync online
cloud services montreal
microsoft crm montreal
windows intune
and many more services and product from Microsoft Corporation. Thanks.

karmaloop code said...

matryoshka
russian gift
nesting doll
russian dolls
russian hat
l

green products said...

Thanks for the Great Post. We want a Green earth for LIVE, why we need- green home, green products, green product, buy green, eco friendly products, environmental products, eco green products, environmentally friendly products. Thank you for staying with us.

mou liton said...

We are the contractor since 1980 for alternations, renovations and extensions. On our website you find more information and you can request a quotation. aannemer, aannemersbedrijf, bouwbedrijf, verbouwen, renovatie, kosten uitbouw,kosten verbouwing
Thank for the Post.

oil change va said...

Nice blog. Try to repair engine oil leaks to keep the environment clean. Thanks- oil change and oil change va

stop car engine oil leaks said...

Nice blog. Try to repair car oil leak to keep the environment clean. Thanks- stop car engine oil leaks

how to remove grease said...

Nice blog. Try to know how to remove grease and how to clean oil spills va to keep the environment clean. Thanks- fix engine oil leaks

joana lui said...

It is great to observe somebody put enough craze towards a matter. I'm thankful that I stumbled upon this process. I am lucky I used the time to read on past the 1st paragraph. You have got a whole lot to say, very much to offer. I really hope guys understand this and look into your page.

safety deposit box
bank safety deposit boxes
mailbox post
residential mailboxes
gold bar for sale

best wedding coordinator washington state said...

Nice Article topics. This is really excellent and fantastic. I hope it’ll be helpful for our readers. We are - Muse Event Planning and Production, LLC is a full service event planning and production company that can accommodate many types of events. From intimate dinner parties to events/weddings for over three hundred, we can handle every last detail. We work as your seattle wedding planner . We have an extensive list of award winning vendors and venues to build a team for our client’s events that will make any event picture perfect and stress free. We are well known as tacoma wedding planner or tacoma event planner . Our goal is to ensure that our client’s event is a reflection of them and their vision. Using creativity, attention to detail and experience in management and bookkeeping, we vow to create a one-of-a-kind event within their budget. We were rewarded as number one wedding planner seattle last year. We demand, we are the best wedding planner washington state . Thanks for your cooperation.