How Many Interviews Does It Take to Assess A Project?

Posted by nnlmneo on October 21st, 2016 Posted in: Blog

---

FAQ from NEO users: How many interviews or focus groups do we need for our qualitative assessment project?

Our typical response: Um, how much money and time do you have?

At which point, our users probably want to throw a stapler at us. (Karen and I work remotely out of an abundance of caution.)

Although all NEO users are, in fact, quite well-mannered, I was happy to discover a study that allows us to provide a better response to that question.  A study published by Namey et al, which appears in the American Journal of Evaluation’s September issue, provides empirically based estimates of the number of one-to-one or group interviews needed for qualitative interviewing projects.  More specifically, their study compared the cost effectiveness of individual and focus group interviews. The researchers conducted an impressive 40 interviews and 40 focus groups (with an average of eight people per group). They then used a boot-strap sampling methodology, which essentially allowed them to do 10,000 mini-studies on their research questions.

They first looked at how many individual and focus group interviews it took to reach what qualitative researchers call thematic saturation. In lay terms, saturation means “Not really hearing anything new here.”  Operationally, it occurs when 80-90% of the information provided in an interview has already been covered in the previous interviews.

The researchers found that 80% saturation occurred after 2-4 focus groups or eight individual interviews. It took 16 interviews and five focus groups to reach 90% saturation. Note their estimates apply to studies that focus on one specific population.  If you want to explore the experiences of two groups, such as doctors and nurse practitioners, you would hold eight interviews per group to reach 80% thematic saturation.

For comparative cost assessment, the researchers used a formula that combined hourly rate of the data collector’s time, incentive costs per participant, and cost of transcription for recordings. They chose not to include cost for factors that vary widely, such as space rental or refreshments. Using more predictable costs made for cleaner and more generalizable comparisons.

Bottom line, they found individual interview methods cost 12-20% less than focus group methods.

Of course, many of us operate on shoestring budgets, so we are our own moderators and transcribers.  Even though most of us DIYers collect hourly wages, the cost for outsourcing these tasks is probably higher than for conducting them internally. Knowing this, the researchers looked at variations on moderator, transcriptionist, and incentive costs.  They also compared cost effectiveness of the two methods when lowering the standards for thematic saturation (i.e., aiming for 70% saturation instead of 80%). Across the board, individual interviews were more cost-effective than focus groups.

Cost is not always the only consideration when choosing between focus groups and individual interviews. Some assessment questions beg for group brainstorming, while others demand the privacy of one-to-one discussions. However, for many assessment studies, either method is equally viable.  In that case, cost and convenience will drive your decision. Personally, I often find individual interviews to be more convenient than focus groups, both for the participants and for me. It’s nice to know that the cost justifies using the more convenient approach.

The full article provides details on their methods, so it is a nice primer on qualitative analysis of interview transcripts. Here’s the full citation:

Namey E, Guest G, McKenna K, Chen M. Evaluating bang for the buck: a cost effectiveness comparison between individual interviews and focus groups based on thematic saturation levels. 2016 September; 37(3): 425-440.