The Good News about Bad News

Posted by nnlmneo on May 19th, 2017 Posted in: Blog

---

Years ago, I conducted a campus climate assessment for a smallish university. Campus climate assessments are just an academic-fancy word for student satisfaction surveys. Our results were mostly positive because, in general, our college students were a happy bunch.

But, even in the Magic Kingdom of College, nothing is perfect.  Ratings for the campus computer labs were less than stellar. Which statistic do you think the campus vice presidents wanted to talk about?  That’s right: that darkish mark on their otherwise pristine record. Specifically, they wanted me to explain why students were not completely satisfied with the computer labs.

Of course, I had no idea.  We didn’t asked students to explain their satisfaction levels.

I learned an important lesson that day: No one has much to say about good news. Don’t get me wrong: positive program metrics rock. They make great sound bites for elevator speeches. Thing is, you can say all that needs to be said in five floors or less.

On the other hand, discussions about negative findings can’t be resolved in a lift to the moon. Everyone wants to know why respondents aren’t happy, don’t learn, won’t engage. Unfortunately, metrics are just facts. They have no meaning without context. If I’m in a windowless office and my phone app tells me the outdoor temperature dropped 15 degrees in the last hour, I know almost nothing.  I need to tune into the Weather Channel or, at least, look out a window.

Learning Opportunity Alert

The good news is, a negative evaluation finding alerts you to a learning opportunity.  However, you need to be prepared to capitalize on it. Your evaluation plan should include follow-up discussions with key informants, so you can understand your metrics. Preferably, you prepared to  talk with a mix of stakeholders. Customers or users can explain the why behind the findings and provide ideas for improvement. Service providers or program implementers can tell you how they might act on customer feedback. Key decision makers, such as administrators or funders, will express their level of concern for findings and provide feedback about their willingness to invest in improvements.

Be Prepared

If your resources or time is limited, the easiest way to get feedback is through key informant interviews.  If you go this route, choose key findings to address and present them in a highly visual manner. You don’t want to use up valuable interview time forcing your interviewees to process your information. You can send them a more extensive report in advance, but don’t assume they will read it. Advanced reports should be “top shelf,” with distilled findings in bullet points.

Ideally, you can convene discussion groups. You get more varied feedback and groups often generate more creative solutions compared to sole interviewees. Plus, you provide more people with an opportunity to engage in your evaluation findings. However, you need careful plans and reliable assistants to systematically collect feedback from larger groups of people  Here are some tips for multi-group evaluation discussions:

• Develop a structured discussion guide for your small groups. See our blog article on data parties  or check out the discussion methods described at the Liberating Structures website.
• Carefully select your invitees (your sample). You may want to strategically create groups with different types of stakeholders.  For example, service providers and users might create awesome brainstorming groups.   On the other hand, you want to be careful mixing people of varying statuses.  For post hoc discussion on campus climate data, I would think twice about seating administrators with service providers, who might not speak freely in front of the “higher ups.”
• Have a plan for capturing comments. This is critical. If possible, have a team of reliable notetakers whose sole function is to take notes and write them up. They need to have good listening and writing skills. They also should be available to you after the meeting to answer questions about the conversations. If you do not have access to a supply of table facilitators, plan for individual table report-backs to the larger group. Record the discussion, if possible.  Listen carefully and ask questions.
• Validate your findings from the discussion process:  If you have notetakers or co-facilitators, ask them to read your final report for accuracy and completion. You can also present findings to others who participated in the discussion. When you are inviting participants, ask a handful if they will commit to reading a summary of the meeting.

The challenges in life are where we grow and improve.  It is as true for programs as it is for humans.  Be prepared to take the advantage of the learning opportunities offered by bad news.

 
Note: link updated 9/21/2017.