Years ago, I conducted a campus climate assessment for a smallish university. Campus climate assessments are just an academic-fancy word for student satisfaction surveys. Our results were mostly positive because, in general, our college students were a happy bunch.
But, even in the Magic Kingdom of College, nothing is perfect. Ratings for the campus computer labs were less than stellar. Which statistic do you think the campus vice presidents wanted to talk about? That’s right: that darkish mark on their otherwise pristine record. Specifically, they wanted me to explain why students were not completely satisfied with the computer labs.
Of course, I had no idea. We didn’t asked students to explain their satisfaction levels.
I learned an important lesson that day: No one has much to say about good news. Don’t get me wrong: positive program metrics rock. They make great sound bites for elevator speeches. Thing is, you can say all that needs to be said in five floors or less.
On the other hand, discussions about negative findings can’t be resolved in a lift to the moon. Everyone wants to know why respondents aren’t happy, don’t learn, won’t engage. Unfortunately, metrics are just facts. They have no meaning without context. If I’m in a windowless office and my phone app tells me the outdoor temperature dropped 15 degrees in the last hour, I know almost nothing. I need to tune into the Weather Channel or, at least, look out a window.
The good news is, a negative evaluation finding alerts you to a learning opportunity. However, you need to be prepared to capitalize on it. Your evaluation plan should include follow-up discussions with key informants, so you can understand your metrics. Preferably, you prepared to talk with a mix of stakeholders. Customers or users can explain the why behind the findings and provide ideas for improvement. Service providers or program implementers can tell you how they might act on customer feedback. Key decision makers, such as administrators or funders, will express their level of concern for findings and provide feedback about their willingness to invest in improvements.
If your resources or time is limited, the easiest way to get feedback is through key informant interviews. If you go this route, choose key findings to address and present them in a highly visual manner. You don’t want to use up valuable interview time forcing your interviewees to process your information. You can send them a more extensive report in advance, but don’t assume they will read it. Advanced reports should be “top shelf,” with distilled findings in bullet points.
Ideally, you can convene discussion groups. You get more varied feedback and groups often generate more creative solutions compared to sole interviewees. Plus, you provide more people with an opportunity to engage in your evaluation findings. However, you need careful plans and reliable assistants to systematically collect feedback from larger groups of people Here are some tips for multi-group evaluation discussions:
The challenges in life are where we grow and improve. It is as true for programs as it is for humans. Be prepared to take the advantage of the learning opportunities offered by bad news.
Note: link updated 9/21/2017.