Evaluation Questions: GPS for Your Data Analysis

Posted by nnlmneo on September 8th, 2017 Posted in: Blog

---

Have you ever found yourself starring at a flip drive or folder full of evaluation data, wondering how the heck to make sense of it?  Maybe you collected a bunch of information as a requirement for a funded project.  Maybe a VIP handed you data files and tasked you with “producing a report on the findings.” Now, there you sit, trying to figure out how to move forward.

You know what’s missing in this scenario? Evaluation questions. Data cannot provide you with answers if no one ever articulated questions.

It is ideal, of course, to write your evaluation questions before you design your data collection processes. Otherwise, to misquote the great Yogi Berra, “if you don’t know what your evaluation questions are, you might not answer them.” As with most evaluation planning, it’s best to get your stakeholders involved in question development, to be sure you answer their questions as well as your own.

However, we don’t live in an ideal world most of the time. The good news is, you can introduce evaluation questions during the analysis process.  Whenever you suffer from data-based confusion, stop and think “Wait, what exactly do I need to know here?”  List your questions, then figure out how to analyze your data to answer them.

For clarification, I’m not talking about the questions used to collect a single bit of information, like those used in surveys and interview guides.  Evaluation questions are related to your program, broadly defining all the information you need in order to implement a program and assess its value.  Specifically, your questions identify information to (a) plan well (b) conduct and adapt your activities as needed or (c) make informed decisions about the value of a program. These questions also set boundaries for your data collection methods, helping you collect only information that is useful to you. Ultimately, your evaluation questions guide your data collection methods, analysis, and reporting.

Revisiting Sunnydale

To demonstrate sample evaluation questions, let’s revisit our From Dusk to Dawn Project, the fictional health information outreach project to vampires featured in other NEO blog posts.  In case you are unfamiliar with our un-outreach to the un-dead, the image to the left provides a project summary.  Evaluation questions should be written for three key decision-making points in a program: planning; implementation; and outcomes/value assessment. Here are examples of some evaluation questions I would use if I were evaluating the From Dusk to Dawn program.

Planning: What do we need to know to plan this program?

• What qualities does our program need to attract vampires?
• What community organizations currently serve vampires and might want to partner on this program?
• What challenges do we face in implementing our plans?

Implementation: What are we doing and how well are we doing it?

• How many calls does the From Dusk to Dawn Hotline get per night from vampires seeking health information?
• How satisfied are the vampires with the training sessions?
• How do the reference librarians and instructors suggest we improve the program activities?

Outcomes What did we accomplish with this program?

• Did we meet our objective targets?
• What unanticipated positive or negative outcomes came from our efforts?
• What reactions do different stakeholder groups have to the outcomes of this program? (Stakeholders include vampires and their families,, Sunnydale city leaders, vampire-serving CBOs, and human residents)

If the project team starts with these questions, we can develop questionnaires, interview guides, and other evaluation methods with precision, collecting the most necessary and useful data.  These questions also will frame our data analysis and structure our reports to stakeholders.

Other Examples

If you want more examples of evaluation questions, check out these resources:

• Developing an Evaluation Plan from the Community Toolbox
• Specify The Key Evaluation Questions from Better Evaluation
• Figure 8: Community Assessment Questions  from Getting Started with Community-Based Outreach, which is Booklet 1 of the NEO’s Planning and Evaluating Outreach Evaluation Projects booklet series.
• Worksheet 2: Process Assessment Questions and Methods from Planning Outcomes-Based Outreach Programs, Booklet 2 in the NEO series.

It takes resolve to begin planning an evaluation by articulating evaluation questions, but it pays off BIG in the end.  If you are on teams or work groups initiating evaluation projects, press for the group to do their prep work.  Persuade them to start with evaluation questions.

Repeat after me: Friends don’t let friends do question-free evaluation.

 

 

 Note: link updated 9/21/2017.