Have you ever found yourself starring at a flip drive or folder full of evaluation data, wondering how the heck to make sense of it? Maybe you collected a bunch of information as a requirement for a funded project. Maybe a VIP handed you data files and tasked you with “producing a report on the findings.” Now, there you sit, trying to figure out how to move forward.
You know what’s missing in this scenario? Evaluation questions. Data cannot provide you with answers if no one ever articulated questions.
It is ideal, of course, to write your evaluation questions before you design your data collection processes. Otherwise, to misquote the great Yogi Berra, “if you don’t know what your evaluation questions are, you might not answer them.” As with most evaluation planning, it’s best to get your stakeholders involved in question development, to be sure you answer their questions as well as your own.
However, we don’t live in an ideal world most of the time. The good news is, you can introduce evaluation questions during the analysis process. Whenever you suffer from data-based confusion, stop and think “Wait, what exactly do I need to know here?” List your questions, then figure out how to analyze your data to answer them.
For clarification, I’m not talking about the questions used to collect a single bit of information, like those used in surveys and interview guides. Evaluation questions are related to your program, broadly defining all the information you need in order to implement a program and assess its value. Specifically, your questions identify information to (a) plan well (b) conduct and adapt your activities as needed or (c) make informed decisions about the value of a program. These questions also set boundaries for your data collection methods, helping you collect only information that is useful to you. Ultimately, your evaluation questions guide your data collection methods, analysis, and reporting.
To demonstrate sample evaluation questions, let’s revisit our From Dusk to Dawn Project, the fictional health information outreach project to vampires featured in other NEO blog posts. In case you are unfamiliar with our un-outreach to the un-dead, the image to the left provides a project summary. Evaluation questions should be written for three key decision-making points in a program: planning; implementation; and outcomes/value assessment. Here are examples of some evaluation questions I would use if I were evaluating the From Dusk to Dawn program.
Planning: What do we need to know to plan this program?
Implementation: What are we doing and how well are we doing it?
Outcomes What did we accomplish with this program?
If the project team starts with these questions, we can develop questionnaires, interview guides, and other evaluation methods with precision, collecting the most necessary and useful data. These questions also will frame our data analysis and structure our reports to stakeholders.
If you want more examples of evaluation questions, check out these resources:
It takes resolve to begin planning an evaluation by articulating evaluation questions, but it pays off BIG in the end. If you are on teams or work groups initiating evaluation projects, press for the group to do their prep work. Persuade them to start with evaluation questions.
Repeat after me: Friends don’t let friends do question-free evaluation.
Note: link updated 9/21/2017.