NEO Shop Talk October 23rd, 2017
CategoriesCategories Contact Us Archives NEO Main Site Search

Sep

08

Date prong graphic

Evaluation Questions: GPS for Your Data Analysis

Posted by on September 8th, 2017 Posted in: Blog


GPS

Have you ever found yourself starring at a flip drive or folder full of evaluation data, wondering how the heck to make sense of it?  Maybe you collected a bunch of information as a requirement for a funded project.  Maybe a VIP handed you data files and tasked you with “producing a report on the findings.” Now, there you sit, trying to figure out how to move forward.

You know what’s missing in this scenario? Evaluation questions. Data cannot provide you with answers if no one ever articulated questions.

It is ideal, of course, to write your evaluation questions before you design your data collection processes. Otherwise, to misquote the great Yogi Berra, “if you don’t know what your evaluation questions are, you might not answer them.” As with most evaluation planning, it’s best to get your stakeholders involved in question development, to be sure you answer their questions as well as your own.

However, we don’t live in an ideal world most of the time. The good news is, you can introduce evaluation questions during the analysis process.  Whenever you suffer from data-based confusion, stop and think “Wait, what exactly do I need to know here?”  List your questions, then figure out how to analyze your data to answer them.

For clarification, I’m not talking about the questions used to collect a single bit of information, like those used in surveys and interview guides.  Evaluation questions are related to your program, broadly defining all the information you need in order to implement a program and assess its value.  Specifically, your questions identify information to (a) plan well (b) conduct and adapt your activities as needed or (c) make informed decisions about the value of a program. These questions also set boundaries for your data collection methods, helping you collect only information that is useful to you. Ultimately, your evaluation questions guide your data collection methods, analysis, and reporting.

Revisiting Sunnydale

To demonstrate sample evaluation questions, let’s revisit our From Dusk to Dawn Project, the fictional health information outreach project to vampires featured in other NEO blog posts.  In case you are unfamiliar with our un-outreach to the un-dead, the image to the left provides a project summary.  Evaluation questions should be written for three key decision-making points in a program: planning; implementation; and outcomes/value assessment. Here are examples of some evaluation questions I would use if I were evaluating the From Dusk to Dawn program.

Planning: What do we need to know to plan this program?

Dawn to Dusk write up with icon of a vampire and a brief description of the program that reads:The goal of the From Dusk to Dawn project is to improve the health and well being of vampires in the Sunnydale community. In order to reach this goal, The program offers Hands-on evening classes on the use of MedlinePlus and PubMed find health information and up-to-date research about health concerns. An overnight “Dusk-to-Dawn” health reference hotline to help the vampires with their reference questions With these activities, we hope to see Increased ability of the Internet-using Sunnydale vampires to research needed health information Vampires using their increased skills to research health information for themselves and their broods Improved health of Sunnyvale vampires and Improved relationships between the vampire and human community of Sunnydale.

  • What qualities does our program need to attract vampires?
  • What community organizations currently serve vampires and might want to partner on this program?
  • What challenges do we face in implementing our plans?

Implementation: What are we doing and how well are we doing it?

  • How many calls does the From Dusk to Dawn Hotline get per night from vampires seeking health information?
  • How satisfied are the vampires with the training sessions?
  • How do the reference librarians and instructors suggest we improve the program activities?

Outcomes What did we accomplish with this program?

  • Did we meet our objective targets?
  • What unanticipated positive or negative outcomes came from our efforts?
  • What reactions do different stakeholder groups have to the outcomes of this program? (Stakeholders include vampires and their families,, Sunnydale city leaders, vampire-serving CBOs, and human residents)

If the project team starts with these questions, we can develop questionnaires, interview guides, and other evaluation methods with precision, collecting the most necessary and useful data.  These questions also will frame our data analysis and structure our reports to stakeholders.

Other Examples

If you want more examples of evaluation questions, check out these resources:

It takes resolve to begin planning an evaluation by articulating evaluation questions, but it pays off BIG in the end.  If you are on teams or work groups initiating evaluation projects, press for the group to do their prep work.  Persuade them to start with evaluation questions.

Repeat after me: Friends don’t let friends do question-free evaluation.

 

 

 Note: link updated 9/21/2017.

Image of the author ABOUT Cindy Olney
Cindy Olney is the Assistant Director of the NNLM Evaluation Office. She leads NNLM's evaluation efforts, designs evaluation methods, and guides analysis and reporting of evaluation findings.

Email author View all posts by

Subscribe to our blog!

NEO’s Latest Tweet

Annual Archive

This project is funded by the National Library of Medicine (NLM), National Institutes of Health (NIH) under cooperative agreement number UG4LM012343 with the University of Washington.

NNLM and NATIONAL NETWORK OF LIBRARIES OF MEDICINE are service marks of the US Department of Health and Human Services | Copyright | Download PDF Reader