The Cookie Exercise: Setting Criteria

Posted by nnlmneo on June 9th, 2017 Posted in: Blog, Data Analysis

---

What do Oreo cookies have to do with evaluation?

In program evaluation, we want to know the extent to which a program is successful.  If we’re evaluating an activity we’re doing at our job, we might be trying to figure out if more money should be put towards future iterations of that activity or if it should be discontinued. Or maybe there are ways the activity can be improved to achieve our desired outcomes. (“But what about the cookies?”)

Evaluating your data is dependent on having something to compare it to. If we don’t have something to compare our data to, we are just describing it– we aren’t able to know if it matters or not. If we don’t know that our project isn’t meeting our criteria, we won’t know where to improve. (“Um… the cookies?”)

In order to get people to understand how to create a set of criteria and standards, we use this yummy exercise* with Oreo-type cookies that we use in the NEO’s workshop on analyzing evaluation data. (“Finally, the cookies!”). Here’s how it works.  We tell people that they are on a council to select the next Girl Scout Cookie. The decision has already been made that the next cookie will be a chocolate vanilla cream cookie, like an Oreo.

Participants need to create a set of criteria on which the cookies will be judged, like: crunchiness of the chocolate cookie, smooth texture of the cream filling, flavor of chocolate cookie, etc. Then for each criteria we ask everyone to establish a standard that the cookie must perform at or above for it to be qualified to be selected, on a scale of 1-10.  So for example, if you say that the cream filling texture must be smooth, and you give that a 6, and you say that the chocolate cookie part must be extremely chocolaty, and you give that an 8, then, in order for this cookie to be in the running, it must have at least a 6 for creaminess and at least an 8 for chocolaty-ness.

In the class we actually give everyone two or three different brands of chocolate vanilla cream cookies and ask them to go through the exercise.  As a note: if you’re planning to do this, it’s harder than you might think to find cookies that don’t have a recognizable mark in the design that tells you exactly who makes them. If the cookie says “Oreo” right in the middle of the cookie, it might bias the cookie tasters.

After this experiment, we ask people in the class to discuss their results. Which cookie did best overall? Did any of your cookies meet your minimum requirements? In addition, think about what kinds of external factors might be important.  Maybe your stakeholders are looking for a gluten free cookie, or a cookie that doesn’t have palm oil in it.

I think this is a fun project that you could try on real-life things other than cookies to get used to it.  Maybe you want to do a chart like this when deciding what school you want your child to attend, or whether or not you want to finish reading a book (speaking as a person who feels guilty putting a book down even when I dislike it).

If you want to use our class materials to do this exercise, they are located here https://nnlm.gov/classes/dataanalysis and the cookie assignment is called Data Analysis Exercise 1: Logic of Eval (it is a Word document).

*This cookie exercise is adapted from Hallie Preskill and Darlene Russ-eft’s Building Evaluation Capacity: 72 Activities for Teaching and Training.