NEO Shop Talk December 18th, 2017
CategoriesCategories Contact Us Archives NEO Main Site Search



Date prong graphic

Cindy’s AEA Report

Posted by on November 10th, 2006 Posted in: News, Practical Evaluation

AEA has been my favorite professional conference since I started attending about 5 years ago. This year, I attended some of the same sessions as my colleagues, so I won’t report on those. Here are some other sessions that I thought were noteworthy.

Improving Data Quality Through Two-Dimensional Surveying: The Kano Method

On my plane trip to Portland, I had the pleasure of sitting beside Vathsala Stone, an evaluator from University of Buffalo who had a poster session on the Kano Method (her presentation partner was Stephen Bauer, University at Buffalo). I did not get to the poster sessions, but Dr. Stone explained this technique on our plane trip. The Kano Method is used in product development to get an accurate assessment of how important various product features are to consumers, allowing developers to make more accurate product-design decisions. I think the Kano Method has application for development of products like web-based health resources, educational materials, and training curriculum. I found an article at a web site for the Center for Quality of Management that provides a description of the method :

Three Effective Ways to Build Organizational Capacity in Local Nonprofit Agencies

The terms “evaluation capacity building” refers to increasing organizations’ ability to identify meaningful indicators to evaluate, collect useful evaluation data, and use the data to improve their programs. The ideas that I took away from this panel presentation emphasized methods for motivating grantees to use evaluation data. Until program implementers see data as a tool for improving their programs, evaluation will continue to seem like a meaningless, funder-imposed requirement.

Abraham Wandersman (University of South Carolina) and Jan Yost (Health Foundation of Central Massachusetts) presented their “Results-Oriented Grantmaking and Grant-Implementation” approach, in which nonprofit grantees receive an orientation in use of tools that allow them evaluate their programs by addressing 10 accountability questions. Michael Hendricks, an independent consultant, described his work with a community health foundation to help grantees develop a “Managing for Results” orientation. One key strategy for the foundation was to ask for short reports bi-monthly about the indicators – essentially they report on what the indicators are showing and what changes in services were made as a result. (Funding recipients, of course, receive evaluation training so they can create these reports.) The third presenter, Ann Bessell, University of Miami, talked about using spider diagrams to display data and help organizations interpret and apply their evaluation findings.

What’s the Score: Disentangling Qualitative Data by Building Scoring Guides

Ann Davis and Phyllis Ault of Northwest Regional Educational Laboratory conducted a demonstration on developing scoring guides to assess products (e.g., multimedia presentations) or performances (e.g., effectiveness of learning communities). They described how to build a framework and establish criteria, then how to review and revise the rating form. They also provided concrete explanations of how to establish inter-rater reliability and how to assign a score based on ratings from multiple raters. I think scoring guides could be used to assess trainees’ ability to locate health information. Also, if high school peer tutor programs become more popular, they could be used to assess students’ presentations and give them feedback.

Image of the author ABOUT Cindy Olney

Cindy Olney is the Assistant Director of the NNLM Evaluation Office. She leads NNLM’s evaluation efforts, designs evaluation methods, and guides analysis and reporting of evaluation findings.

Email author View all posts by

Subscribe to our blog!

NEO’s Latest Tweet

Annual Archive

This project is funded by the National Library of Medicine (NLM), National Institutes of Health (NIH) under cooperative agreement number UG4LM012343 with the University of Washington.

NNLM and NATIONAL NETWORK OF LIBRARIES OF MEDICINE are service marks of the US Department of Health and Human Services | Copyright | Download PDF Reader