NEO Shop Talk November 19th, 2017
CategoriesCategories Contact Us Archives NEO Main Site Search



Date prong graphic

American Evaluation Association 2007 Meeting

Posted by on January 10th, 2008 Posted in: News

The American Evaluation Association held its annual meeting in Baltimore this year and, as usual, the workshops were great and the program was packed with useful information aimed at evaluators with varied levels of expertise. Baltimore itself was wonderful; somehow they managed to provide very nice weather even in November! Of the sessions I attended, my very favorites were the ones with Brian Yates from American University, who is a sort of evangelist about “cost-inclusive” evaluation. Apparently, his outlook that “costs are at least as important to measure as outcomes” is not universally accepted amongst all evaluators. His theme is that all outcomes have costs and some outcomes are monetary. One of his presentations was about starting a cost study, the other was titled “Costs Are All That Matters.” I also took some workshops:

Quantitative Methods I spent two days in this workshop, which was quite fast-paced–the instructor claimed to be providing us with a semester’s worth of information! Participants who had never had methods courses were scrambling a bit to keep up; I have had methods courses but not since the last century, so I experienced the workshop as a challenging review. One of the main things that I took from the workshop was the instructor’s delightful phrase, “reasonable people will disagree.” It turns out that statistics can be so complicated that not even the statisticians always agree about when and how to use which approaches. For example, there is controversy about whether hypothesis testing is the best way to improve knowledge. Our instructor suggested that, since it’s established tradition for research, we should combine hypothesis testing with other approaches. Here are some other, probably more useful points I noted:

  • One person’s “6” might be another person’s “3”
  • More than 10 choices will be hard for a respondent to manage. With fewer response choices, results tend to be more reliable
  • Always label the middle in a range of rating choices (and the two ends, also)
  • Variance in program outcomes can be very important to stakeholders; evaluators are looking to find out whether a program actually had an effect or whether an observed change was simply due to chance (or due to something other than the program)
  • Inferential statistics, which are used to generalize to populations of interest (ie, should I bring more people into my program?) are based on the assumption that the population in question is normally distributed
  • A number of statistical tests have their origins in agricultural research and were never really intended for use with humans

Our instructor ended the 2-day session with this warning: “I’ve taught you enough now that you’re really dangerous. Don’t try to do this without expert guidance.” True enough, but a class like this will help to communicate with experts.

Online Survey Research This was a half-day class that the instructors had not taught before. I empathized with them as they ran out of time and expressed the realization that it should have been a full-day class. Some of my notes:

  • Survey objectives are not the same as program objectives. Surveys can’t fix problems with programs
  • The instructors cited a 27% response rate as a “very good” one
  • Research has shown that people write longer responses to open-ended questions in online questionnaires than they do on printed ones

Note: Links updated 9/20/2017.

Image of the author ABOUT susan

Email author View all posts by

Subscribe to our blog!

NEO’s Latest Tweet

Annual Archive

This project is funded by the National Library of Medicine (NLM), National Institutes of Health (NIH) under cooperative agreement number UG4LM012343 with the University of Washington.

NNLM and NATIONAL NETWORK OF LIBRARIES OF MEDICINE are service marks of the US Department of Health and Human Services | Copyright | Download PDF Reader