{"id":4739,"date":"2016-09-02T11:19:07","date_gmt":"2016-09-02T18:19:07","guid":{"rendered":"https:\/\/nnlm.gov\/evaluation\/blog\/?p=4739"},"modified":"2019-09-24T10:10:45","modified_gmt":"2019-09-24T17:10:45","slug":"from-logic-model-to-proposal-evaluation-part-2","status":"publish","type":"post","link":"https:\/\/news.nnlm.gov\/nec\/2016\/09\/02\/from-logic-model-to-proposal-evaluation-part-2\/","title":{"rendered":"From Logic Model to Proposal Evaluation – Part 2: The Evaluation Plan"},"content":{"rendered":"

\"Photo<\/a>Last week<\/a> we wrote some basic goals and objectives for a proposal about teaching health literacy skills to vampires in Sunnydale.\u00a0 Here\u2019s what the goals and objectives look like, taken from the Executive Summary statement in last week\u2019s post:<\/p>\n

Goal: The goal of our From Dusk to Dawn <\/em>project is to improve the health and well-being of vampires in the Sunnydale community.<\/p>\n

Objective 1: We will teach 4 hands-on evening classes on the use of MedlinePlus and PubMed to improve Sunnydale vampires\u2019 ability to find consumer health information and up to date research about health conditions.<\/p>\n

Objective 2: We will open a 12-hour \u201cDusk-to-Dawn\u201d health reference hotline to help the vampires with their reference questions.<\/p>\n

There are also three outcomes that we have identified:<\/p>\n

    \n
  1. Short-term: Increased ability of the Internet-using Sunnydale vampires to research needed health information.<\/li>\n
  2. Intermediate: These vampires will use their increased skills to research health information for their brood.<\/li>\n
  3. Long-term: Overall, the Sunnydale vampires will have improved health and as a result form better relationships with the human community of Sunnydale.<\/li>\n<\/ol>\n

    To get to an evaluation plan from here you have to know that there are basically two kinds of things you\u2019ll want to measure: process and outcomes.<\/p>\n

    Process assessment<\/strong> measures that you did what you said you would do and the way you said you would do it. For example, you can count the number of classes you taught, how many people attended, and whether their survey responses showed that they thought you did a good job teaching.<\/p>\n

    Also you might want to show that you were willing to make changes in the plan if review of your process assessment showed that you weren\u2019t getting the results you wanted.\u00a0 For example, if you planned all your classes in early evening, but few vampires attended, you might interview some vampires and find out that early evening is mealtime for most vampires, and move your classes to a different time to increase attendance.\u00a0 Your evaluation plan could show that you are collecting that information and that you will be responsive to what you see happening.<\/p>\n

    Outcome assessment<\/strong> measures the extent to which your project had the impact that you hoped it would on the recipients of the project, or even greater on their overall organizations or communities. We showed the first step of outcome assessment in last week\u2019s assignment, but I\u2019m going to break it down a little more here.\u00a0 Put in basic terms, to do an outcome assessment, you state your outcome, you add in an indicator, <\/em>a target<\/em>, and a time frame <\/em>to come up with a measurable objective, and then you write out the source <\/em>of your data, your data collection method, <\/em>and your data collection timing <\/em>to complete the picture.\u00a0 Let\u2019s talk about each item here:<\/p>\n

    Indicator: <\/em>This is the evidence you can gather that shows whether or not you met your outcomes.\u00a0 If one of your outcomes is that the vampires have increased ability to research health information, how would you know if that had happened? The indicator could be their increased confidence level in finding health information, or it could be improvement in skills test scores given before and after a training session.<\/p>\n

    Target: <\/em>The target is the goal that makes this project look like a success to you.\u00a0 For example, if the vampires improve their test scores by 50% over a baseline test, is that enough to say you have successfully reached that outcome?\u00a0 And how many of the vampires need to reach that 50% goal?\u00a0 All of them? One of them?\u00a0 Targets can be hard to identify, because you don\u2019t want them to be too hard to reach but if they\u2019re too easy your funder may not be impressed with your ambition.\u00a0 Sometimes you can work with the funder or other stakeholders on setting targets that are credible.<\/p>\n

    Time frame: <\/em>This is the point in time that when the threshold for success will be achieved.\u00a0 So if you want to make sure the vampires increased their ability by the end of your training, then the time frame would be by the end of your training.<\/p>\n

    Data Source:<\/em> This is the location where your information is found. Often, data sources are people (such as participants or observers) but they also may be records, pictures, or meeting notes. Here are some examples of data sources<\/a>.<\/p>\n

    Data Collection Methods:<\/em> Evaluation methods are the tools you use to collect data, such as a survey, observation, or quiz.\u00a0 Here is more examples of data collection methods<\/a>.<\/p>\n

    Data Collection Timing:<\/em> The data collection timing is describing exactly when you will be collecting the data.<\/p>\n

    What does your final evaluation plan look like?\u00a0 <\/strong><\/p>\n

    Here is a sample piece of an evaluation plan for the Dusk to Dawn<\/em> proposal.<\/p>\n

    Objective 1<\/strong>: teach 4 hands-on evening classes on the use of MedlinePlus and PubMed to improve Sunnydale vampires\u2019 ability to research consumer health information and up to date research about health conditions.<\/p>\n

    Process Assessment<\/strong>: The PI will collect the following information to ensure that classes are being taught; expected attendance figures are being reached; teachers are doing a good job teaching classes (including surviving the classes).\u00a0 Data will be reviewed after each class and changes will be made to the program as needed to reach target goals:<\/p>\n

    \u25ca Participant roster to measure attendance figures
    \n\u25ca Class evaluations to measure teacher performance
    \n\u25ca Count of number of teachers at the beginning and ending of each class to measure survival of instructors
    \n\u25ca Project team will meet after the second class to review success and lessons learned and to consider course corrections to ensure objectives are met<\/p>\n

    Outcome Assessment<\/strong>:
    \nMeasureable Objective: In a post-test given immediately after each class, a minimum of 75% of Sunnydale vampire attendees demonstrate that they learned how to find needed resources in PubMed and MedlinePlus by showing at least a 50% improvement over the pre-test.<\/em><\/p>\n

    Based on Level 2 (Learning) in the Kirkpatrick Model<\/a>, a test will be created with some basic health questions to be researched. Class participants will be given these questions as a pre-test before the class, and then will be given the same questions after the class as a post-test.\u00a0 This learning outcome will be considered successful if a minimum of 75% of Sunnydale vampire participants demonstrate that their scores improved by at least 50%.<\/p>\n

    Last wishes, I mean, thoughts<\/strong><\/p>\n

    This is not a complete evaluation plan, but the purpose of these two posts has been to show how you can go from a logic model to the evaluation plan of a proposal.\u00a0 Don\u2019t worry if all your outcomes cannot be measured in the scope of your project.\u00a0 For example, in this Dusk to Dawn<\/em> project, it might have been dangerous to find out if the vampires had passed on needed health information to their brood, even harder to find out whether the vampires had become more healthy as a result of the information.\u00a0 This doesn\u2019t mean to leave these outcomes out, but you may want to acknowledge that measuring some outcomes is out of the scope of the project\u2019s resources.<\/p>\n

    As Grandpa Munster once said \u201cDon’t let time or space detain ya, here you go, to Transylvania.\u201d<\/em><\/p>\n

    Photo credit: Photo of 365::79 – Vampire Cat by Sarah Reid on Flickr under Creative Commons license CC BY 2.0<\/a>.\u00a0 No changes were made.<\/p>\n

     <\/p>\n","protected":false},"excerpt":{"rendered":"

    Last week we wrote some basic goals and objectives for a proposal about teaching health literacy skills to vampires in Sunnydale.\u00a0 Here\u2019s what the goals and objectives look like, taken from the Executive Summary statement in last week\u2019s post: Goal: The goal of our From Dusk to Dawn project is to improve the health and… Read More »<\/a><\/p>\n","protected":false},"author":2959,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[27],"tags":[],"class_list":["post-4739","post","type-post","status-publish","format-standard","hentry","category-blog"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p8ICUo-1er","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/4739","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/users\/2959"}],"replies":[{"embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/comments?post=4739"}],"version-history":[{"count":2,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/4739\/revisions"}],"predecessor-version":[{"id":6519,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/4739\/revisions\/6519"}],"wp:attachment":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/media?parent=4739"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/categories?post=4739"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/tags?post=4739"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}