{"id":4630,"date":"2016-08-05T11:16:25","date_gmt":"2016-08-05T18:16:25","guid":{"rendered":"https:\/\/nnlm.gov\/evaluation\/blog\/?p=4630"},"modified":"2019-09-24T10:13:28","modified_gmt":"2019-09-24T17:13:28","slug":"the-kirkpatrick-model-part-2-with-humans","status":"publish","type":"post","link":"https:\/\/news.nnlm.gov\/nec\/2016\/08\/05\/the-kirkpatrick-model-part-2-with-humans\/","title":{"rendered":"The Kirkpatrick Model (Part 2) &#8212; With Humans"},"content":{"rendered":"<p>Disclaimer: Karen\u2019s <a href=\"https:\/\/nnlm.gov\/evaluation\/blog\/2016\/07\/28\/program-outcomes-using-kirkpatrick-model\/\">blog pos<\/a>t last week on the Kirkpatrick Model used an example that was hypothetical.\u00a0 We want to be clear that the NEO has never evaluated any programs directed toward improving health outcomes for vampires.<\/p>\n<p>However, we can claim success in applying the <a href=\"http:\/\/www.kirkpatrickpartners.com\/OurPhilosophy\/TheKirkpatrickModel\/tabid\/302\/Default.aspx\">Kirkpatrick Model <\/a>for National Network of Libraries of Medicine (NN\/LM) training programs.<\/p>\n<p>The NN\/LM\u2019s mission is to promote the biomedical and consumer health resources of the National Library of Medicine.\u00a0 One strategy that is popular with NN\/LM\u2019s Regional Medical Libraries, which lead and manage the network, is the \u201ctrain-the-trainer\u201d program. These programs teach librarians and others about NLM resources so that they, in turn, will teach their peers, patients, or clients. When the NEO provides evaluation consulting for train-the-trainer programs, we rely heavily on the Kirkpatrick Model.<\/p>\n<p><strong>Kirkpatrick Outcomes Levels and Logic Models<\/strong><\/p>\n<p>For example, the NN\/LM\u2019s initiative to reach out to community college librarians incorporated \u201ctrain-the-trainer\u201d as one of several strategies to promote use of NLM resources in community college health professions programs. While the initiative was multi-pronged, train-the-trainer programs for community college librarians was a major strategy of the project. The Kirkpatrick Model helped our task force define outcomes and develop measures for this activity.<\/p>\n<p>Our logic model led us to the following program theory:<\/p>\n<p>If we train community college librarians to use National Library of Medicine Resources<\/p>\n<ul>\n<li>They will respond favorably to our message <em>(Reaction)<\/em><\/li>\n<li>And discover new, useful health information resources that <em>(Learning)<\/em><\/li>\n<li>They will use when working with faculty and students <em>(Behavior)<\/em><\/li>\n<li>Which will lead to increased use of NLM resources among community college faculty and staff <em>(Results)<\/em><\/li>\n<\/ul>\n<p style=\"text-align: center\"><a href=\"\/neo\/wp-content\/uploads\/sites\/3\/2016\/08\/Slide1.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-4640\" src=\"\/neo\/wp-content\/uploads\/sites\/3\/2016\/08\/Slide1-300x65.jpg\" alt=\"Progression from like to learn to do to make a difference\" width=\"978\" height=\"212\" \/><\/a><\/p>\n<p><strong>Measuring Outcomes<\/strong><\/p>\n<p>We developed two simple measurement tools to assess the four outcome levels.\u00a0 To measure <em>Reaction<\/em>, RML instructors administered a standard one-page session evaluation form that has been used for years by instructors who provide NN\/LM training sessions. The form collects participants\u2019 feedback, including the grade (A through F) they would assign to the class. This form was our measure of participant reaction.<\/p>\n<p>The other three levels were assessed using a follow-up questionnaire sent to the training participants several months after their training. On this questionnaire, we asked them a series of yes\/no questions:<\/p>\n<p><strong>Learning: <\/strong>At this training session, did you learn about health information resources that were NEW to you?<\/p>\n<p><strong>Behavior: <\/strong>Regarding the NEW resources you learned at the training, have you done any of the following?<\/p>\n<ul>\n<li>Shown these resources to students?<\/li>\n<li>Used the resources when preparing lesson plans?<\/li>\n<li>Shown the resources to community college faculty or staff?<\/li>\n<li>Used the resources to answer reference questions?<\/li>\n<\/ul>\n<p><strong>Results: Do you know if the resources are being used by<\/strong><\/p>\n<ul>\n<li>Students?<\/li>\n<li>Faculty, administration, or staff at your organization?<\/li>\n<li>The librarians at your institution?<\/li>\n<\/ul>\n<p>We knew our <em>Results<\/em> questions were weak. They obviously were very subjective. Most of the respondents said they didn\u2019t know about use beyond their library staff members. Unfortunately, we did not have resources for a more objective measure of our anticipated results (e.g., surveying faculty and students at participating schools). Our dilemma was not unusual. Many practitioners of the Kirkpatrick Model agree that assessing <em>Results-level<\/em> outcomes can be costly and challenging.<\/p>\n<p>However, in anticipation that this Results-level measure might not work, we had a back-up plan inspired by Robert Brinkerhoff\u2019s Success Case Method (which we posted about <a href=\"https:\/\/nnlm.gov\/evaluation\/blog\/2014\/07\/02\/telling-trainings-story-the-success-case-method\/\">here<\/a>). In this approach, evaluators ask training participants to describe how their training benefited the organization.\u00a0 We ended the questionnaire with the following open-ended question:\u00a0<em>Please describe how the training you received on National Library of Medicine resources has made a difference for you or your organization.<\/em><\/p>\n<p>This question worked well, with 57% of respondents providing examples of how the training improved their customer services. They reported using the NLM resources to provide reference services and incorporating NLM resources into their information literacy classes for health professions students.\u00a0 Some also were talking to faculty about the importance of teaching health professions students about RML resources that students could use after graduation.<\/p>\n<p style=\"text-align: left\">In the end, the Kirkpatrick Model helped us get metrics and qualitative information that helped to assess the effectiveness of our train-the-trainer activities. \u00a0Most of the training participants who responded to our follow-up questionnaire learned new resources and were promoting them to student and faculty. Their stories showed that the NN\/LM training improved the services they were delivering to their users.<\/p>\n<p>The NEO has drawn on the Kirkpatrick Model to design evaluation methods for similar projects, including our own evaluation training programs. \u00a0It is a great tool for helping program planners to define concrete objectives and create measures that are closely linked to desired outcomes.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Disclaimer: Karen\u2019s blog post last week on the Kirkpatrick Model used an example that was hypothetical.\u00a0 We want to be clear that the NEO has never evaluated any programs directed toward improving health outcomes for vampires. However, we can claim success in applying the Kirkpatrick Model for National Network of Libraries of Medicine (NN\/LM) training&#8230; <a href=\"https:\/\/news.nnlm.gov\/nec\/2016\/08\/05\/the-kirkpatrick-model-part-2-with-humans\/\">Read More &raquo;<\/a><\/p>\n","protected":false},"author":2959,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[27],"tags":[],"class_list":["post-4630","post","type-post","status-publish","format-standard","hentry","category-blog"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p8ICUo-1cG","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/4630","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/users\/2959"}],"replies":[{"embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/comments?post=4630"}],"version-history":[{"count":1,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/4630\/revisions"}],"predecessor-version":[{"id":6463,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/4630\/revisions\/6463"}],"wp:attachment":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/media?parent=4630"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/categories?post=4630"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/tags?post=4630"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}