{"id":5220,"date":"2017-02-20T07:53:10","date_gmt":"2017-02-20T15:53:10","guid":{"rendered":"https:\/\/nnlm.gov\/evaluation\/blog\/?p=5220"},"modified":"2019-09-24T10:10:44","modified_gmt":"2019-09-24T17:10:44","slug":"abp-always-be-pilot-testing","status":"publish","type":"post","link":"https:\/\/news.nnlm.gov\/nec\/2017\/02\/20\/abp-always-be-pilot-testing\/","title":{"rendered":"ABP: Always Be Pilot-testing (some experiences with questionnaire design)"},"content":{"rendered":"<p style=\"text-align: right\"><img loading=\"lazy\" decoding=\"async\" class=\"alignright size-medium wp-image-5225\" src=\"\/neo\/wp-content\/uploads\/sites\/3\/2017\/02\/Booklet-3-Collecting-and-Analyzing-Data-232x300.jpg\" alt=\"Cover of NEO's Booklet 3 on collecting and analyzing evaluation data\" hspace=\"10\" width=\"232\" height=\"300\" align=\"right\" \/><\/p>\n<p>This week I have been working on a questionnaire for the Texas Library Association (TLA) on the cultural climate of TLA.\u00a0 Having just gone through this process, I will tell you that NEO\u2019s <a href=\"https:\/\/nnlm.gov\/neo\/professional-development\/guides\">Booklet #3: Collecting and Analyzing Evaluation Data<\/a> has really useful tips on how to write questionnaires (p. 3-7).\u00a0 I thought it might be good to talk about some of the tips that had turned out particularly useful for this project, but the theme of all of these is &#8220;always be pilot-testing.&#8221;<\/p>\n<p><strong>NEO\u2019s #1 Tip: Always pilot test!<\/strong><\/p>\n<p>This questionnaire is still being pilot tested. So far I have thought the questionnaire was perfect at least 10 times, and we are still finding out about important changes from people who are pilot testing it for our committee.\u00a0 One important part of this tip is to include stakeholders in the pilot testing.\u00a0 Stakeholders have points of view that may not be included in the points of view of the people creating the survey.\u00a0 After we have what we think is a final version, our questionnaire will be tested by the TLA Executive Board.\u00a0 While this process sounds exhausting, every single change that has been made (to the questionnaire that I thought was finished) has fundamentally improved it.<\/p>\n<p><strong>There is a response for everyone who completes the question<u> <\/u><\/strong><\/p>\n<p>Our questionnaire asks questions about openness and inclusiveness to people of diverse races, ethnicities, nationalities, ages, gender identities, sexual identities, cognitive and physical disabilities, perceived socioeconomic status, etc. \u00a0We are hoping to get personal opinions from all kinds of librarians who live all over Texas.\u00a0 By definition this means that many of the questions are possibly sensitive, and may be hot button issues for some people.<\/p>\n<p>In addition to wording the questions carefully, it\u2019s important that every question has a response for everyone who completes the question. We would hate for someone not to find the response that best works for them, and then leave the questionnaire unanswered, or even worse get their feelings hurt or feel insulted. For example, we have a question about whether our respondents feel that their populations are represented in TLA\u2019s different groups (membership, leadership, staff, etc). At first the answers were just \u201cyes\u201d or \u201cno.\u201d\u00a0 But then (from responses in the pilot testing) we realized that a person may feel that they belong to more than one population. For example, what if someone was both Asian and had a physical disability.\u00a0 Perhaps they feel that one group is well represented and the other group not represented at all.\u00a0 How would they answer the question?\u00a0 Without creating a complicated response, we decided to change our response options to \u201cyes\u201d \u201csome are\u201d and \u201cno.\u201d<\/p>\n<p><strong>\u201cDon\u2019t Know\u201d or \u201cNot Applicable\u201d<\/strong><\/p>\n<p>In a similar vein, sometimes people do not know the answer to the question you are asking.\u00a0 They can feel pressured to make a choice among questions rather than skip the question (and if they do skip the question, the data will not show why). \u00a0For example, we are asking a question about whether people feel that TLA is inclusive, exclusionary or neither. \u00a0Originally I thought those three choices covered all the bases.\u00a0 But among discussions with Cindy (who was pilot testing the questionnaire), we realized that if someone simply didn\u2019t know, they wouldn\u2019t feel comfortable saying that TLA was neither inclusive nor exclusionary.\u00a0 So we added a \u201cDon\u2019t know\u201d option.<\/p>\n<p>Speaking from experience, the most important thing is keeping an open mind. Remember that the people taking your questionnaire will be seeing it from different eyes than yours, and they are the ones you are hoping to get information from.\u00a0 So, while I recommend following all the tips in Booklet 3, to get the best results, make sure that you test your questionnaire with a wide variety of people who represent those who will be taking it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This week I have been working on a questionnaire for the Texas Library Association (TLA) on the cultural climate of TLA.\u00a0 Having just gone through this process, I will tell you that NEO\u2019s Booklet #3: Collecting and Analyzing Evaluation Data has really useful tips on how to write questionnaires (p. 3-7).\u00a0 I thought it might&#8230; <a href=\"https:\/\/news.nnlm.gov\/nec\/2017\/02\/20\/abp-always-be-pilot-testing\/\">Read More &raquo;<\/a><\/p>\n","protected":false},"author":2959,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[13],"tags":[],"class_list":["post-5220","post","type-post","status-publish","format-standard","hentry","category-questionnaires-and-surveys"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p8ICUo-1mc","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/5220","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/users\/2959"}],"replies":[{"embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/comments?post=5220"}],"version-history":[{"count":1,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/5220\/revisions"}],"predecessor-version":[{"id":6508,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/5220\/revisions\/6508"}],"wp:attachment":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/media?parent=5220"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/categories?post=5220"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/tags?post=5220"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}