{"id":5249,"date":"2017-03-06T07:30:38","date_gmt":"2017-03-06T15:30:38","guid":{"rendered":"https:\/\/nnlm.gov\/evaluation\/blog\/?p=5249"},"modified":"2019-09-24T10:10:44","modified_gmt":"2019-09-24T17:10:44","slug":"the-dark-side-of-questionnaires-how-to-identify-questionnaire-bias","status":"publish","type":"post","link":"https:\/\/news.nnlm.gov\/nec\/2017\/03\/06\/the-dark-side-of-questionnaires-how-to-identify-questionnaire-bias\/","title":{"rendered":"The Dark Side of Questionnaires: How to Identify Questionnaire Bias"},"content":{"rendered":"<p style=\"text-align: right\"><img loading=\"lazy\" decoding=\"async\" class=\"alignright wp-image-5250\" src=\"\/neo\/wp-content\/uploads\/sites\/3\/2017\/03\/Survey-Villain-284x300.jpg\" alt=\"Villain cartoon with survey questions\" width=\"385\" height=\"407\" align=\"right\" hspace=\"10\" \/><\/p>\n<p>People in my social media circles have been talking lately about bias in questionnaires.\u00a0 There <em>are<\/em> biased questionnaires.\u00a0 Some of them are biased by accident and some are on purpose.\u00a0 Some are biased in the questions and some are biased in other ways, such as the selection of the people who are asked to complete the questionnaires. Recently, a couple of my friends posted on Facebook that people should check out the NNLM Evaluation Office to learn about better questionnaires. Huzzah! This week\u2019s post was born!<\/p>\n<p>Here are a few things to look for when creating, responding to, or looking at the results of questionnaires.<\/p>\n<p><strong>Poorly worded questions<\/strong><\/p>\n<p>Sometimes simple problems with questions can lead to bias, whether accidental or on purpose.\u00a0 Watch out for these kinds of questions:<\/p>\n<ul>\n<li>Questions that have unequal number of positive and negative responses.<\/li>\n<\/ul>\n<p style=\"padding-left: 60px\">Example:<\/p>\n<p style=\"padding-left: 60px\"><em>Overall, how would you rate NIHSeniorHealth? <\/em><\/p>\n<p style=\"padding-left: 60px\"><em>Excellent | Very Good | Good | Fair | Poor\u00a0 <\/em><\/p>\n<p style=\"padding-left: 60px\">Notice that \u201cGood\u201d is the middle option (which should be neutral), and some people consider \u201cFair\u201d to be a slightly positive term.<\/p>\n<ul>\n<li>Leading questions, which are questions that are asked in a way that is intended to produce a desired answer.<\/li>\n<\/ul>\n<p style=\"padding-left: 60px\">Example:<\/p>\n<p style=\"padding-left: 60px\"><em>Most people find MedlinePlus very easy to navigate.\u00a0 Do you find it easy to navigate? \u00a0(Yes\u00a0\u00a0 No)<\/em><\/p>\n<p style=\"padding-left: 60px\">How would you feel if you had trouble navigating MedlinePlus? It would be hard to say \u2018No\u2019 to that question.<\/p>\n<ul>\n<li><em>\u00a0<\/em>Double-barreled questions, which are two questions in one.<\/li>\n<\/ul>\n<p style=\"padding-left: 60px\"><em>\u00a0<\/em>Example:<\/p>\n<p style=\"padding-left: 60px\"><em>\u00a0Do you want so lower the cost of health care and limit compensation to medical malpractice lawsuits?<\/em><\/p>\n<p style=\"padding-left: 60px\"><em>\u00a0<\/em>This question has two parts \u2013 to answer yes or no, you have to agree or disagree with both parts. And who doesn&#8217;t want to lower health care costs?<\/p>\n<ul>\n<li>\u00a0Loaded questions, which are questions that have a false or questionable logic inherent in the question (a &#8220;Have you stopped beating your wife\u201d kind of question). Political surveys are notorious for using loaded questions.<\/li>\n<\/ul>\n<p style=\"padding-left: 60px\">Example:<\/p>\n<p style=\"padding-left: 60px\"><em>Are you in favor of slowing the increase in autism by allowing people to choose whether or not to vaccinate their child?<\/em><\/p>\n<p style=\"padding-left: 60px\">This question makes the assumption that vaccinations cause autism. It might be difficult to answer if you don\u2019t agree with that assumption.<\/p>\n<p>The NEO has some suggestions for writing questions in their <a href=\"https:\/\/nnlm.gov\/sites\/default\/files\/neo\/files\/booklet-three.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Booklet 3: Collecting and Analyzing Evaluation Data, page 5-7<\/a>.<\/p>\n<p><strong>Questionnaire respondents<\/strong><\/p>\n<p>People think of the questions as a way to bias questionnaires, but another form of bias can be found in the questionnaire respondents.<\/p>\n<ul>\n<li>Straw polls or convenience polls are polls that are given to people the easiest way. For example polling the people who are attending an event, or putting a questionnaire on a newspaper homepage (or your Facebook page).\u00a0 The reason they are problematic is that they attract response from people who are particularly interested or energized by a topic, so you are hearing from the noisy minority.<\/li>\n<li>Who you send the questionnaire to has a lot to do with why you are sending out the questionnaire. If you want to know about the opinions of people in a small club, then that\u2019s who you would send them to. But if you are trying to reach a large number of people, you might want to try sampling, which involves learning about randomizing.\u00a0 (Consider checking out the Appendix C of NNLM PNR\u2019s <a href=\"https:\/\/nnlm.gov\/neo\/professional-development\/guides\" target=\"_blank\" rel=\"noopener noreferrer\"><em>Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach<\/em><\/a>). Keep in mind that the potential bias here isn\u2019t necessarily in sending the questionnaires to a small group of people, but in how you represent the results of that questionnaire.<\/li>\n<li>Low response rate may bias questionnaire results because it&#8217;s hard to know if your respondents truly represent the group being surveyed.\u00a0 The best way to prevent response rate bias is to follow the methods described in this NEO post <em><a href=\"https:\/\/nnlm.gov\/evaluation\/blog\/2015\/10\/30\/boosting-response-rates-with-invitation-letters\/\" target=\"_blank\" rel=\"noopener noreferrer\">Boosting Response Rates with Invitation Letters<\/a><\/em> to ensure you get the best response rate possible.<\/li>\n<\/ul>\n<p><strong>Lastly, the Purpose of the Questionnaire<\/strong><\/p>\n<p>Just like looking for bias in news or health information or anything else, you want to think about is who is putting out the questionnaire and what is its purpose?\u00a0 A\u00a0 questionnaire isn\u2019t always a tool for objectively gathering data.\u00a0 Here are some other things a questionnaire can be used for:<\/p>\n<ul>\n<li>To energize a constituent base so that they will donate money (who hasn\u2019t filled out a questionnaire that ends with a request for donations?)<\/li>\n<li>To confirm what someone already thinks on a topic (those Facebook polls are really good for that)<\/li>\n<li>To give people information while pretending to find out their opinion (a lot of marketing polls I get on my landline seem to be more about letting me know about some products than really finding out what I know).<\/li>\n<\/ul>\n<p>If you want to know more about questionnaires, here are some of the NEO resources that can help:<\/p>\n<p><a href=\"https:\/\/nnlm.gov\/neo\/professional-development\/guides\" target=\"_blank\" rel=\"noopener noreferrer\"><em>Planning and Evaluating Health Information Outreach Projects, Booklet 3: Collecting and Evaluating Evaluation Data<\/em><\/a><\/p>\n<p><a href=\"https:\/\/nnlm.gov\/evaluation\/blog\/2015\/10\/30\/boosting-response-rates-with-invitation-letters\/\" target=\"_blank\" rel=\"noopener noreferrer\"><em>Boosting Response Rates with Invitation Letters<\/em> <\/a><\/p>\n<p><a href=\"https:\/\/nnlm.gov\/evaluation\/blog\/category\/questionnaires-and-surveys\/\" target=\"_blank\" rel=\"noopener noreferrer\">More NEO Shop Talk blog posts about Questionnaires and Surveys <\/a><\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"text-decoration: underline\">Picture attribution<\/span><\/p>\n<p><a href=\"https:\/\/commons.wikimedia.org\/wiki\/File:Villano.png\" target=\"_blank\" rel=\"noopener noreferrer\">Villano by J.J.<\/a>, licensed under the Creative Commons <a href=\"https:\/\/creativecommons.org\/licenses\/by-sa\/3.0\/deed.en\" target=\"_blank\" rel=\"noopener noreferrer\">Attribution-Share Alike 3.0 Unported<\/a> license.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>People in my social media circles have been talking lately about bias in questionnaires.\u00a0 There are biased questionnaires.\u00a0 Some of them are biased by accident and some are on purpose.\u00a0 Some are biased in the questions and some are biased in other ways, such as the selection of the people who are asked to complete&#8230; <a href=\"https:\/\/news.nnlm.gov\/nec\/2017\/03\/06\/the-dark-side-of-questionnaires-how-to-identify-questionnaire-bias\/\">Read More &raquo;<\/a><\/p>\n","protected":false},"author":2959,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[13],"tags":[],"class_list":["post-5249","post","type-post","status-publish","format-standard","hentry","category-questionnaires-and-surveys"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p8ICUo-1mF","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/5249","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/users\/2959"}],"replies":[{"embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/comments?post=5249"}],"version-history":[{"count":1,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/5249\/revisions"}],"predecessor-version":[{"id":6507,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/5249\/revisions\/6507"}],"wp:attachment":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/media?parent=5249"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/categories?post=5249"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/tags?post=5249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}