![\"Likert](\"\/neo\/wp-content\/uploads\/sites\/3\/2016\/07\/Beiber-1.jpg\")
<\/a><\/p>\nThis is not a question from my colleague\u2019s survey.\u00a0 (I thought I should point that out in case you were wondering about my professional network.)\u00a0 However, her response options were laid out similarly, with the most positive ones to the left.\u00a0 So I shared with her the best practice I learned from my own training in survey design: Reverse the order of the response options so the most negative ones are at the left.<\/p>\n
<\/p>\n
<\/p>\n This \u201cnegative first\u201d response order tends to be accepted as a best practice because it is thought to reduce positive response bias (that is, people overstating how much they agree with a given statement). \u00a0Because I find myself repeating this advice often, I thought the topic of \u201cresponse option order\u201d would make a good blog post. After all, who doesn\u2019t like simple little rules to follow?\u00a0 To write a credible blog post, I decided to track down the empirical evidence that underpins this recommended practice.<\/p>\n And I learned something new: that evidence is pretty flimsy.<\/p>\n This article by Jeff Sauro, from Measuring U, provides a nice summary article<\/a> and references about the evidence for our \u201cleft-side bias.\u201d\u00a0 \u201cLeft-side bias\u201d refers to the tendency for survey-takers to choose the left-most choices in a horizontal list of response options. No one really knows why we do this.\u00a0 Some speculate it\u2019s because we read from left to right and the left options are the first ones we see.\u00a0 I suppose we\u2019re lazy: we stop reading mid-page and make a choice, picking one of the few options we laid eyes on. This speculation comes from findings that show the left-side bias is more pronounced if the question is vague or confusing, or if the survey takers flat-out don\u2019t care about the question topic.<\/p>\n Here\u2019s how left-side bias is studied. Let\u2019s say you really care what people think about Justin Bieber. (I know it\u2019s a stretch, but work with me here.)\u00a0 Let\u2019s say you asked 50 people their opinion of the pop star using the sample question from above. You randomly assign the first version to 25 of the respondents (group 1) and the second version 2 to the other 25 respondents (group 2). \u00a0Findings would predict that group 1 will seem<\/em> to have a more favorable opinion of Justin Bieber, purely because the positive options are on the left.<\/p>\n Sauro\u2019s references for his article do provide evidence of \u201cleft-side bias.\u201d However, after reviewing his references, I drew the same conclusion that he did: the effect of response option order is small, to the point of being insignificant. I became more convinced that this was the case when I looked for guidance in the work of Donald Dillman, who has either conducted or synthesized studies on almost every imaginable characteristic of surveys. Yet I could not find any Dillman source that addressed how to order response options for Likert questions.\u00a0 In his examples, Dillman follows the \u201cnegative option to the left\u201d principle, but I couldn\u2019t find his explicit recommendation for the format. Response option order does not seem to be on Dillman’s radar.<\/p>\n So, how does all this information change my own practice going forward?<\/p>\n For new surveys, I\u2019ll still format Likert-type survey questions with the most negative response options to the left. \u00a0You may be saying to yourself, \u201cBut if the negative options are on the left, doesn\u2019t that introduce negative<\/em> bias?\u201d Maybe, but I would argue that using the \u201cnegative to the left\u201d format will give me the most conservative estimate of my respondents\u2019 level of endorsement<\/em> on a given topic.<\/p>\n However, if I\u2019m helping to modify an existing survey, particularly one that has been used several times, I won’t suggest\u00a0changing the order of the response options. If people are used to seeing your questions with positive responses to the left, keep doing that.\u00a0 You\u2019ll introduce a different and potentially worse type of error by switching the order. More importantly, if you are planning to compare findings from a survey given at two points in time, you want to keep the order of response options consistent. \u00a0That way, you’ll have a comparable amount of error cause by bias in time 1 and time 2.<\/p>\n Finally, I would pay much closer attention to the survey question itself.\u00a0 Extreme language seems to be a stronger source of bias than order of the response options. Edit out those extreme words like \u201cvery,\u201d\u00a0 \u201cextremely,\u201d \u201cbest\u201d and \u201cworst.\u201d\u00a0\u00a0 For example, I would rewrite our sample question to \u201cJustin Bieber is a good singer.\u201d\u00a0\u00a0 Dillman would suggest neutralizing the question further with wording like this: \u201cDo you agree or disagree that Justin Bieber is a good singer?\u201d<\/p>\n Of course, this is one of many nuanced considerations you have to make when writing survey questions.\u00a0 The most comprehensive resource I know for survey design is Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method<\/em> (4th<\/sup> ed.), by Dillman, Smyth, and Christian (Hoboken, NJ: John Wiley and Sons, Inc, 2014).<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" I recently reviewed a questionnaire for a colleague, who used one of the most common question formats around: the Likert question. Here is an example: This is not a question from my colleague\u2019s survey.\u00a0 (I thought I should point that out in case you were wondering about my professional network.)\u00a0 However, her response options were… Read More »<\/a><\/p>\n","protected":false},"author":2959,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[27],"tags":[],"class_list":["post-4519","post","type-post","status-publish","format-standard","hentry","category-blog"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p8ICUo-1aT","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/4519","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/users\/2959"}],"replies":[{"embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/comments?post=4519"}],"version-history":[{"count":2,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/4519\/revisions"}],"predecessor-version":[{"id":6468,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/posts\/4519\/revisions\/6468"}],"wp:attachment":[{"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/media?parent=4519"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/categories?post=4519"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/news.nnlm.gov\/nec\/wp-json\/wp\/v2\/tags?post=4519"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<\/a><\/p>\n<\/a><\/p>\n