Nonrandom Ideas for Using Random Samples

Posted by nnlmneo on April 7th, 2017 Posted in: Questionnaires and Surveys

---

I bet random sampling is something most of you learned about in your Intro to Statistics class, then never thought about again.  Truthfully, many program evaluators seldom use random sampling.  We often are working with smallish groups of program participants and can seek feedback from everyone (also known as conducting a census).

Besides, even if you have access to contact information for a huge group of people (e.g., your membership, city, user base, fan club), why not invite everyone? Survey software packages usually charge a flat annual fee, so it costs the same to invite 80 or 8,000 people to fill out your questionnaire. In fact, with social media, you may even be able to post surveys at no cost.

Today, I am here to give you examples of how you can use random sampling for pesky evaluation challenges. Then, I end with simple resources for creating your own random samples.

Challenge #1: A convenience sample just isn’t good enough. If you invite all 10,000 people on a mailing list (aka your population) to complete a questionnaire and 3% respond, that is not a census. It’s a convenience sample. Those few respondents likely were energized by extreme positive or negative opinions about the topic. You have no idea the degree to which their opinions represent the majority. Convenience samples aren’t necessarily useless. Often, you can corroborate findings and assess bias if you have other evaluation data for comparison.  A better response rate, however, automatically increases faith in the validity of your findings. A personalized communication process  is a key tactic for boosting response rate. It does not come without time cost. You must carefully review your email list. Are names spelled correctly? Are email addresses properly formatted? When email messages bounce, you should correct addresses and resend invitations. Nonrespondents should be contacted through personally written email messages or phone calls. Really late responders, if budget permits, should be mailed print copies of the questionnaires. A random sample, therefore, allows you to create a statistically valid shortlist of respondents and treat them with TLC.

Challenge #2: You have too many questions you want to ask: A long questionnaire is another potential threat to a good response rate.  If you have a significant number of people in your population, you can create several mini-questionnaires with a portion of your questions on each.  You can then divide up your population randomly and send different questions to each group.

Challenge #3: You are conducting a qualitative interviewing project on a controversial topic:  Usually, random sampling is not recommended for qualitative interviewing projects. Purposeful sampling is more likely to provide a small sample of interviewees with the appropriate mix of experiences and viewpoints for your project. However, when subject matter is controversial, a random sampling method protects you from appearing to “cherry pick” participants who side with your perceived position. For example, your organization may want to implement changes that are getting mixed reactions from various influential stakeholders.  You would use random sampling to demonstrate a good faith effort to represent a broad range of opinions.

Challenge #4: You want to collect evaluation information in a less-than-convenient manner.  It’s true, some evaluation methods can be annoying.  However, random sampling can limit the burden to you and those who help you gather data.  Suppose you want point-of-contact feedback from help desk customers.  You can randomly select a manageable number of weeks out of the 52 in a given year to collect user feedback. Maybe you want to systematically confirm the most popular usage times for your library computers. You could randomly choose different hours each day for a month or two to conduct and record a head count.

Challenge #5: You want to analyze existing data and there’s a lot of it.  In this information age, we are surrounded by organizational and even public information that might be useful in our evaluation projects.  Social media posts, “comment box” feedback, and training participants’ suggestions can be sources of valuable information. These sources also may provide daunting amounts of data. Rather than wade through 50,000 public ideas of how to remodel your library, for example, it is acceptable to randomly select a more manageable portion of responses and analyze those.

So, how do you pull a random sample?

You first have to determine your sample size.  For the qualitative examples described here, your time and resources will be your guide. For surveys conducted with larger populations, you can use an online calculator, such as this one from RAOSOFT. Then, to get your actual sample, you can put your list in Excel and use the RAND function to assigned a random number to individuals in the list and pull a random sample. Check out this video on how to pull a random sample using Excel.

You never know when your evaluation project may call out for a nice random sample. I have described just a few examples here. Happy sampling!