Sampling Strategies for Opinion Surveys – More is Not Necessarily Merrier
Posted by David McMurray on February 18th, 2013
In my experience, many institutional surveys are coordinated by internal company resources and are inexperienced in the best practices of sound survey research. Inexperience is often the case when sampling a population for opinion surveys. Today, I will focus on general customer surveys and simple random sampling. My comments apply best to an organization surveying general customer populations of about 5,000 or more. Sampling is necessary because it may be impractical to survey every customer, and, statistically speaking, surveying every customer is unnecessary.
Let’s start with a few basic definitions:
- Population – A group of people or things that are of interest to us. The size of a population depends on how we define our group of interest. A population could be all customers, all new customers, customers in a specific region or in a certain age group, customers who use a specific product or service, etc.
- Census – Regardless of its size, when we survey the entire population it’s a census. Because conducting a census can be so expensive (and unnecessary) we frequently adopt the alternative – sampling.
- Sample – A subset of the population, regardless of how large or small the population is. Feedback from the sample can be representative of the entire population, assuming there are enough responses and the sample is not inherently biased.
So the question is, “How large must a sample be in order to accurately represent the opinions of the population?” One would intuitively think that as the population size increases, the required sample size would increase proportionately. That is not the case, however, as a random sample of 400 is adequate for any general population larger than about 5,000. 400 random customer responses are equally good for measuring the opinions of a population of 5,000 as it is good for populations of 50,000 or 500,000! We use “400” as the general rule, even though technically the sample requirement may be slightly less. As shown below, after a population reaches 5,000 cases, we conveniently round up to a required sample size of 400.
| Population Size | Sample Size* |
| 1,000 | 286 |
| 2,000 | 333 |
| 3,000 | 353 |
| 4,000 | 364 |
| 5,000 | 370 |
| 50,000 | 397 |
| 500,000 | 400 |
| 5,000,000 | 400 |
*Note: this is based on a 95% confidence level and a +/-5% margin of error, which is the widely-accepted standard for random sampling.
Remember that to get the 400 required responses you will have to send out a lot more surveys. Simply divide 400 (or the desired number of responses) by the predicted response rate to get the number of surveys you need to send out. For instance, if you expect a response rate of 12% and want 400 surveys returned, you will send out 3,333 surveys (400/.12=3,333). But if you only expect an 8% response rate, you will send out 5,000 surveys (400/.08=5,000). If you do not have a good feel for how many surveys you will get back, estimate a lower response rate. That way, you’ll be more likely to get the required number of surveys back.
Additionally, the survey must be sent out to a random sample. Think of this as ‘stirring the pot’ before testing a pot of soup for taste and temperature. In simple random sampling, every member of a population has an equal chance of being chosen as part of the sample. This is most easily accomplished by randomly sorting the list of potential survey recipients and then taking every nth case.
It is important to avoid biasing the sample. For instance, if you only distribute surveys to those who have email addresses or telephone numbers, you may be missing many in the general population who provide you with only a physical mailing address. In other words, you need to know your population before you can accurately determine your sample. This also underscores the value of having email addresses and telephone numbers in your customer databases, since contacting customers via USMail adds a lot of expense to your research.
In my next blog I’ll talk about more sophisticated sampling methods, as well as other pitfalls to avoid when deciding who will be invited to respond to a survey.
