Online surveys are a great research tool, but they are only one tool. Alternative tools are:
- Other kinds of surveys, such as telephone, mail and door-to-door surveys
- Qualitative research, such as focus groups and one-on-one interviews
- Secondary research, such as online searches of the Internet and specific data bases as well as good old library research
Secondary and Qualitative Research
In the example I've used in my last two blogs (One Way Research Can Help You to the Management Table and More Successful Online Surveys Tip 2: Get the Question Right), we've used as an example a not for profit that's suffering from decreasing contributions. The first research I recommended in this hypothetical case was reviewing internal records to determine where the decrease was coming from. That would be secondary research relying on company data. I also recommended that once we determined the kinds of people who were not giving as much, as frequently or at all, that we talk to a few to find out why. This would be either one-on-one interviews or small focus groups--both forms of qualitative research. After learning some reasons for people not giving as much, we could incorporate those into a survey and quantify it. That is, we could determine which reasons accounted most for the fall in giving.
Surveys
I assume we could do this survey online, because I assume we would have the e-mails of people who had given in the past. Even if we sent the survey to every person who ever had given to the organization, meaning we sent it to the entire population or a census, we still would probably not get a representative sample of this population, because there likely would be some bias related to those who wanted to respond and those who didn't. But in this case, I don't think it would matter very much whether the sample was completely representative. We're simply trying to get a handle on what's going on, and rough numbers will probably be good enough for that.
If, however, we were doing a survey that was going to be critical to making a very important prediction, such as who will win an election, or on which your organization were going to make a high stakes decision, such as siting a new multi-million- or billion-dollar facility, an online survey might not be the best methodology.
Representative Sample a Prerequisite for Statistic Validity
When the stakes are high, you want your numbers to be as close as possible to the real numbers for the population you are surveying. This means you want your sample to be representative of your population. If your sample is representative (and you avoid other biases), the results you get from your survey will conform to statistical rules, and you can assign probabilities (confidence levels) to how likely it is the percentages for the sample (within specified ranges) represent the true percentages for the population.
Clear Population Definition
Before you pursue your representative sample, you need to clearly define your population. If you wanted to determine who was likely to win an election in Illinois, you would not include people registered to vote in Wisconsin in your poll. You probably would not even include all people registered to vote in Illinois. You would narrow your population to those likely to vote in the next election and might even use a question like that to screen respondents. You would not be able to survey the entire population, so you would want a representative sample of, say, 1,000 of them.
Definition of Representative (Random) Sample
This means you want a sample in which every member of the population is as likely to be included in the sample as every other member of the population. This is the definition of a random sample, and in order to get one, you need access to every member of the population.
Can You Get a Random Sample Through an Online Survey?
To achieve a representative sample through an online survey, you would need to have a list of the e-mail addresses of every member of the population. You may well have all the emails for all the employees of a given organization or all the customers who have ever purchased a product online. But when you start talking about the U.S. population or the community around a potential factory site, you almost certainly do not have an e-mail for every individual. So, in these cases, some of the people in the population (those for whom you do not have e-mails) do not have the same likelihood of being included in the sample, and you cannot survey a random sample online.*
How Polling Companies Get a Random Sample
Traditionally, polling companies have done this kind of survey by telephone using telephone landlines. Now, because more and more households (especially younger households) are dropping land lines in favor of mobile phones, polling companies use a mixture of randomly selected land-line and mobile-phone numbers to generate representative samples.
Think Through What You Need
I apologize for going into so much detail regarding sampling methodology, but I wanted to make the point that sometimes even when a survey is appropriate, an online survey may not be.
So, returning to my main point, if you want your research to truly help your organization, don't just rely on surveys. Instead, determine what kind of information you actually need and think through the best way to get it. It may be an online survey, but then again, it may not.
_____________________
* Another issue with e-mail lists is that you may also have more than one e-mail address for some individuals and only one for others. This increases the likelihood of those with multiple addresses being included in the sample. This, too, reduces the randomness of the sample.
No comments:
Post a Comment