Friday, March 7, 2014

More Successful Online Surveys Tip 3: Design the Research Correctly

In my earlier Blog, 11 Tips for Doing More Successful Online Surveys, the third tip was "Design the Research Correctly." By this I mean: don't do an online survey if some other kind of research is more appropriate.

Online surveys are a great research tool, but they are only one tool. Alternative tools are:
  • Other kinds of surveys, such as telephone, mail and door-to-door surveys
  • Qualitative research, such as focus groups and one-on-one interviews
  • Secondary research, such as online searches of the Internet and specific data bases as well as good old library research

Secondary and Qualitative Research
In the example I've used in my last two blogs (One Way Research Can Help You to the Management Table and More Successful Online Surveys Tip 2: Get the Question Right), we've used as an example a not for profit that's suffering from decreasing contributions. The first research I recommended in this hypothetical case was reviewing internal records to determine where the decrease was coming from. That would be secondary research relying on company data. I also recommended that once we determined the kinds of people who were not giving as much, as frequently or at all, that we talk to a few to find out why. This would be either one-on-one interviews or small focus groups--both forms of qualitative research. After learning some reasons for people not giving as much, we could incorporate those into a survey and quantify it. That is, we could determine which reasons accounted most for the fall in giving.

I assume we could do this survey online, because I assume we would have the e-mails of people who had given in the past. Even if we sent the survey to every person who ever had given to the organization, meaning we sent it to the entire population or a census, we still would probably not get a representative sample of this population, because there likely would be some bias related to those who wanted to respond and those who didn't. But in this case, I don't think it would matter very much whether the sample was completely representative. We're simply trying to get a handle on what's going on, and rough numbers will probably be good enough for that.

If, however, we were doing a survey that was going to be critical to making a very important prediction, such as who will win an election, or on which your organization were going to make a high stakes decision, such as siting a new multi-million- or billion-dollar facility, an online survey might not be the best methodology.

Representative Sample a Prerequisite for Statistic Validity
When the stakes are high, you want your numbers to be as close as possible to the real numbers for the population you are surveying. This means you want your sample to be representative of your population. If your sample is representative (and you avoid other biases), the results you get from your survey will conform to statistical rules, and you can assign probabilities (confidence levels) to how likely it is the percentages for the sample (within specified ranges) represent the true percentages for the population.

Clear Population Definition
Before you pursue your representative sample, you need to clearly define your population. If you wanted to determine who was likely to win an election in Illinois, you would not include people registered to vote in Wisconsin in your poll. You probably would not even include all people registered to vote in Illinois. You would narrow your population to those likely to vote in the next election and might even use a question like that to screen respondents. You would not be able to survey the entire population, so you would want a representative sample of, say, 1,000 of them. 

Definition of Representative (Random) Sample
This means you want a sample in which every member of the population is as likely to be included in the sample as every other member of the population. This is the definition of a random sample, and in order to get one, you need access to every member of the population.

Can You Get a Random Sample Through an Online Survey?
To achieve a representative sample through an online survey, you would need to have a list of the e-mail addresses of every member of the population. You may well have all the emails for all the employees of a given organization or all the customers who have ever purchased a product online. But when you start talking about the U.S. population or the community around a potential factory site, you almost certainly do not have an e-mail for every individual. So, in these cases, some of the people in the population (those for whom you do not have e-mails) do not have the same likelihood of being included in the sample, and you cannot survey a random sample online.*

How Polling Companies Get a Random Sample
Traditionally, polling companies have done this kind of survey by telephone using telephone landlines. Now, because more and more households (especially younger households) are dropping land lines in favor of mobile phones, polling companies use a mixture of randomly selected land-line and mobile-phone numbers to generate representative samples.

Think Through What You Need
I apologize for going into so much detail regarding sampling methodology, but I wanted to make the point that sometimes even when a survey is appropriate, an online survey may not be.

So, returning to my main point, if you want your research to truly help your organization, don't just rely on surveys. Instead, determine what kind of information you actually need and think through the  best way to get it. It may be an online survey, but then again, it may not.

* Another issue with e-mail lists is that you may also have more than one e-mail address for some individuals and only one for others. This increases the likelihood of those with multiple addresses being included in the sample. This, too, reduces the randomness of the sample.

Monday, February 10, 2014

More Successful Online Surveys Tip 2: Get the Question Right

In an earlier Blog I offered 11 tips for doing more successful online surveys. Tip 2 was:
  • Get the research question right. Know what you really want to know and how you will use the information in decision support.
What the Experts Say
Two of the marketing texts I have on my bookshelf quote the same adage on this topic. So, I will continue this potentially plagiaristic tradition and quote them: "a problem well defined is half solved." 

What You Learned in High School and College
Think back to the research papers you had to do in high school and college, and you'll get a sense of how the research question should work. First it needs to capture the specific information need for your organization. But it also needs to be narrow enough to focus the research so your survey doesn't end up taking 30 minutes to complete and generate data that confuses the issue at hand. 

A Case History
In my last Blog I discussed an imaginary case in which a not for profit organization's donations were going down. We determined that our research needed to identify which scenario was drying up funding (fewer funders, funders giving less, fewer new funders, etc.), and once we'd determined that, we would do research with the right group to determine why their behavior had changed and what we needed to do to get them to give at previous or higher levels.

Applying Analytics to the Data We Have
I'd like to point out that the first part of this research--determining which scenario was responsible for the loss of funding--would almost certainly be research into the data the organization already has. It would be applying analytics to the organization's receipt records. So this wouldn't be an external survey but would generate information essential to asking the right research questions and creating a successful survey. This is an important principle: we should take advantage of all the data we have to enrich our understanding of business situations.

Questions and Probable Methodology
So the research questions we would want to address in our followup research are:

  • "Why are funders no longer giving the same amounts they used to (or why are fewer funders giving)?"
  • "What can we do to turn this around?" 

We'd probably want to start with some one-on-one interviews to get a list of reasons why and actions we could take to reverse the situation so we could then survey the right target and determine which reasons and actions are most important to respondents. The answers to these questions likely will be rife with both policy and communications implications.

While we are at it, we might add the question

  • "What communications channels are most appropriate for reaching funders?" 

The Right Questions Focus the Research
So we have three questions that will focus our survey. We've also done background research to ensure we know which set of stakeholders we should be doing our research with. By formulating clear research questions we have given ourselves a solid base for a successful survey.

Tuesday, January 21, 2014

One Way Research Can Help You to the Management Table

In my last Blog, I offered 11 tips for doing more successful online surveys. The first tip was:
  1. Understand the business context for the research. Know what business decision will be affected by the research and how. 
Research Should Help Align Your Communications with Your Business Objectives
One of the strengths of research is the thought that goes into it. Done right, it can link your communications programs to the business goals of your organization and help you on your way into the top management group.

An Imaginary Case History
Since most of my readers and I are in communications and marketing, let's suppose we work at a not-for profit organization that fights a disease. Our management comes to us saying we need to do a communications campaign to raise the organization's visibility.

Our first question is, of course: "Why?"

As it turns out, our organization fights the disease by raising funds and then donating those funds to institutions for medical research to find a cure. For the last two years, donations have dropped, and management is in crisis mode. If this trend continues, the organization will fold. 

Management's immediate reaction is that the organization must need more visibility. Or at least more visibility can't hurt, while they try to figure out what the real problems might be. 

However, as communicators, we know that if there are issues with the organization itself, (e.g. donors are losing faith in it, or for some reason do not trust the organization's management) raising visibility could make matters worse, if the trust issue is not addressed first.

   What Research Should We Do?
Now we understand where the request is coming from. We also know, we need to research the business situation before we can do an effective visibility campaign or argue that we should not. We know, too, that the research needs to focus on funding. But what information do we need to help our organization and to communicate effectively about it?

It seems clear the real questions are:
  1. What can we do to reverse the downward funding trend?
  2. Why has our funding dropped? 
Thinking as researchers we might come up with some hypotheses to test. For example:
  1. Funding has dropped because there has been a drop in the number of funders.
  2. Funding has dropped because funders are donating less on average than they used to.
  3. Funding has dropped because we used to have a regular influx of new funders and that has dropped off in either number or average amount. 
And there may be others.

   Internal Analysis
Testing these hypotheses probably is a matter of reviewing internal records on donation sources. And once we know which of the above hypotheses is true, we know who we should target for our external research.

   External Research
Whichever group the initial research suggests, we need to ask these people why they are no longer donating in the way they were. We might start with one-on-one interviews and follow with a survey to quantify our findings. 

   Communications and Business Insights
This information should lead us to understand the root cause of the drop in funding and suggest a messaging strategy. More important, though, it likely will suggest business strategies to better align the organization with its key funding stakeholders. For example, if the problem has to do with something about the organization or its behavior, management likely will need to address that to reverse the funding situation. And making those changes is what the communication should be about. 

A Real Case History
For example, I did work with a not for profit organization that was suffering falling donations. We did interviews with four stakeholder groups and learned that every one had the same complaint: They felt the organization listened to them but never did any of the things they asked.

As it turned out, the organization actually did do some of the things its stakeholders asked of it, but didn't communicate to stakeholders that it had. This could be easily fixed through  increased communications. 

In addition, management decided the organization should be more responsive to its stakeholders. It increased the amount it communicated about the actions it took. It also took the time and effort to explain to stakeholders why it did not take some of the actions stakeholders requested. 

This turned around the fortunes of the organization as donations picked up significantly.

So, by using the research to understand the business issues as well as the communications issues, the communications people who initiated the project made a significant contribution to the management of the organization. And this was recognized by senior management.

                                                                           # # #

If you'd like help working through issues like this with your organization, please contact me.

Monday, January 6, 2014

11 Tips for Doing More Successful Online Surveys

Happy New Year!

A number of my future Blogs will address how to do better online surveys, such as those we do with SurveyMonkey or Qualtrics. I use and like both.

Surveys, when done properly, are a great way to gather the kind of information you can use to create communications campaigns that achieve business objectives because they are based on real information and insight into the target audiences surveyed rather than guesses and assumptions about them. Real information on what they care about, how best to reach them and so forth.

These tips are for all surveys. I mention online surveys, because they are so easy to use that more and more folks are doing surveys themselves. Unfortunately, the insights users get from these surveys frequently are less than they had hoped. They sometimes are even be misleading because of methodological issues I will discuss.

So here are my top tips. These are the ones that have come to my mind, and I'm sure many of my research colleagues will have others. I invite them to add their comments.
  1. Understand the business context for the research. Know what business decision will be affected by the research and how. 
  2. Get the research question right. Know what you really want to know and how you will use the information in decision support.
  3. Design the research correctly. People frequently think they should do a survey when they should be doing focus groups, one-on-one interviews, searching data bases or even going to the library.
  4. Select and sample the right population. Online surveys depend on e-mail addresses, so people doing this kind of research tend to survey the people they have e-mails addresses for. However, they should be thinking about the actual target stakeholder universe and how they can get a representative sample of that universe. Consider that presidential election polls are not done online.
  5. Write questions that do not lead respondents to answer in a specific way (bias the question).
  6. Write questions that offer the answer choices respondents would select, so they can answer honestly and appropriately. For example, I have seen rating scales used that offered only positive responses ("Did you like this class, or did you really like this class?"). In such a case you get a positive rating, but you've learned nothing and your results may be misleading.
  7. Develop questionnaires with a logical flow. Flow problems can be jarring to respondents or inadvertently reveal information that biases answers to later questions.
  8. Use question formats that are easy for respondents to manage. For example, do not ask them to rank order the importance of 20 elements.
  9. Keep your questionnaires to two to three minutes, if possible. Try to focus on one topic.
  10. Use graphs and charts to present the data in a form that is easy to grasp. Online survey services and Excel offer ways to easily visualize data. But, for your audience to grasp the data, it is important that you use the right chart or graph for the situation and format it to be easily understood. For example, if you ask respondents which topic they would most like to hear about, your chart should be rank ordered to show the most popular topic at one end and the least popular at the other. This way, your audience will quickly grasp what is important and what is not.
  11. Make the leap from data to insight. This goes back to our first point about understanding the business purpose behind the research. When you have the answers to your survey questions, you need to make a case for what kind of decision the data supports. But business decisions are not just about data. They are about what is best for the organization. So, making insightful recommendations requires integrating your understanding of the data with your understanding of the business context. And the broader your contextual understanding is (that is, if it goes beyond communications to finance, accounting, HR, research and development, marketing and sales, etc.) the more valuable it will be to your organization.
I will go into more detail on these bullets in future posts.

If you'd like help developing, executing or interpreting surveys, please contact me.

Friday, November 29, 2013

Science News Post on Big Data and Science

A Science News article argues Big Data is bad for science. One comment says it is still good for marketing.

Friday, October 4, 2013

A Great Example of PR Using Data

In my most recent posts I've talked about the importance of PR using data -- whether big or little -- to drive what it does, so that PR contributes to achieving business goals. David B. Rockland, who heads the research and change management businesses at Ketchum has written this case history on how Fed Ex did just that. Here is the link: