Weight – Are Your Survey Results Biased?

Photo credit: plenty.r. on Flickr.As you may already know, there is an ongoing debate about the merits of Do-It-Yourself (DIY) market research. This debate is usually focused on survey research, in particular.

As a blog post on blog.vovici.com, titled “Centralized vs. Decentralized Research: The Verdict Is In,” by Jeffrey Henning, Chief Marketing Officer at Affinnova, points out, “Sorry, Cathy, the verdict is in, but I didn’t deliver it. End users did. Today, more surveys are done outside market research departments and market research firms than inside them. The days of the traditional market research department are numbered; they just haven’t been counting them down.”

As you can tell from my last post, titled “Making a Case for Hiring Marketing Strategy & Research Firms: Unknown Unknowns,” I tend to agree with the argument that Cathy Harrison, Client Services Executive at Chadwick Martin Baily, makes in her blog post on blog.cmbinfo.com, titled “Be Wary of Decentralizing the Corporate Market Research Function.”

However, Jeffrey Henning makes a good point.

That is part of the reason why I often refer to myself as a former survey researcher. That and the fact that I think there are other areas of marketing that are more interesting and have better growth potential.

Nevertheless, I feel that there are issues regarding the accuracy and validity of survey data that businesses should be aware of, whether they choose to hire a marketing strategy and research firm to help them with their survey research needs or if they choose to use the more inexpensive DIY market research tools out there.

Why Are You Doing the Survey?

If you are conducting exploratory research, or in other words, if the objective of your survey research is to identify possible problems or issues that are out there without making any population estimates, the DIY market research route is probably the best way to go, as it is the much cheaper option. And, if this is the case, you probably don’t need to worry about the topic that I am going to be writing about for the rest of this blog post.

However, if you are attempting to conduct a more scientific survey, with the goal of making population estimates similar to the findings that are reported by major market research firms (e.g., Nielsen, Gallup, etc.,) then there are several issues that you need to consider to ensure that you are basing your business decisions on survey research findings that you can trust.

For the rest of this blog post, I plan to talk about one of those issues, in particular.

Did You Weight Your Data?

People who are not trained in market research are often unaware of the need to weight the data collected via a survey if they are trying to use the data to make population estimates.

That is, if want to estimate amounts, distributions or proportions for a given population based on your survey data, you need to make sure that your data reflects the actual population that you are trying to estimate. (Note: I am not going to get into oversampling and other sampling techniques today. I will save that for future blog posts.)

Even if you have chosen a simple random sample, your data can still be biased if you have a disproportionate number of people from certain demographic groups or subpopulations responding to your survey. This is known as differential nonresponse.

Differential Nonresponse

People tend to have different thoughts, beliefs and consumption patterns based on their past experiences and their personal identities.

Therefore, if you get a disproportionate number of responses from any particular demographic group or subpopulation, it can potentially have a huge impact on the accuracy of your population estimates.

In an article in The Public Opinion Quarterly, titled “When to Weight: Determining Nonresponse Bias In Survey Data,” Dr. Lewis Mandell explains, “In this manner, differential nonresponse may introduce bias in population estimates. As a very simple illustration, consider the use of a telephone survey to ascertain income. Since lower-income families are less likely to have a telephone, their response rate will be lower than that of higher-income families. Without weighting for nonresponse, the estimate of mean population income will be biased upward since too few lower-income families are included.” (Note: This example was written in 1974. Since then, telephone penetration rates have increased. However, the example still illustrates the point that I am trying to make.)

Although it will vary by research topic, some of the independent variables that can influence the overall results include income, educational attainment, race, gender and region. (Note: If you are doing B-to-B research, you might want to look at asset size, number of fulltime employees, gross revenue, region, etc.)

If you have a disproportionate number of people from any demographic group, it can lead to biased data. This is particularly true if there are major differences in the way people respond to a question based on their demographic affiliation.

Weighting the Data

Weighting is basically a way to make sure that the sample that you are using to make population estimates actually reflects the true population distribution.

I am not going to get into the technical details in this blog post. However, you can find out more information about weighting survey data by searching on Google.


Many businesses are choosing to conduct their survey research in-house using the inexpensive DIY market research tools that are currently available to them.

If the purpose of the survey is exploratory in nature, in other words, if it is designed to identify potential issues that your customers are having with your products or services, this could be an acceptable solution.

However, if your business is trying to make actual population estimates, the DIY route might not be such a good idea.

As I pointed out in the last blog post, there are “unknown unknowns” that people who are not trained in market research may be unaware of.

In this post, I pointed out one of the possible “unknown unknowns” by focusing on the need to weight survey data to reflect the actual population distribution.

There are many others that I will talk about in future blog posts.

Keep in mind, the problem is not always with the DIY market research tools themselves. In fact, some of the tools do give you the ability to weight your data. Or, at a minimum, they give you the ability to export your data to a statistical software program like SPSS or SAS. These programs give you the ability to do a lot of cool things, including weighting.

The real issue is the lack of knowledge that you should be doing some of these things in the first place.

While weighting your data might not seem like a big deal, it can potentially have a huge impact on the overall estimates of amounts, distributions or proportions for a given population that are based on your survey data. (I have seen this happen many times.)

If you don’t weight your survey research data, it could potentially have a huge impact on your bottom line because you will be making business decisions that are based on biased survey research findings.

Photo credit: plenty.r. on Flickr.

Chad Thiele

Chad Thiele

Marketing analyst and strategist, content curator, applied sociologist, proud UW-Madison alumnus, and an Auburn-trained mobile marketer. My goal is to help businesses identify trends that will help them achieve their marketing objectives and business goals. I'm currently looking for my next career challenge. Please feel free to contact me anytime at: chadjthiele@gmail.com.