Tag Do-It-Yourself (DIY) Market Research

Weight – Are Your Survey Results Biased?

Photo credit: plenty.r. on Flickr.As you may already know, there is an ongoing debate about the merits of Do-It-Yourself (DIY) market research. This debate is usually focused on survey research, in particular.

As a blog post on blog.vovici.com, titled “Centralized vs. Decentralized Research: The Verdict Is In,” by Jeffrey Henning, Chief Marketing Officer at Affinnova, points out, “Sorry, Cathy, the verdict is in, but I didn’t deliver it. End users did. Today, more surveys are done outside market research departments and market research firms than inside them. The days of the traditional market research department are numbered; they just haven’t been counting them down.”

As you can tell from my last post, titled “Making a Case for Hiring Marketing Strategy & Research Firms: Unknown Unknowns,” I tend to agree with the argument that Cathy Harrison, Client Services Executive at Chadwick Martin Baily, makes in her blog post on blog.cmbinfo.com, titled “Be Wary of Decentralizing the Corporate Market Research Function.”

However, Jeffrey Henning makes a good point.

That is part of the reason why I often refer to myself as a former survey researcher. That and the fact that I think there are other areas of marketing that are more interesting and have better growth potential.

Nevertheless, I feel that there are issues regarding the accuracy and validity of survey data that businesses should be aware of, whether they choose to hire a marketing strategy and research firm to help them with their survey research needs or if they choose to use the more inexpensive DIY market research tools out there.

Why Are You Doing the Survey?

If you are conducting exploratory research, or in other words, if the objective of your survey research is to identify possible problems or issues that are out there without making any population estimates, the DIY market research route is probably the best way to go, as it is the much cheaper option. And, if this is the case, you probably don’t need to worry about the topic that I am going to be writing about for the rest of this blog post.

However, if you are attempting to conduct a more scientific survey, with the goal of making population estimates similar to the findings that are reported by major market research firms (e.g., Nielsen, Gallup, etc.,) then there are several issues that you need to consider to ensure that you are basing your business decisions on survey research findings that you can trust.

For the rest of this blog post, I plan to talk about one of those issues, in particular.

Did You Weight Your Data?

People who are not trained in market research are often unaware of the need to weight the data collected via a survey if they are trying to use the data to make population estimates.

That is, if want to estimate amounts, distributions or proportions for a given population based on your survey data, you need to make sure that your data reflects the actual population that you are trying to estimate. (Note: I am not going to get into oversampling and other sampling techniques today. I will save that for future blog posts.)

Even if you have chosen a simple random sample, your data can still be biased if you have a disproportionate number of people from certain demographic groups or subpopulations responding to your survey. This is known as differential nonresponse.

Differential Nonresponse

People tend to have different thoughts, beliefs and consumption patterns based on their past experiences and their personal identities.

Therefore, if you get a disproportionate number of responses from any particular demographic group or subpopulation, it can potentially have a huge impact on the accuracy of your population estimates.

In an article in The Public Opinion Quarterly, titled “When to Weight: Determining Nonresponse Bias In Survey Data,” Dr. Lewis Mandell explains, “In this manner, differential nonresponse may introduce bias in population estimates. As a very simple illustration, consider the use of a telephone survey to ascertain income. Since lower-income families are less likely to have a telephone, their response rate will be lower than that of higher-income families. Without weighting for nonresponse, the estimate of mean population income will be biased upward since too few lower-income families are included.” (Note: This example was written in 1974. Since then, telephone penetration rates have increased. However, the example still illustrates the point that I am trying to make.)

Although it will vary by research topic, some of the independent variables that can influence the overall results include income, educational attainment, race, gender and region. (Note: If you are doing B-to-B research, you might want to look at asset size, number of fulltime employees, gross revenue, region, etc.)

If you have a disproportionate number of people from any demographic group, it can lead to biased data. This is particularly true if there are major differences in the way people respond to a question based on their demographic affiliation.

Weighting the Data

Weighting is basically a way to make sure that the sample that you are using to make population estimates actually reflects the true population distribution.

I am not going to get into the technical details in this blog post. However, you can find out more information about weighting survey data by searching on Google.

Conclusion

Many businesses are choosing to conduct their survey research in-house using the inexpensive DIY market research tools that are currently available to them.

If the purpose of the survey is exploratory in nature, in other words, if it is designed to identify potential issues that your customers are having with your products or services, this could be an acceptable solution.

However, if your business is trying to make actual population estimates, the DIY route might not be such a good idea.

As I pointed out in the last blog post, there are “unknown unknowns” that people who are not trained in market research may be unaware of.

In this post, I pointed out one of the possible “unknown unknowns” by focusing on the need to weight survey data to reflect the actual population distribution.

There are many others that I will talk about in future blog posts.

Keep in mind, the problem is not always with the DIY market research tools themselves. In fact, some of the tools do give you the ability to weight your data. Or, at a minimum, they give you the ability to export your data to a statistical software program like SPSS or SAS. These programs give you the ability to do a lot of cool things, including weighting.

The real issue is the lack of knowledge that you should be doing some of these things in the first place.

While weighting your data might not seem like a big deal, it can potentially have a huge impact on the overall estimates of amounts, distributions or proportions for a given population that are based on your survey data. (I have seen this happen many times.)

If you don’t weight your survey research data, it could potentially have a huge impact on your bottom line because you will be making business decisions that are based on biased survey research findings.

Photo credit: plenty.r. on Flickr.

Chad Thiele

Marketing analyst and strategist, content curator, applied sociologist, proud UW-Madison alumnus, and an Auburn-trained mobile marketer. My goal is to help businesses identify trends that will help them achieve their marketing objectives and business goals. I'm currently looking for my next career challenge. Please feel free to contact me anytime at: chadjthiele@gmail.com.

More Posts

Making a Case for Hiring Marketing Strategy & Research Firms: Unknown Unknowns

Photo credit: Valerie Everett on Flickr.Businesses need to make some tough decisions when allocating their marketing budgets.

With the current economic conditions, it is tempting for businesses to attempt to save money by cutting back on their marketing strategy and research spend.

All of the inexpensive Do-It-Yourself (DIY) market research tools out there make it even more enticing for businesses to attempt to do their market research in-house.

However, I would argue that the money spent on hiring a quality marketing strategy and research firm to help you with your marketing strategy and research needs can often have a greater positive impact on your business’s bottom line in the long run than the money that you save by doing your market research in-house. (If the data that you are collecting in-house isn’t valid, are you actually saving your business any money?)

Unknown Unknowns

As the former United States Secretary of Defense, Donald Rumsfeld, once said, “There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.”

No matter what your political affiliation is, most people should be able to recognize that this is a very profound statement.

In fact, identifying the unknown unknowns is one of the main reasons why businesses hire marketing strategy and research firms and business consultants in the first place.

In other words, businesses not only hire marketing strategy and research firms and business consultants so that they can gain from their knowledge and insights and find answers to the questions that they want answered, but they also hire them to identify potential issues that they are unaware of that can have an effect on their bottom line.

The Importance of Specialization

One of the benefits of hiring a marketing strategy and research firm to help your business with its marketing strategy and research is the fact that this is what they live for.

As a result, marketing strategy and research firms will generally hire people with the proper training to handle your specific business needs. This is their area of expertise and how they make their living; therefore they have a vested interest in keeping up with the current trends and issues that can have an effect on their clients’ businesses, including yours.

Furthermore, their experience working with other clients will help them identify key issues that are common in all businesses. In other words, you can benefit from their experience and avoid making some of the same mistakes that other businesses have made in the past. As you know, this can save you time and money.

Never Underestimate the Value of Collaboration

Marketing strategy and research firms often have several people working on similar projects, giving your business access to even more knowledge.

Therefore, even if a new issue is identified that the people who are working on your project haven’t encountered in the past, there is a good chance that one of their colleagues has.

This is a resource that most likely wouldn’t be available to your in-house staff.

Conclusion

Some difficult decisions need to be made when businesses allocate their marketing budgets.

With the current DIY market research tools out there, it is often tempting for businesses to try to do their market research in-house.

However, the money spent on hiring a quality marketing strategy and research firm can often have a greater positive impact on your business’s bottom line in the long run than the money that you save by doing your market research in-house.

With the current economic conditions, having accurate and timely information to base your business decisions on can potentially make the difference between success and failure.

When you hire a marketing strategy and research firm to help your business develop its marketing strategy, your business will gain access to knowledge and experience that will help you solve your current problems, as well as help identify some of the unknown unknowns.

As an added bonus, when you hire firms to help with your marketing strategy and research, you can use the time that you save to focus on what you do best.

Keep in mind, I’m not saying that all of your market research needs to be outsourced. There are several factors that you should take into account when deciding whether or not to hire a marketing strategy and research firm.

However, what I am saying is that there are definitely benefits to hiring a quality marketing strategy and research firm.

I plan to add supporting evidence to my argument in future blog posts. (To find those blog posts, search for “Do-It-Yourself (DIY) Market Research” in the blog topics.)

For now, if you have any thoughts or comments, please let me know in the comments section below.

Photo credit: Valerie Everett on Flickr.

Chad Thiele

Marketing analyst and strategist, content curator, applied sociologist, proud UW-Madison alumnus, and an Auburn-trained mobile marketer. My goal is to help businesses identify trends that will help them achieve their marketing objectives and business goals. I'm currently looking for my next career challenge. Please feel free to contact me anytime at: chadjthiele@gmail.com.

More Posts

Copyright © chadjthiele.com
Every interaction with a consumer IS marketing.

Built on Notes Blog Core
Powered by WordPress